Earlier, R&D activities at the firm level were often considered as a ‘black box’ and an isolated function, which was nearly impossible to be systematically managed and controlled. A defined amount of money was given to R&D, and in the long run the managers expected something useful to turn up. The characteristics associated with R&D activities and R&D personnel also increase the difficulty of performance analysis. New, different organizational structures and control models have forced many managers to rethink the mechanisms related to R&D and its control. One of the most critical motives for measuring R&D performance at the firm level is the validation of the chosen investment level on R&D, i.e. the R&D function has to prove its productivity and significance for the whole company. The fact is that R&D investments often must compete with the other investments in the company. [Ville Ojanen 2003]
A great number of measures or evaluation methods indicating R&D performance have been reported in the literature with different degrees of complexity. The final set of measures to be utilized depends on a number of factors and is specific for the type of organization. This means that common, universal sets of R&D measures do not exist. However, with the help of the recognition of different types of reported measures and measurement dimensions, organizations can become more aware of different possibilities, and can use the earlier reported measure proposals as checklists for certain measurement purposes.
An example of integrated measures of R&D performance by Hewlett-Packard
Integrated methods of R&D performance evaluation combine objective and subjective metrics, thereby enhancing the advantages of both types of measurement. Integrated metrics are often more complicated than individual measures, and their use can be more costly and time-consuming than simple metrics. However, they do not only measure R&D performance but also suggest means for improvement and are reliable (Werner and Souder 1997a).
They presented an example of an integrated metric that combines several objective and subjective metrics:
Effectiveness index (Ei) = (Present value of revenue generated from products introduced in the last 5 years ) / (Present value of last 5 years cumulative R & D costs
Timeless index (Ti) = (Number of projects completed on time during some representative period) / (Number of projects started in that period)
Future potential index (PfI)= (Present value of expected future revenues from technologies currently under development) / (Present value of all costs to develop these technologies)
Pra=Peer rating audit of unfilled future needs that will inhibit the achievement of future greatness, expressed on a scale from 0 to 100%.
Overall assessment of the value of R&D (Oa) = Ei + [(Pfi × Ti) × Pra]
Mcgrath and Romeri (1994) have developed an R&D effectiveness index for measuring the overall success of product development. The formula is
Effectiveness index (EI) =[ (% New Product Revenues × (Net Profit % + R & D %)) / R&D %]
The index has been validated through a study of 45 electronic systems companies. The researchers have found a strong relationship between R&D effectiveness and other performance factors.
Ahmed and Zairi (2000) have reported practical examples on integrated methods to R&D and innovation performance measurement. For instance, Hewlett-Packard has tracked overall effectiveness of product development with the help of two measures:
Staffing level effectiveness (Sle) = (Staff initially forecast as needed for a project) / (Staff actually needed by project)
This measure monitors how close the projections for the staff needed on a project matched the actual staffing required by the project (Ahmed and Zairi 2000).
Stability of the design (Sd) = ( Number of design changes in a project) / (Total cost of project)
This measure tracks the number of design changes made. As large projects might need more changes simply because they are larger, this metric, by dividing against the cost of the project, adjusts for the size of the project (Ahmed and Zairi 2000).
Additionally, the overall effectiveness of the innovation process at HP is followed by the following measure (Ahmed and Zairi 2000):
Innovation effectiveness (Ie) = (Number of projects finishing development) / (Number of projects started development)
We can distinguish a number of different factors that have an influence on the selectable measures or evaluation methods of R&D performance, as well as measurement dimensions to be taken into account in the early phase of the selection and development process of R&D performance measures. The main measurement dimensions we need to categorize including the purpose of R&D performance analysis, the level of R&D performance analysis, the type of R&D to be evaluated and the phase of the R&D process to be measured. One dimension influencing all the other dimensions is the strategy of R&D and the strategic objectives that set the emphasized areas to the perspectives of measurement. Other influencing dimensions could be as the type of industry and the size of the organization. These, as well as the strategic control model chosen for the R&D organization have been found to be of importance in developing the measurement systems for R&D. These factors could be seen as constraints that have consequential influences on the overall dimensions discussed above.
We can include the critical success factors of R&D and the whole company, the recognition of R&D impact chain, and other contingency factors in the set of factors that have an influence on the selection of R&D measures. The influencing factors set requirements for the selectable measures and the final selection criteria. In the literature we can find several suggestions for the measurement of R&D
** This is a lay summary and a part of Ville Ojanen’s work called “Categorizing the Measures and Evaluation Methods of R&D Performance” in 2003.
