Chapter 6

Organizational Performance

---

Introduction

In our framework for profiling an organization, overall performance is seen as a function of the interplay of the organization's unique motivation, its organizational capacity, and forces in the external environment.

Over the past 30 years there have been many attempts to define performance generally and to apply performance concepts to various organizational types. A number of ideas emerge from the organizational performance literature:

• In all organizations, performance relates to organizational purpose.
• Performance also needs to reflect achievements relative to the resources used by the organization.
• Performance must be considered within the environment in which the institution does its work.

The first component reflects the organization's mission, the second component reflects how well the organization manages its resources, and the third, its adaptability within the context of external forces.

Within research institutions, the quantity and quality of research produced is fundamental to the achievement of the mission. But a research institution's performance must also encompass aspects of organizational functioning that are the necessary underlying conditions for researchers to be productive.

To apply traditional assessment terminology to research organizations, organizational performance must integrate the concepts of "effectiveness" and "efficiency." That is, the organization must be able to meet its goals (effectiveness) and to do so with an acceptable outlay of resources (efficiency). Vitally important as well, particularly to IDRC and other Northern granting agencies, is the Southern organization's sustainability over the long term (ongoing relevance). The organization must be able to develop and implement strategies which will ensure research performance over extended periods of time. To do so, its activities and services must remain realistic and connected to stakeholder needs. For when an organization's services and activities are not relevant or are too far-reaching and costly, organizational survival is at risk.

In summary, the performance of institutions can be conceived as falling within three broad areas: performance in activities that support the mission (effectiveness), performance in relation to the resources available (efficiency), and performance in relation to long-term viability or sustainability (ongoing relevance).

Performance in Moving Towards Mission (Effectiveness)

A research organization's performance is made visible through the totality of the research (and sometimes training) activities it generates in pursuit of the mission. These outputs and effects are the most discernible aspects of organizational performance. IDRC and others who support the endeavours of institutions are naturally interested in these outputs, which are seen as the tangible results of investment dollars.

Ideas associated with the performance of research organizations in fulfilment of their missions vary considerably. Each interest group or stakeholder may have a totally different conception of what counts. For instance, scholarly researchers might define performance in terms of the number of refereed articles, whereas senior administrators might define performance as the quantity of financial resources brought into the research centre through grants. Donors might define performance in terms of the beneficial impact of findings or activities on indigenous groups.

Researchers themselves seldom speak with one voice on such matters. Is applied research as valued as theoretical research? Are agronomic practices adapted to local farmer's needs more or less valued than high yield export technologies? Are publications valuable in themselves, or should they only be considered in relation to citation indices by other researchers?

Although few organizations have performance data readily available about their research and training programs and services, it is not difficult to develop mechanisms and approaches for gathering performance data about these outputs. The information used by organizations can take the form of input data (e.g. the number of people or students served), process data (e.g. the number of research projects in progress), output data (e.g. the number of articles accepted for publication), or impact data (e.g. the number of patients impacted by the application of a particular medical technique).

While it is relatively easy to develop an information system to help institutions assess their performance, it is far more difficult to obtain consensus on the merits of particular performance indicators. It is more difficult yet to arrive at value judgments regarding acceptable levels of quantity and quality for each performance indicator. At issue is, how does the specific institution define "good" performance, and, perhaps most fundamental, does good performance move the organization towards attaining its mission?

Exhibit 6.1: Typical indicators of performance in research institutions: effectiveness.

Effectiveness number of publications accepted by refereed journals number of citations number of patents and other intellectual property software developed collaborative links with other researchers external funds/contracts received number of requests for information/participation related to national or regional development initiatives interest/recognition of research results by other institutions demands for input to government policies number of people served (for action research) health, educational benefits peer ratings of relevance of research conferences attended in which papers/posters were presented client satisfaction social/economic effects (as per mandate) number of students supervised number of trainee researchers supervised origin of students and trainees (country, institution) links with higher education institutions number of publications in which students are coauthors students'/trainees' assessments of training environment

Performance in Relation to Efficiency

In today's economy, research institutions must not only be able to provide exceptional research and teaching services, but they must also be able to provide them within an appropriate cost structure. Tight times have meant that performance is increasingly judged by the efficiency of the organization, e.g. the cost per service, the number of outputs per researcher, publications per person per year, average value of grants per person. Whatever the overall size of the unit, performing organizations are viewed as those which provide good value for the dollars expended.

Exhibit 6.2: Typical indicators of performance in research institutions: efficiency.
Efficiency ratios of internal and external funding comparative organizational costs for research, training, and other services overhead/program cost ratio number of outputs per researcher (publications per year, average value of grants per person) costs per client served costs per publication costs vs. benefits publication rates per staff

Performance in Relation to Ongoing Relevance

Institutions in any society take time to evolve and develop, but over time they must institutionalize in ways that consolidate their strengths. While all organizations ultimately face internal and external crises, the survivors are those that succeed in adapting to changing contexts and capacities. Partly because of their relatively short organizational histories, and because of widely differing environmental contexts, research organizations in the South have varied dramatically in their ability to become institutionalized in society. Moreover, no organization is protected from the vagaries of being out of date, irrelevant, and subject to closure. In this volatile context, organizational performance relates to the ability of the organization to keep its mission, goals, programs, and activities aligned with its key stakeholders and constituents. Issues of organizational survival are broad in scope, ranging from the reputation of the organization in the wider community to the effects of the organization's programs, services, and their management on staff morale.

Exhibit 6.3: Typical indicators of performance in research institutions: ongoing relevance.

Ongoing relevance relevance of work to national development relevance of work to field relevance of services to users support earmarked for professional development number of old and new financial contributors (risk of discontinuance, leverage of funding) organizational innovation and adaptiveness (appropriate changes to needs, methodologies) institutional reputation among key stakeholders number of new services and programs changes in services and programs related to changing client systems

Measurement

Four major questions permeate the performance literature and should be considered by IDRC when formulating an approach to evaluating its partner institutions:

1. What areas of performance should be measured?

In the framework implicit in this section, performance of an organization should be assessed in three domains: efficiency, effectiveness, and adaptability. Identifying appropriate performance areas in all three domains that are key in a particular organization is a crucial step for both IDRC and the partner institution at the outset of the assessment process.

Regarding the effectiveness of research output, organizational goals and priorities provide the starting point for performance measurement. Performance indicators can and should include both quantitative and qualitative measures. A matter of some concern is that certain institutions have exclusively adopted numerical measures (e.g. the number of publications in peer-reviewed journals, the number of citations per author, the amount of money received by the organization for contract research, the number of patents earned, the number of students receiving graduate and post-graduate training, and so on.) In IDRC's view, qualitative judgements by stakeholders on the impact of research production are equally vital data.

IDRC also needs data on the management of institutions — the efficiency domain. There are many approaches to such analysis, ranging from financial audits to surveys of organizational culture.

As well, there should be some analysis of the organization's ability to adapt to changing conditions. Organizational priorities, either written or inferred, transcend the individual programs and services being provided and include broad issues vital to organizational survival. The extent to which issues on this dimension will be measured is a matter for negotiation.

2. How should performance be measured?

Once "what to measure" has been decided, how to conduct the actual measurement is the next consideration. Which components within various performance areas should be measured, what kinds of data are appropriate to collect, and how should this be done? The consensus of the international evaluation community is that multiple sources of information, including a mixture of qualitative and quantitative data should be employed in order to obtain an adequate and valid understanding of performance.

Some areas are more difficult than others to measure. For instance, while productivity is relatively simple to assess using numerical data, more abstract performance concepts such as creativity or adaptiveness elude clear-cut measurement. (Their measurement is not impossible, however, as observable qualities can be delineated for both.) The costs of various measurement methodologies is another crucial consideration. Performance measures are politically sensitive and must be open to careful scrutiny. Surveys can be laborious and expensive to construct and administer.

3. When should measurement be conducted?

Timing is an important consideration in evaluating research institutions. The conduct of research is, by nature, a slow-moving, laborious process. But practical considerations often dictate short-, medium-, and long-term performance measurement strategies. These needs do not alter the fundamental character of the research endeavour, however, and this should always be taken into consideration in evaluating performance.

Historical trends, be they within the institution as a whole, in the evolution of the research group, or in the career of the individual researcher, all influence research output. When measuring the present (i.e. recent) performance of research centres, it is important to consider the historical context of performance for each of these entities. A sketch of the evolutionary progress of the organization or of individual groups within the organization or of the career of a particular researcher can be revealing. Considerations such as whether any of these is in nascent stages, or whether significant milestones have occurred in the organization or in the field, have a bearing on output; sensitivity to such contextual issues will enable more thoughtful interpretation of performance.

On a more practical level, the timing of the organizational assessment process should respect built-in organizational cycles. Assessments should not be conducted at a time of the year when grant applications are due, when staff are unavailable due to vacation season or attendance at conferences, and so forth.

4. What standards ought to apply?

Once data are obtained, issues of performance standards arise, namely, what constitutes "good" or "acceptable" research and training activities? For a fair assessment of specific research organizations, the level of acceptability for each performance indicator should be negotiated on a case-by-case basis between IDRC and the partner institutions. Abstract norms arrived at in isolation from real environmental and historical events are inappropriate to apply to any research institute.

Over time, it might be useful for IDRC and other international granting agencies to develop a data base of normative information about the performance indicators and standards that a wide variety of research centres worldwide have adopted. For example, what are reasonable expectations regarding the number and type of publications? For non-core project funding? For training researchers? Such cumulative, normative data would perhaps help future evaluators (as well as those within the research institution) make judgments. Unfortunately, the state of our knowledge presently causes us to rely on expert judgment as the primary tool for setting standards.

Conducting an Organizational Assessment

There are various reasons for conducting organizational assessments. They are conducted to ensure the organization fulfilled its terms of agreement, or to review a request to provide institutional funding. They might be conducted when there is a change in the nature of a funding agreement to multiyear or core funding, or in order to review organizational changes which might affect its eligibility for funding. They are often conducted to satisfy stipulations that an evaluation must take place.

Although the steps taken in conducting an organizational assessment may vary from case to case, there is a general sequence of activities that take place. The following is a list of the basic steps in an organizational assessment. These steps may be carried out by a team or combination of individuals that includes evaluators, the donor, the organizational members, and various stakeholders.

Exhibit 6.4: Basic steps in organizational analysis.

Determine purposes of the assessment. Develop constructive working relationship between and among the research institution and the assessment team during the assessment. Identify main issues for the assessment. Identify main questions and subquestions for the assessment. Determine roles and responsibilities for evaluators, organizational members, IDRC personnel or other donor agency, and other stakeholders. Develop and write terms of reference. Prepare costing for the assessment. Identify and select evaluators. Develop workplans. Implement workplan. Monitor quality control measures to ensure collection of reliable and valid data. Provide ongoing feedback. Draft the report. Identify general lessons learned. Communicate conclusions (i.e. debriefing sessions, written reports, workshops).

Sources of Data

Much pertinent information may already exist in the institution in one form or another, and all potential sources should be mined. Some suggestions of where to look:

• organizational documents: financial statements, annual reports, strategy documents, and so on
• bibliographic citation analyses
• reviews of research by scholarly groups
• interviews of key informants that affect organizational activities
• surveys appraising the organization's reputation
• CVs of staff

Performance as It Relates to Capacity

Performance and capacity are interrelated concepts. Organizational performance arises from the use of capacity. Assessing performance also leads us to areas where capacity needs building (the subject of Chapter 5).

It is important that organizational performance be viewed as more than the sum of organizational products. Performance should have a synergetic quality. Institutions ideally give back to society outputs whose value is greater than the total resources invested; they are organized to realize these gains, and they should be accountable for providing added value to the investments made in them.

Assessment of performance occurs informally, on an ongoing basis, whether or not the organization engages formally in performance assessments. Such assessments can be driven by various stakeholders and clients. For instance, governments may decide to increase or decrease funds to an institution in part because of perceptions of the organization's existing or potential performance. Clients can decide to use or not to use the services of a research institution because of their own assessments of its performance. Stakeholders often either implicitly or explicitly link funding to performance and perceived capacity.

By making the informal formal, IDRC, in addition to guiding its own funding strategies, is supporting the development of more transparent and open institutions. By approaching an assessment as a learning process conducted in partnership with its funded institutions, IDRC is fostering their adaptability and sustainable development.

Conclusion

This book has outlined a framework with which to assess the capacity and performance of an organization within the context of the organization's motivation and its unique environment. Experience with a wide range of research institutions worldwide suggests that understanding the environmental context is fundamental to a sympathetic analysis of how an organization operates. The environment may present difficult constraints, yet the organization may still be doing important and relevant work. Environmental analysis leads to a determination of capacity and performance relative to the context.

The organization's motivation relates in many ways to the environment, but supersedes it in the sense that many successful organizations rise above the constraints of their context. Through leadership and collective vision, such organizations are able to gather resources and produce quality research despite a nonsupportive context. Such organizations are often nourished by external funding, which makes analysis and understanding of the context and motivation essential if IDRC and other donors are to invest strategically.

Because performance is relative to an organization's basic capacity, the analysis of capacity sets the stage for understanding organizational performance. Capacity is a quantitative notion, whereas performance is both absolute and relative. Performance needs to be assessed in qualitative terms, quantitative terms, and in terms that relate performance to basic organizational capacity.

Given sufficient time and resources, external experts can do a good job of assessing organizations. Such assessments might serve IDRC's and other donor's short-term needs, but the process can be far stronger when the organization becomes a partner in the assessment and participates in the analysis. Ideally, the process contributes to the development of a learning organization capable of improving its own performance through critical self-analysis.

---

Back to Contents - To next section

This file was created 29 March 1996

Copyright International Development Research Centre. Source: http://www.idrc.ca/books/focus/771/chap6.html

Chapter 6

Organizational Performance

---

Introduction

In our framework for profiling an organization, overall performance is seen as a function of the interplay of the organization's unique motivation, its organizational capacity, and forces in the external environment.

Over the past 30 years there have been many attempts to define performance generally and to apply performance concepts to various organizational types. A number of ideas emerge from the organizational performance literature:

• In all organizations, performance relates to organizational purpose.
• Performance also needs to reflect achievements relative to the resources used by the organization.
• Performance must be considered within the environment in which the institution does its work.

The first component reflects the organization's mission, the second component reflects how well the organization manages its resources, and the third, its adaptability within the context of external forces.

Within research institutions, the quantity and quality of research produced is fundamental to the achievement of the mission. But a research institution's performance must also encompass aspects of organizational functioning that are the necessary underlying conditions for researchers to be productive.

To apply traditional assessment terminology to research organizations, organizational performance must integrate the concepts of "effectiveness" and "efficiency." That is, the organization must be able to meet its goals (effectiveness) and to do so with an acceptable outlay of resources (efficiency). Vitally important as well, particularly to IDRC and other Northern granting agencies, is the Southern organization's sustainability over the long term (ongoing relevance). The organization must be able to develop and implement strategies which will ensure research performance over extended periods of time. To do so, its activities and services must remain realistic and connected to stakeholder needs. For when an organization's services and activities are not relevant or are too far-reaching and costly, organizational survival is at risk.

In summary, the performance of institutions can be conceived as falling within three broad areas: performance in activities that support the mission (effectiveness), performance in relation to the resources available (efficiency), and performance in relation to long-term viability or sustainability (ongoing relevance).

Performance in Moving Towards Mission (Effectiveness)

A research organization's performance is made visible through the totality of the research (and sometimes training) activities it generates in pursuit of the mission. These outputs and effects are the most discernible aspects of organizational performance. IDRC and others who support the endeavours of institutions are naturally interested in these outputs, which are seen as the tangible results of investment dollars.

Ideas associated with the performance of research organizations in fulfilment of their missions vary considerably. Each interest group or stakeholder may have a totally different conception of what counts. For instance, scholarly researchers might define performance in terms of the number of refereed articles, whereas senior administrators might define performance as the quantity of financial resources brought into the research centre through grants. Donors might define performance in terms of the beneficial impact of findings or activities on indigenous groups.

Researchers themselves seldom speak with one voice on such matters. Is applied research as valued as theoretical research? Are agronomic practices adapted to local farmer's needs more or less valued than high yield export technologies? Are publications valuable in themselves, or should they only be considered in relation to citation indices by other researchers?

Although few organizations have performance data readily available about their research and training programs and services, it is not difficult to develop mechanisms and approaches for gathering performance data about these outputs. The information used by organizations can take the form of input data (e.g. the number of people or students served), process data (e.g. the number of research projects in progress), output data (e.g. the number of articles accepted for publication), or impact data (e.g. the number of patients impacted by the application of a particular medical technique).

While it is relatively easy to develop an information system to help institutions assess their performance, it is far more difficult to obtain consensus on the merits of particular performance indicators. It is more difficult yet to arrive at value judgments regarding acceptable levels of quantity and quality for each performance indicator. At issue is, how does the specific institution define "good" performance, and, perhaps most fundamental, does good performance move the organization towards attaining its mission?

Exhibit 6.1: Typical indicators of performance in research institutions: effectiveness.

Effectiveness number of publications accepted by refereed journals number of citations number of patents and other intellectual property software developed collaborative links with other researchers external funds/contracts received number of requests for information/participation related to national or regional development initiatives interest/recognition of research results by other institutions demands for input to government policies number of people served (for action research) health, educational benefits peer ratings of relevance of research conferences attended in which papers/posters were presented client satisfaction social/economic effects (as per mandate) number of students supervised number of trainee researchers supervised origin of students and trainees (country, institution) links with higher education institutions number of publications in which students are coauthors students'/trainees' assessments of training environment

Performance in Relation to Efficiency

In today's economy, research institutions must not only be able to provide exceptional research and teaching services, but they must also be able to provide them within an appropriate cost structure. Tight times have meant that performance is increasingly judged by the efficiency of the organization, e.g. the cost per service, the number of outputs per researcher, publications per person per year, average value of grants per person. Whatever the overall size of the unit, performing organizations are viewed as those which provide good value for the dollars expended.

Exhibit 6.2: Typical indicators of performance in research institutions: efficiency.
Efficiency ratios of internal and external funding comparative organizational costs for research, training, and other services overhead/program cost ratio number of outputs per researcher (publications per year, average value of grants per person) costs per client served costs per publication costs vs. benefits publication rates per staff

Performance in Relation to Ongoing Relevance

Institutions in any society take time to evolve and develop, but over time they must institutionalize in ways that consolidate their strengths. While all organizations ultimately face internal and external crises, the survivors are those that succeed in adapting to changing contexts and capacities. Partly because of their relatively short organizational histories, and because of widely differing environmental contexts, research organizations in the South have varied dramatically in their ability to become institutionalized in society. Moreover, no organization is protected from the vagaries of being out of date, irrelevant, and subject to closure. In this volatile context, organizational performance relates to the ability of the organization to keep its mission, goals, programs, and activities aligned with its key stakeholders and constituents. Issues of organizational survival are broad in scope, ranging from the reputation of the organization in the wider community to the effects of the organization's programs, services, and their management on staff morale.

Exhibit 6.3: Typical indicators of performance in research institutions: ongoing relevance.

Ongoing relevance relevance of work to national development relevance of work to field relevance of services to users support earmarked for professional development number of old and new financial contributors (risk of discontinuance, leverage of funding) organizational innovation and adaptiveness (appropriate changes to needs, methodologies) institutional reputation among key stakeholders number of new services and programs changes in services and programs related to changing client systems

Measurement

Four major questions permeate the performance literature and should be considered by IDRC when formulating an approach to evaluating its partner institutions:

1. What areas of performance should be measured?

In the framework implicit in this section, performance of an organization should be assessed in three domains: efficiency, effectiveness, and adaptability. Identifying appropriate performance areas in all three domains that are key in a particular organization is a crucial step for both IDRC and the partner institution at the outset of the assessment process.

Regarding the effectiveness of research output, organizational goals and priorities provide the starting point for performance measurement. Performance indicators can and should include both quantitative and qualitative measures. A matter of some concern is that certain institutions have exclusively adopted numerical measures (e.g. the number of publications in peer-reviewed journals, the number of citations per author, the amount of money received by the organization for contract research, the number of patents earned, the number of students receiving graduate and post-graduate training, and so on.) In IDRC's view, qualitative judgements by stakeholders on the impact of research production are equally vital data.

IDRC also needs data on the management of institutions — the efficiency domain. There are many approaches to such analysis, ranging from financial audits to surveys of organizational culture.

As well, there should be some analysis of the organization's ability to adapt to changing conditions. Organizational priorities, either written or inferred, transcend the individual programs and services being provided and include broad issues vital to organizational survival. The extent to which issues on this dimension will be measured is a matter for negotiation.

2. How should performance be measured?

Once "what to measure" has been decided, how to conduct the actual measurement is the next consideration. Which components within various performance areas should be measured, what kinds of data are appropriate to collect, and how should this be done? The consensus of the international evaluation community is that multiple sources of information, including a mixture of qualitative and quantitative data should be employed in order to obtain an adequate and valid understanding of performance.

Some areas are more difficult than others to measure. For instance, while productivity is relatively simple to assess using numerical data, more abstract performance concepts such as creativity or adaptiveness elude clear-cut measurement. (Their measurement is not impossible, however, as observable qualities can be delineated for both.) The costs of various measurement methodologies is another crucial consideration. Performance measures are politically sensitive and must be open to careful scrutiny. Surveys can be laborious and expensive to construct and administer.

3. When should measurement be conducted?

Timing is an important consideration in evaluating research institutions. The conduct of research is, by nature, a slow-moving, laborious process. But practical considerations often dictate short-, medium-, and long-term performance measurement strategies. These needs do not alter the fundamental character of the research endeavour, however, and this should always be taken into consideration in evaluating performance.

Historical trends, be they within the institution as a whole, in the evolution of the research group, or in the career of the individual researcher, all influence research output. When measuring the present (i.e. recent) performance of research centres, it is important to consider the historical context of performance for each of these entities. A sketch of the evolutionary progress of the organization or of individual groups within the organization or of the career of a particular researcher can be revealing. Considerations such as whether any of these is in nascent stages, or whether significant milestones have occurred in the organization or in the field, have a bearing on output; sensitivity to such contextual issues will enable more thoughtful interpretation of performance.

On a more practical level, the timing of the organizational assessment process should respect built-in organizational cycles. Assessments should not be conducted at a time of the year when grant applications are due, when staff are unavailable due to vacation season or attendance at conferences, and so forth.

4. What standards ought to apply?

Once data are obtained, issues of performance standards arise, namely, what constitutes "good" or "acceptable" research and training activities? For a fair assessment of specific research organizations, the level of acceptability for each performance indicator should be negotiated on a case-by-case basis between IDRC and the partner institutions. Abstract norms arrived at in isolation from real environmental and historical events are inappropriate to apply to any research institute.

Over time, it might be useful for IDRC and other international granting agencies to develop a data base of normative information about the performance indicators and standards that a wide variety of research centres worldwide have adopted. For example, what are reasonable expectations regarding the number and type of publications? For non-core project funding? For training researchers? Such cumulative, normative data would perhaps help future evaluators (as well as those within the research institution) make judgments. Unfortunately, the state of our knowledge presently causes us to rely on expert judgment as the primary tool for setting standards.

Conducting an Organizational Assessment

There are various reasons for conducting organizational assessments. They are conducted to ensure the organization fulfilled its terms of agreement, or to review a request to provide institutional funding. They might be conducted when there is a change in the nature of a funding agreement to multiyear or core funding, or in order to review organizational changes which might affect its eligibility for funding. They are often conducted to satisfy stipulations that an evaluation must take place.

Although the steps taken in conducting an organizational assessment may vary from case to case, there is a general sequence of activities that take place. The following is a list of the basic steps in an organizational assessment. These steps may be carried out by a team or combination of individuals that includes evaluators, the donor, the organizational members, and various stakeholders.

Exhibit 6.4: Basic steps in organizational analysis.

Determine purposes of the assessment. Develop constructive working relationship between and among the research institution and the assessment team during the assessment. Identify main issues for the assessment. Identify main questions and subquestions for the assessment. Determine roles and responsibilities for evaluators, organizational members, IDRC personnel or other donor agency, and other stakeholders. Develop and write terms of reference. Prepare costing for the assessment. Identify and select evaluators. Develop workplans. Implement workplan. Monitor quality control measures to ensure collection of reliable and valid data. Provide ongoing feedback. Draft the report. Identify general lessons learned. Communicate conclusions (i.e. debriefing sessions, written reports, workshops).

Sources of Data

Much pertinent information may already exist in the institution in one form or another, and all potential sources should be mined. Some suggestions of where to look:

• organizational documents: financial statements, annual reports, strategy documents, and so on
• bibliographic citation analyses
• reviews of research by scholarly groups
• interviews of key informants that affect organizational activities
• surveys appraising the organization's reputation
• CVs of staff

Performance as It Relates to Capacity

Performance and capacity are interrelated concepts. Organizational performance arises from the use of capacity. Assessing performance also leads us to areas where capacity needs building (the subject of Chapter 5).

It is important that organizational performance be viewed as more than the sum of organizational products. Performance should have a synergetic quality. Institutions ideally give back to society outputs whose value is greater than the total resources invested; they are organized to realize these gains, and they should be accountable for providing added value to the investments made in them.

Assessment of performance occurs informally, on an ongoing basis, whether or not the organization engages formally in performance assessments. Such assessments can be driven by various stakeholders and clients. For instance, governments may decide to increase or decrease funds to an institution in part because of perceptions of the organization's existing or potential performance. Clients can decide to use or not to use the services of a research institution because of their own assessments of its performance. Stakeholders often either implicitly or explicitly link funding to performance and perceived capacity.

By making the informal formal, IDRC, in addition to guiding its own funding strategies, is supporting the development of more transparent and open institutions. By approaching an assessment as a learning process conducted in partnership with its funded institutions, IDRC is fostering their adaptability and sustainable development.

Conclusion

This book has outlined a framework with which to assess the capacity and performance of an organization within the context of the organization's motivation and its unique environment. Experience with a wide range of research institutions worldwide suggests that understanding the environmental context is fundamental to a sympathetic analysis of how an organization operates. The environment may present difficult constraints, yet the organization may still be doing important and relevant work. Environmental analysis leads to a determination of capacity and performance relative to the context.

The organization's motivation relates in many ways to the environment, but supersedes it in the sense that many successful organizations rise above the constraints of their context. Through leadership and collective vision, such organizations are able to gather resources and produce quality research despite a nonsupportive context. Such organizations are often nourished by external funding, which makes analysis and understanding of the context and motivation essential if IDRC and other donors are to invest strategically.

Because performance is relative to an organization's basic capacity, the analysis of capacity sets the stage for understanding organizational performance. Capacity is a quantitative notion, whereas performance is both absolute and relative. Performance needs to be assessed in qualitative terms, quantitative terms, and in terms that relate performance to basic organizational capacity.

Given sufficient time and resources, external experts can do a good job of assessing organizations. Such assessments might serve IDRC's and other donor's short-term needs, but the process can be far stronger when the organization becomes a partner in the assessment and participates in the analysis. Ideally, the process contributes to the development of a learning organization capable of improving its own performance through critical self-analysis.

---

Back to Contents - To next section

This file was created 29 March 1996

Copyright International Development Research Centre. Source: http://www.idrc.ca/books/focus/771/chap6.html

Chapter 6

Organizational Performance

---

Introduction

In our framework for profiling an organization, overall performance is seen as a function of the interplay of the organization's unique motivation, its organizational capacity, and forces in the external environment.

Over the past 30 years there have been many attempts to define performance generally and to apply performance concepts to various organizational types. A number of ideas emerge from the organizational performance literature:

• In all organizations, performance relates to organizational purpose.
• Performance also needs to reflect achievements relative to the resources used by the organization.
• Performance must be considered within the environment in which the institution does its work.

The first component reflects the organization's mission, the second component reflects how well the organization manages its resources, and the third, its adaptability within the context of external forces.

Within research institutions, the quantity and quality of research produced is fundamental to the achievement of the mission. But a research institution's performance must also encompass aspects of organizational functioning that are the necessary underlying conditions for researchers to be productive.

To apply traditional assessment terminology to research organizations, organizational performance must integrate the concepts of "effectiveness" and "efficiency." That is, the organization must be able to meet its goals (effectiveness) and to do so with an acceptable outlay of resources (efficiency). Vitally important as well, particularly to IDRC and other Northern granting agencies, is the Southern organization's sustainability over the long term (ongoing relevance). The organization must be able to develop and implement strategies which will ensure research performance over extended periods of time. To do so, its activities and services must remain realistic and connected to stakeholder needs. For when an organization's services and activities are not relevant or are too far-reaching and costly, organizational survival is at risk.

In summary, the performance of institutions can be conceived as falling within three broad areas: performance in activities that support the mission (effectiveness), performance in relation to the resources available (efficiency), and performance in relation to long-term viability or sustainability (ongoing relevance).

Performance in Moving Towards Mission (Effectiveness)

A research organization's performance is made visible through the totality of the research (and sometimes training) activities it generates in pursuit of the mission. These outputs and effects are the most discernible aspects of organizational performance. IDRC and others who support the endeavours of institutions are naturally interested in these outputs, which are seen as the tangible results of investment dollars.

Ideas associated with the performance of research organizations in fulfilment of their missions vary considerably. Each interest group or stakeholder may have a totally different conception of what counts. For instance, scholarly researchers might define performance in terms of the number of refereed articles, whereas senior administrators might define performance as the quantity of financial resources brought into the research centre through grants. Donors might define performance in terms of the beneficial impact of findings or activities on indigenous groups.

Researchers themselves seldom speak with one voice on such matters. Is applied research as valued as theoretical research? Are agronomic practices adapted to local farmer's needs more or less valued than high yield export technologies? Are publications valuable in themselves, or should they only be considered in relation to citation indices by other researchers?

Although few organizations have performance data readily available about their research and training programs and services, it is not difficult to develop mechanisms and approaches for gathering performance data about these outputs. The information used by organizations can take the form of input data (e.g. the number of people or students served), process data (e.g. the number of research projects in progress), output data (e.g. the number of articles accepted for publication), or impact data (e.g. the number of patients impacted by the application of a particular medical technique).

While it is relatively easy to develop an information system to help institutions assess their performance, it is far more difficult to obtain consensus on the merits of particular performance indicators. It is more difficult yet to arrive at value judgments regarding acceptable levels of quantity and quality for each performance indicator. At issue is, how does the specific institution define "good" performance, and, perhaps most fundamental, does good performance move the organization towards attaining its mission?

Exhibit 6.1: Typical indicators of performance in research institutions: effectiveness.

Effectiveness number of publications accepted by refereed journals number of citations number of patents and other intellectual property software developed collaborative links with other researchers external funds/contracts received number of requests for information/participation related to national or regional development initiatives interest/recognition of research results by other institutions demands for input to government policies number of people served (for action research) health, educational benefits peer ratings of relevance of research conferences attended in which papers/posters were presented client satisfaction social/economic effects (as per mandate) number of students supervised number of trainee researchers supervised origin of students and trainees (country, institution) links with higher education institutions number of publications in which students are coauthors students'/trainees' assessments of training environment

Performance in Relation to Efficiency

In today's economy, research institutions must not only be able to provide exceptional research and teaching services, but they must also be able to provide them within an appropriate cost structure. Tight times have meant that performance is increasingly judged by the efficiency of the organization, e.g. the cost per service, the number of outputs per researcher, publications per person per year, average value of grants per person. Whatever the overall size of the unit, performing organizations are viewed as those which provide good value for the dollars expended.

Exhibit 6.2: Typical indicators of performance in research institutions: efficiency.
Efficiency ratios of internal and external funding comparative organizational costs for research, training, and other services overhead/program cost ratio number of outputs per researcher (publications per year, average value of grants per person) costs per client served costs per publication costs vs. benefits publication rates per staff

Performance in Relation to Ongoing Relevance

Institutions in any society take time to evolve and develop, but over time they must institutionalize in ways that consolidate their strengths. While all organizations ultimately face internal and external crises, the survivors are those that succeed in adapting to changing contexts and capacities. Partly because of their relatively short organizational histories, and because of widely differing environmental contexts, research organizations in the South have varied dramatically in their ability to become institutionalized in society. Moreover, no organization is protected from the vagaries of being out of date, irrelevant, and subject to closure. In this volatile context, organizational performance relates to the ability of the organization to keep its mission, goals, programs, and activities aligned with its key stakeholders and constituents. Issues of organizational survival are broad in scope, ranging from the reputation of the organization in the wider community to the effects of the organization's programs, services, and their management on staff morale.

Exhibit 6.3: Typical indicators of performance in research institutions: ongoing relevance.

Ongoing relevance relevance of work to national development relevance of work to field relevance of services to users support earmarked for professional development number of old and new financial contributors (risk of discontinuance, leverage of funding) organizational innovation and adaptiveness (appropriate changes to needs, methodologies) institutional reputation among key stakeholders number of new services and programs changes in services and programs related to changing client systems

Measurement

Four major questions permeate the performance literature and should be considered by IDRC when formulating an approach to evaluating its partner institutions:

1. What areas of performance should be measured?

In the framework implicit in this section, performance of an organization should be assessed in three domains: efficiency, effectiveness, and adaptability. Identifying appropriate performance areas in all three domains that are key in a particular organization is a crucial step for both IDRC and the partner institution at the outset of the assessment process.

Regarding the effectiveness of research output, organizational goals and priorities provide the starting point for performance measurement. Performance indicators can and should include both quantitative and qualitative measures. A matter of some concern is that certain institutions have exclusively adopted numerical measures (e.g. the number of publications in peer-reviewed journals, the number of citations per author, the amount of money received by the organization for contract research, the number of patents earned, the number of students receiving graduate and post-graduate training, and so on.) In IDRC's view, qualitative judgements by stakeholders on the impact of research production are equally vital data.

IDRC also needs data on the management of institutions — the efficiency domain. There are many approaches to such analysis, ranging from financial audits to surveys of organizational culture.

As well, there should be some analysis of the organization's ability to adapt to changing conditions. Organizational priorities, either written or inferred, transcend the individual programs and services being provided and include broad issues vital to organizational survival. The extent to which issues on this dimension will be measured is a matter for negotiation.

2. How should performance be measured?

Once "what to measure" has been decided, how to conduct the actual measurement is the next consideration. Which components within various performance areas should be measured, what kinds of data are appropriate to collect, and how should this be done? The consensus of the international evaluation community is that multiple sources of information, including a mixture of qualitative and quantitative data should be employed in order to obtain an adequate and valid understanding of performance.

Some areas are more difficult than others to measure. For instance, while productivity is relatively simple to assess using numerical data, more abstract performance concepts such as creativity or adaptiveness elude clear-cut measurement. (Their measurement is not impossible, however, as observable qualities can be delineated for both.) The costs of various measurement methodologies is another crucial consideration. Performance measures are politically sensitive and must be open to careful scrutiny. Surveys can be laborious and expensive to construct and administer.

3. When should measurement be conducted?

Timing is an important consideration in evaluating research institutions. The conduct of research is, by nature, a slow-moving, laborious process. But practical considerations often dictate short-, medium-, and long-term performance measurement strategies. These needs do not alter the fundamental character of the research endeavour, however, and this should always be taken into consideration in evaluating performance.

Historical trends, be they within the institution as a whole, in the evolution of the research group, or in the career of the individual researcher, all influence research output. When measuring the present (i.e. recent) performance of research centres, it is important to consider the historical context of performance for each of these entities. A sketch of the evolutionary progress of the organization or of individual groups within the organization or of the career of a particular researcher can be revealing. Considerations such as whether any of these is in nascent stages, or whether significant milestones have occurred in the organization or in the field, have a bearing on output; sensitivity to such contextual issues will enable more thoughtful interpretation of performance.

On a more practical level, the timing of the organizational assessment process should respect built-in organizational cycles. Assessments should not be conducted at a time of the year when grant applications are due, when staff are unavailable due to vacation season or attendance at conferences, and so forth.

4. What standards ought to apply?

Once data are obtained, issues of performance standards arise, namely, what constitutes "good" or "acceptable" research and training activities? For a fair assessment of specific research organizations, the level of acceptability for each performance indicator should be negotiated on a case-by-case basis between IDRC and the partner institutions. Abstract norms arrived at in isolation from real environmental and historical events are inappropriate to apply to any research institute.

Over time, it might be useful for IDRC and other international granting agencies to develop a data base of normative information about the performance indicators and standards that a wide variety of research centres worldwide have adopted. For example, what are reasonable expectations regarding the number and type of publications? For non-core project funding? For training researchers? Such cumulative, normative data would perhaps help future evaluators (as well as those within the research institution) make judgments. Unfortunately, the state of our knowledge presently causes us to rely on expert judgment as the primary tool for setting standards.

Conducting an Organizational Assessment

There are various reasons for conducting organizational assessments. They are conducted to ensure the organization fulfilled its terms of agreement, or to review a request to provide institutional funding. They might be conducted when there is a change in the nature of a funding agreement to multiyear or core funding, or in order to review organizational changes which might affect its eligibility for funding. They are often conducted to satisfy stipulations that an evaluation must take place.

Although the steps taken in conducting an organizational assessment may vary from case to case, there is a general sequence of activities that take place. The following is a list of the basic steps in an organizational assessment. These steps may be carried out by a team or combination of individuals that includes evaluators, the donor, the organizational members, and various stakeholders.

Exhibit 6.4: Basic steps in organizational analysis.

Determine purposes of the assessment. Develop constructive working relationship between and among the research institution and the assessment team during the assessment. Identify main issues for the assessment. Identify main questions and subquestions for the assessment. Determine roles and responsibilities for evaluators, organizational members, IDRC personnel or other donor agency, and other stakeholders. Develop and write terms of reference. Prepare costing for the assessment. Identify and select evaluators. Develop workplans. Implement workplan. Monitor quality control measures to ensure collection of reliable and valid data. Provide ongoing feedback. Draft the report. Identify general lessons learned. Communicate conclusions (i.e. debriefing sessions, written reports, workshops).

Sources of Data

Much pertinent information may already exist in the institution in one form or another, and all potential sources should be mined. Some suggestions of where to look:

• organizational documents: financial statements, annual reports, strategy documents, and so on
• bibliographic citation analyses
• reviews of research by scholarly groups
• interviews of key informants that affect organizational activities
• surveys appraising the organization's reputation
• CVs of staff

Performance as It Relates to Capacity

Performance and capacity are interrelated concepts. Organizational performance arises from the use of capacity. Assessing performance also leads us to areas where capacity needs building (the subject of Chapter 5).

It is important that organizational performance be viewed as more than the sum of organizational products. Performance should have a synergetic quality. Institutions ideally give back to society outputs whose value is greater than the total resources invested; they are organized to realize these gains, and they should be accountable for providing added value to the investments made in them.

Assessment of performance occurs informally, on an ongoing basis, whether or not the organization engages formally in performance assessments. Such assessments can be driven by various stakeholders and clients. For instance, governments may decide to increase or decrease funds to an institution in part because of perceptions of the organization's existing or potential performance. Clients can decide to use or not to use the services of a research institution because of their own assessments of its performance. Stakeholders often either implicitly or explicitly link funding to performance and perceived capacity.

By making the informal formal, IDRC, in addition to guiding its own funding strategies, is supporting the development of more transparent and open institutions. By approaching an assessment as a learning process conducted in partnership with its funded institutions, IDRC is fostering their adaptability and sustainable development.

Conclusion

This book has outlined a framework with which to assess the capacity and performance of an organization within the context of the organization's motivation and its unique environment. Experience with a wide range of research institutions worldwide suggests that understanding the environmental context is fundamental to a sympathetic analysis of how an organization operates. The environment may present difficult constraints, yet the organization may still be doing important and relevant work. Environmental analysis leads to a determination of capacity and performance relative to the context.

The organization's motivation relates in many ways to the environment, but supersedes it in the sense that many successful organizations rise above the constraints of their context. Through leadership and collective vision, such organizations are able to gather resources and produce quality research despite a nonsupportive context. Such organizations are often nourished by external funding, which makes analysis and understanding of the context and motivation essential if IDRC and other donors are to invest strategically.

Because performance is relative to an organization's basic capacity, the analysis of capacity sets the stage for understanding organizational performance. Capacity is a quantitative notion, whereas performance is both absolute and relative. Performance needs to be assessed in qualitative terms, quantitative terms, and in terms that relate performance to basic organizational capacity.

Given sufficient time and resources, external experts can do a good job of assessing organizations. Such assessments might serve IDRC's and other donor's short-term needs, but the process can be far stronger when the organization becomes a partner in the assessment and participates in the analysis. Ideally, the process contributes to the development of a learning organization capable of improving its own performance through critical self-analysis.

---