Scott Bayley, member of ECDG’s Advisory Group, shares a summary paper he wrote recently for staff in DFAT’s aid program.
Types of Evaluation Use
Early studies on the impact of evaluations were based on directly observable effects, such as a policy change or the introduction of a new program initiative. This form of utilisation is defined as instrumental use and refers to situations where an evaluation directly affects decision-making and influences changes in the program. Evidence for this type of utilisation involves decisions and actions that directly arise from the evaluation, including the implementation of recommendations.
The second type is conceptual use which is more indirect and relates to ‘enlightenment’ or generating knowledge and understanding of a given area. Conceptual use refers to “the use of evaluations to influence thinking about issues in a general way.” Conceptual use occurs when an evaluation influences the way in which stakeholders think about a program, without any immediate new decisions being made about the program. Over time and given changes to the contextual and political circumstances surrounding the program, conceptual impacts can lead to instrumental impacts and hence significant program changes.
Political use involves the legitimising of decisions already made about a program. For example, an evaluation is commissioned with no intention of utilising the evaluation findings, but rather as a strategy to defer a decision. Alternatively, the evaluation follows after decision-making and provides a mechanism for retrospectively justifying decisions made on other grounds.
Symbolic use occurs when an organisation establishes an evaluation unit or undertakes an evaluation study to signal that they are good managers. The actual functions of the evaluation unit or the evaluation’s findings are of limited importance aside from their symbolic public relations value.
Finally, process use concerns how individuals and organizations are impacted upon as a result of participating in an evaluation. Being involved in an evaluation may lead to changes in the thoughts and behaviours of individuals which then results in cultural and organizational change. An example of process use is when those involved in the evaluation later say, “The impact on our program came not so much from the findings as from going through the thinking process that the evaluation required”.
High Quality Evaluation Recommendations
Questions to consider:
- The evaluation’s purpose: formative / summative / pure accountability?
- Consider the program’s stage of development: pilot; new program; long term program.
- Normative criteria to inform the development of the recommendations? (where will these norms come from?)
- Type of use to be promoted?
- Are recommendations required at all?
- Whose recommendations are these?
- Who will participate in their development?
- How much improvement is required? How soon does this need to happen? What resources are available to support this? Is additional data collection required to form the recommendations?
- What type of recommendations are required? Operational directives? Options for action? Setting strategic priorities? (consider political support for the program x adequacy of its ToC x any implementation problems x changing external environment).
Given the evaluation’s broad context & purpose:
- The recommendations should be aligned to the level and content of the evaluation question and related conclusions.
- The recommendations are based on evidence, transparent logic, appropriate consultation and are:
- Technically sound
- Economically / financially feasible
- Legislatively authorised
- Politically practicable
- Administratively feasible i.e. suitable administrative capacity
- Consistent with other related policies
- Absence of negative side effects.
- Roles and responsibilities for implementation are clear.
Program staff are generally the people with expert operational knowledge of their program. Ideally the external evaluator offers a fresh perspective and a broad knowledge of programs of this type.
Principal Specialist, Performance Management & Results
Department of Foreign Affairs and Trade (DFAT)