Module 5 - Questions and Criteria for Evaluations

1. Module 5 - Questions and Criteriafor Evaluations

2. Aligning Questions and Methods What needs to be evaluated Methods and techniques Questions and criteria Quality

3. Role of Questions in Evaluation • Expression of a concern in its simplest form, a need felt by the sponsor (future user?) • Asked in common language • Best way for donors and evaluators to understand each other and agree • Questions are refined using interpersonal communication tools (listening, rewording, etc.)

4. Checking the Feasibility of Evaluation Questions • Specific: is there a topic and a stem for each question? • Measurable: can the question be answered? • Actionable: will the answers provide the information needed for decision-making? • Relevant: “nice to know” or “need to know”? • Timely: is it important to ask this question now?

5. What Can an Evaluation Tell Us? • Success • Are we achieving the results that we were hoping for? • Are results being produced as planned at all levels of our logic model? • Relevance • Do stakeholders care about what we are doing? Are we solving the problem? • Cost-Effectiveness - Are we getting value for money?

6. When Formulating Questions • Ensure a clear link between questions and the issues of the evaluation • Be realistic in the number of questions asked • Choose questions selectively • Ensure questions can be answered • Focus on the important issues

7. There are only 3 types of questions: Sorting Questions Properly … that require the use of different methods:

8. Questions and Techniques

9. Goals, Purposes Main Objective Direct Results Activities Means Evaluation Criteria Impact Relevance Effectiveness Sustainability Replicability Coherence Efficiency Logical structure

10. The Criteria What is their use ? • The ‘politically correct’ side of the evaluation, based on the DAC’ standards • In discussions/reflections on evaluation questions, criteria help ensure nothing is forgotten and helps sort out concerns • A ‘total’ evaluation (on all criteria) reinforces the logical aspect of the findings, and therefore makes them sounder.

11. Relevance Was the program well suited or adapted to reach the overall goal? Is the goal still relevant today? Does the program still meet the expectations of the stakeholders? Coherence In the program’s original design, were the objectives coherent (and are they still) with each other, did the planned activities make it possible to reach the operational objectives that were set, did the planned means make it possible to undertake the planned activities within the allocated time? Criteria and Questions: Correspondence

12. Efficiency How did the project or the policy work? Were the resources properly mobilized? Were the unit costs within the standard range? Were the expected immediate results obtained and under what conditions in terms of quality, costs and time frame? Were the results satisfactory given the amounts of money invested? Were the contractual demands met? Criteria and Questions: Correspondence

13. To what extent has the key objective of the project been reached? Has the program had the expected results from the point of view of the beneficiaries (addressing unbalances, reinforcing institutions, changing behaviours, etc.)? Effectiveness Impact Did the program help address the targeted problem (poverty level, for example) or produce the expected development? Did it have unforeseen negative impacts? Are its overall consequences positive? Criteria and Questions: Correspondence

14. Will the effects of the project or program be sustainable? Are the mechanisms put in place viable? Sustainability Replicability Can the policy be duplicated ? What would it take? Criteria and Questions: Correspondence

15. Summary To ensure that all the questions have a concrete content, it is necessary to go back to the original objectives and indicators (project document and logical framework). When there is no logical framework, the evaluator must reconstruct it.