Carrying out a Monitoring and Evaluation Diagnostic Study Geoffrey Shepherd April 28, 2011

1. Carrying out a Monitoring and Evaluation Diagnostic Study Geoffrey ShepherdApril 28, 2011

2. Introduction: the role and organization of M&E • M&E provides information allowing us to judge the efficiency and effectiveness of public actions and their consistency with other public actions. • An M&E system is a set of linked activities to carry out the regular, systematic, and planned monitoring and evaluation of government activities. • M&E is a building block of a performance-oriented (or development-oriented) policy cycle.

3. What is an M&E diagnosis? • An M&E diagnosis is an analysis of what is and is not working in a country’s (or a ministry’s, or a sub-national government’s) M&E activities, with recommendations for improving activities and systems. • The diagnosis primarily involves: • Institutional analysis: what are the rules and practices that drive demand for M&E? • Analysis of technical capacity: what are the data-collection practices, analytical instruments, and skills that supply the M&E?

4. Constructing a checklist for a diagnosis • Different country situations (levels of development, institutional attributes, etc.) mean that there is no single blueprint for a “good” M&E system or for preparing a diagnostic study. • On the other hand, different studies of national M&E systems cover similar topics. • I have derived a checklist of topics from: • existing diagnostic guides; and • a review of recent M&E cases studies and diagnoses. • I shall illustrate some of the main practical issues under each topic with brief reference to country cases.

5. A checklist: nine topics for preparing an M&E diagnosis

6. Topic A.1. The national policy and institutional framework

7. Topic B.1. M&E systems: historical development

8. Topic B.2. M&E systems: objectives (announced, implicit, or revealed)

9. Topic B.3. M&E systems: processes, tools, and products

10. Topic B.4. M&E systems: relationship with other systems

11. Topic B.5. M&E systems: institutional architecture

12. Topic B.6. M&E systems: characteristics of organizations that are part of the system

13. A recapitulation en route: where are incentivesin the checklist? The major policy frameworks that drive M&E - Incentives to provide good information. - Incentives to use findings • Organizational incentives (usually differing between agencies): • to perform, • to demand M&E, • to cooperate with other organizations

14. Topic B.7. M&E systems: results

15. Topic C.1. Conclusions and recommendations

16. Organizing the diagnosis: some modest advice • An M&E diagnosis will not make sense in every country: there must be a policy framework that drives M&E. • Involve the government (c.f. Topic B.5 on institutional architecture – incentives to provide good information and to use the findings). • A diagnosis can be carried out at different levels – it can range from good journalism (with limited resources) to a research project (with greater resources). In any case, you cannot cover all topics in equal depth. • Prepare as much as you can before you go into the field – it’s amazing what you can find on the Internet. • Do not duck the challenge of making some judgment of results (Topic B.7) – with whatever caveats are necessary: • interviews and focus groups involving a diversity of actors will help. • The checklist is better at helping you identify the right information to collect than it is at indicating how to tell the story: • often, a historical narrative works best, but sometimes the need to compare systems may dictate a more instrumental approach.

17. Thank you Geoffrey Shepherd geoffreyshepherd@msn.com