Monitoring and Evaluation for ELI: An Overview

1. ELI Monitoring & Evaluation Monitoring and Evaluation for ELI: An Overview ELI M&E Overview - July 2000 Workshop

2. Introductions • Carol Mulholland, Academy for Educational Development (AED); IFC M&E Evaluation Manager • Luisa Freeman, Applied Energy Group (AEG),Technical Project Manager • Joseph Lopes; AEG, • Project Officer ELI M&E Overview - July 2000 Workshop

3. Overview 1. Materials in your Notebook 2. Evaluation Goals and Methods 3. Market Transformation Goals and Methods 4. Your Role as Implementers 5. Schedule/Sequence of Events 6. Our Team - Your Contacts for M&E 7. Tools: RBM, Monthly Reports, Guidebook ELI M&E Overview - July 2000 Workshop

4. Notebook Contents • Workshop Agenda • Overview slides • Results Based Management (RBM) Primer Slides and Work Sheets • ELI M&E Work Plan • Practical Guide from APPA • Monthly Program Statistics Report ELI M&E Overview - July 2000 Workshop

5. AGENDA ELI M&E Overview - July 2000 Workshop

6. The Mission of M&E • To accurately and comprehensively document the processes and outcomes of ELI • To draw reasonable and defensible conclusions • To make useful and relevant recommendations for future efforts of this nature ELI M&E Overview - July 2000 Workshop

7. The ELI Program ELI M&E Overview - July 2000 Workshop

8. Why Evaluate? M&E is necessary for: 1. Effective project management 2. Increasing probability of success by providing feedback 3. Documenting results for future efforts 4. Demonstrating accountability for investments made ELI M&E Overview - July 2000 Workshop

9. Process Evaluation Defined: A review of the processes by which programs are implemented, carried out and accepted by all stakeholders. Goal: To identify program improvements. Methods: Both qualitative and quantitative market research methods. ELI M&E Overview - July 2000 Workshop

10. Impact Evaluation Defined: Quantification of the measurable outcomes of a program Measurement: Participation levels, cost effectiveness and resultant effects (direct and indirect) of the program’s outcomes on energy consumption and greenhouse gas emissions. Methods: Quantitative statistical methods ELI M&E Overview - July 2000 Workshop

11. Market Transformation Assessment Defined: Determination of the extent to which the overall market for the target products has changed nationally, regionally and globally as a result of program activities and interventions. Methods: Macroeconomic, quantitative and qualitative methods may be used. ELI M&E Overview - July 2000 Workshop

12. Evaluation Methods • Managerial Review - of documents, assignments, and resources • Qualitative Market Research - focus groups, interviews, small surveys • Quantitative Market Research - surveys, statistical methods • Secondary Research - collection of data from manufacturers and other sources ELI M&E Overview - July 2000 Workshop

13. Our Tasks ELI M&E Overview - July 2000 Workshop

14. The M&E Organization ELI M&E Overview - July 2000 Workshop

15. M&E Country Teams ELI M&E Overview - July 2000 Workshop

16. Your Roles (near term) 1. Complete Market Assessment work. - Used by M&E to develop a baseline characterization of the lighting market from which to compare changes. 2. Develop performance goals and measurement criteria for each program (RBM is a tool for this.) - We will then compare progress against stated goals. 3. Provide monthly updates of progress - Fill out and send in Monthly Program Statistics Form. 4. Meet with local M&E representatives - You will be contacted by our local reps regularly to discuss program changes, updates, issues and needs. Use them to help! ELI M&E Overview - July 2000 Workshop

17. Your Roles (later) 5. Provide contact lists for market research - After one year, we may request participant lists for survey or focus group recruiting purposes. 6. Assist with installation of light loggers - We will be installing recording equipment on samples of homes in each country and may ask for your assistance in identifying candidates. 7. Review and use our findings - We will do our best to provide timely useful feedback, and want your comments and ideas in return. ELI M&E Overview - July 2000 Workshop

18. The Most Important Role • Provide honest feedback at all times: • We can only provide useful ideas if we know what is going on in the field. • Please tell us what issues, needs and barriers to success you are experiencing • We know how to manage sensitive information. Your information will be held in confidence and used only to help make the programs better. ELI M&E Overview - July 2000 Workshop

19. Market Research ELI M&E Overview - July 2000 Workshop

20. Evaluation Baseline ELI M&E Overview - July 2000 Workshop

21. How Can We Help? • To provide assistance regarding data tracking, goal setting and measurement • To help identify improvements during the Interim Process Evaluation phase • To observe and provide periodic feedback and communicate sensitive issues • To report achievements: impacts and market transformation results ELI M&E Overview - July 2000 Workshop

22. M&E Communication • Your in-country M&E Representative contacts you every month • You provide monthly statistical report via ELI M&E web site • Your M&E Country Manager is available to clarify and answer questions ELI M&E Overview - July 2000 Workshop

23. Notes _________________________ _______________________________ _______________________________ _______________________________ _______________________________ ELI M&E Overview - July 2000 Workshop

24. M&E Reporting • Evaluation is designed to measure success of the Program • Tracking is needed - Routinely performed during implementation • Monthly Statistics reported via Web • Time Series links events to results • ELI Marketing, non-ELI events ELI M&E Overview - July 2000 Workshop

25. Mister ELI Evaluator ELI M&E Overview - July 2000 Workshop