The impact oriented M&E- system of the German-costarican program on waste management

1. Theimpactoriented M&E-system of theGerman-costaricanprogramonwastemanagement Dr. Stefanie Krapp EvaluationUnit, GermanTechnicalCooperation (GTZ) AnnualConference of the Canadian EvaluationSociety Victoria, Canada May 4th, 2010

2. Contents • Evaluationsystem of GTZ • Thewastemanagementprogram in Costa Rica • Theprogram M&E-System • Evaluationdesign • Someselectedresults • Conclusions

3. Managing for development results Planning a development measure Sustainability Implementation Completion Results Results-based monitoring Evaluations e.g. ex-ante evaluation e.g. project progress review e.g. final evaluation e.g. ex-post evaluation

4. Evaluation system GTZ Evaluationsystem GTZ instruments Self-evaluation Steering by operative units Independent evaluation Steering by Evaluation Unit Decentralised evaluation Steering by operative units External evaluation Evaluation by the Ministry of Economic Cooperation and Development (BMZ) Project progress review (PPR) e-VAL End of project evaluation e-VAL Mostly interim evaluation Evaluation by auditorson behalf of BMZ Ex-post evaluation 2-5 years after completion of project Continuous results-based monitoring in every development measure is the prerequisite for all evaluations.

5. German-costaricanprogram “Competition and Environment” Goal: Improvement of waste management in municipalities and the industry in selected regions of Costa Rica (04/2005 – 03/2013) Component 1: Comunication and dialogue Component 2: Strategies, plans and legal frame Component 3: Waste management in municipalities Component 4: Competition and environment friendly behavior of the industrie Relevant stakeholders accomplish the new legal frame conditions and political development strategies in waste management. Improved abilities of selected municipalities to carry out an integrated waste management. Industry, services and trade implement production methods and measures for resource recuperation. Public and private sector and the civil society work constructively together in the waste management.

6. German-costaricanprogram “Competition and Environment” • Intended Impacts • Diffusion: The implementedstructuresandprocesses in selectedmunicipalitiesaretakenoverbyothermunicipalitiesandcompanieswhichleadtotheimprovementofwastemanagement in otherregionsandsectors. • Positive imageof Costa Rica as an ecotourismcountry • Improvementofhealth • Costa Rica asknowledgecarrier in wastemanagementand innovative environmental technologies • Growing environmental awareness • Growingresourceefficiency • ….

7. M&E System Theimpactoriented M&E-System of theprogram CYMA Performance Framework: Result chains for each component with indicators on each level (output, use of output, outcomes, impacts) Internal Monitoring: Baselines & continuous performance self-assessment with data collection, time series analysis, household survey, foto documentation (output, use of output, outcome) Internal and external Eval.: a) Project Progress Reviews b) Specific component related topics c) Anual panel according to outcome and impact indicators ¡Participatively developed and implemented! ¡Linked with the program management system!

8. InternalMonitoring Time seriesanalysis Ex. 1: Development ofthequotaofcorrectlydeposited, compostedorrecycledwaste in themunicipalities Before After t2 t1 t3 t5 t4 75% 70% 78% 90% 85% Intervention:Implementation of waste management plans in the municipalities Ex. 2: Development of the quota of households connected to the waste management system in the municipalities

9. InternalMonitoring C3 • Program group (P) = 8 pilot municipalities • Counterfactual: Comparison group by matching procedure (C) = 3 municipalities not related to the program • Baseline and continuous collection of data according to relevant indicators of P and C • Standardised telephone survey of municipal households about the waste situation in the municipalities (P and C) • Randomwalk: Photographical documentation of the „visual“ waste situation in the municipalities (P and C)

10. ExternalEvaluation Panel evaluation design Impact model: Capacity of the counterparts, Outcome and Impact Evaluation frame: indicators of the impact model Longitudinal study, incl. expert assessment, guided interviews

11. ImpactModel Impacts Improved waste management in municipalities and industries in selected regions of CR. Counterparts Outcome Partner structure Organisational structure CYMA Goal system Staff Component 1 Component 4 Communication Fin. Resources Component 3 Component 2

12. ExternalEvaluation: Panel Before After Panel groups • The same variables • with the same operationalisation (guided interviews) • are measured with the same people • at different points of time. Baseline 2nd survey 3rd survey 4th survey 2006 2007 2008 2010

13. Panel groups (2006, 2007, 2008, 2010) 3 Ministries (Planning, Health, Environment), Chamber of Industries, IFAM, 4 pilot municipalities, program respresentatives Experts Municipalities, Tourism, university professors, Journalists Stakeholders Counter- parts Gtz-Program PRODELO, 4 municipalities, chambers, ministeries (Economy, Agriculture), NGOs, Institutes (CNP+L, CEGESTI, CICAP), Associations (ACEPESA, ACIPLAST, FEMETROM, FEDEMUR, CONADECO), local respresentatives (PRU-GAM, UNGL), political respresentatives, companies (Holcim, Dos Pinos, Prolusa, Sur Química, Gente Reciclando, Fundación Escazú Recicla)

14. Sumary • Monitoringdata: • Baseline • Time series • Standardisedtelephonehouseholdsurvey • Foto documentationofthewastesituation in municipalities • Double-Difference Analysis: • Comparisonofdatacollectedat t3 withdatacollectedat t1 (baselinedata) and t2 oftheprogramandcomparisongroup • Data ofthe Panel survey • Comparisonofdatacollectedat t4 withdatacollectedat t1 (baseline), t2 and t3 • comparisonwithmonitoringresultsat different pointsof time • Determinetheattributionofprogramimpacts

15. Selectedresults • Law on waste management almost ratified, National Plan of Waste Management, Municipal Waste Management Plans • Diffusion of resource gentle and environment friendly production in the private industry • Increased separated disposal of waste • Intensified interaction in the waste sector and therefore continuing presence of the topic in the public • Appreciation of the political role of the chamber of industries • Costa Rica as knowledge carrier • BUT: still critical pollution (mainly oncerning water, waste disposal and recycling)

16. Conclusions • Main functionsandgoalsofthe M&E-System • Joint learning • Information on howtheprogramisdoing • Steeringinformationfortheprogrammanagement • Evidenceofprogramresults (outcomesandimpacts) • Basis forthedevelopmentof a national wastemanagementmonitoringsystem in Costa Rica • Main conclusion: • The M&E systemis not perfect but triestoshowhowtheprogramisdoingandgives valid information on outcomesandhints on impactsunderthegivencircumstancesbyapplying different methodsas „rigorous“ asposible!

17. Conclusions: Prerequisites • logical result chains and useful indicators • flexible system that allows adaptations • baselines • continuous data collection • good data basis in the municipalities • participation of counterparts in the development and implementation of the M&E-system (ownership) • willing program manager • M&E system connected with the management system • staff responsible for the monitoring system • combination of internal and external evaluations anchored in the monitoring system • pretension of rigorous evaluation designs to attribute impacts (determination of a comparison group right at the beginning)

18. Thankyou! Merci! Gracias! Danke! Dr. Stefanie Krapp EvaluationUnit, GermanTechnicalCooperation (GTZ) stefanie.krapp@gtz.de www.gtz.de