The Application of the CDM M,E&R Framework at the National and Regional Level

1. The Application of the CDM M,E&R Framework at the National and Regional Level 5th Annual CDM Conference Jamaica – December 8th, 2010 Le Groupe Conseil baastel Ltee

2. Outline • What is the Regional CDM M,E&R Framework and Why is it Important? • The Process of Establishing the Framework • How to Monitor using the Framework • Putting the Framework into Practice – now and potential for the future • Considerations and Next Steps

3. What is the CDM M, E &R Framework? Why is it Needed?

4. Why the need for a CDM M,E&R Framework? The CDM Strategy/Framework is based on a variety of contributing sources including Country Work Plans (CWPs), partner projects and programmes, either nationally or regionally and sectoral programming

5. What is the M,E&R Framework? Why is it Important? • Developed in 2009-2010, the CDM Monitoring, Evaluation and Reporting (M,E&R) Framework outlines the overall strategy and plan for monitoring and reporting of the Enhanced Regional CDM Strategy and Framework • Impetus for the framework is the need to track/report on progress of Disaster Management interventions related to the CDM results (outputs/outcomes) in the Caribbean Region • Uses a utility-based Results Based Management approach for tracking progress in order to reduce monitoring burden and improve data and information being collected and shared

6. What is the M,E&R Framework? Why is it Important? • The purpose is to enable CDEMA to measure ongoing performance of progress being made towards Regional CDM expected results, facilitate reporting from the national and regional levels, assist in meeting accountability standards with development partners, and facilitate the learning agenda • Overall, it serves as a tool for decision-making as well as to inform strategic planning, programming and learning • At country level • At regional level

7. Core of the Model – CDM Impact and Outcomes

8. Core of the Model – CDM Outputs

9. Monitoring the CDM Framework • Development of a Performance Measurement Framework (PMF) as the core monitoring tool and guide • PMF guides planning of the systematic compilation of data on progress and results. It outlines the main elements of the monitoring system and ensures that performance information is collected regularly/on time

10. Salient Features of the CDM MER Framework - PMF Elements Definition of ‘Indicators’ in the CDM Strategy/Framework Context An indicator is a representative aspect or element of a CDM Output (or Outcome) in which change over time is measurable and the change attributable to activities (i.e. projects & programmes) and is used to demonstrate progress towards the attainment of the expected Result (i.e. Output/Outcome) It can be used to ‘detect’, ‘point to’, or indicate some change (or lack thereof) over time in relation to a given expected Result

11. Salient Features of the CDM MER Framework - PMF Elements

12. Salient Features of the CDM MER Framework - PMF Elements Progress towards attainment of CDM Outputs due to completed activities

13. Salient Features of the CDM MER Framework - PMF Elements “Keeping it Manageable” • Guidance of Review Committee (CDEMA, NDCs, Development and Sectoral Partners) • Limitedset of indicators for each Output • To get process going (delayed monitoring & baseline already when mandate began in 2009) • To catalyze buy-in to M, E & R Framework • To keep monitoring feasible • Recalling monitoring & reporting burden on NDOs

14. Salient Features of the CDM MER Framework - PMF Elements

15. Salient Features of the CDM MER Framework - PMF Elements • Data Requirements, Sources and Tools: • Data requirements for each output and outcome indicator are essential to respond to the indicator’s measurement of each result. For each indicator, data requirements are indicated • The source is where or who this data will come from – these include CDEMA CU, NDOs, Key Sectors and other Ministries, etc. • Tools: Nine simple tools have been developed and are contained in Volume II: M,E&R Framework Manual and Tools

16. Salient Features of the CDM MER Framework - PMF Elements • Responsibility: Core responsibility of CDEMA CU. They also will depend heavily on the NDO for provision of data • Frequency: Annually, for the majority of output/outcome indicators. Collection at more frequent intervals would not necessarily result in significant or demonstrable changes due to the regional nature of programming • Collection Methods: Two primary methods of collection: a) document review; and b) questionnaire (via electronic means, telephone or in-person, etc.) When a web-based platform is established this method will change to solely web-based questionnaire, considerably reducing the data collection burden

17. Salient Features of the CDM MER Framework – Key Linkages • Linkages between and to National CWPs, sectoral programming and other projects/programmes • Data can be gleaned from the M&E frameworks of these to contribute to Regional CDM monitoring and reporting. • While all data needs are not satisfied, there is considerable synergy. Conceptually it looks like:

18. Linkages/Mapping to Hyogo Framework of Action (HFA) • Monitoring slated to take place through the M,E&R Framework facilitates this global monitoring as outputs and accompanying indicators are either directly or partially linked to the Priorities for Action (PFA) indicators • Reporting to HFA in the CDM context can be done via: • National PFA Indicators & Regional PFA Indicators

19. Mapping to Hyogo Framework of Action (HFA)

20. How Did We Get Here?

21. Process for Establishing the M,E&R Framework Participatory Process Development of CDM Outcomes and Outputs PHASE 1: Selection of Indicators PHASE 2: Identification of Data Requirements PHASE 3: Identification of Data Sources PHASE 4: Tool Design and Collection Process of Data Final M,E&R Framework PHASE 6: Next Steps Draft M,E&R Framework Regional Baseline Report PHASE 5: Country Missions and Data Collection Ongoing Monitoring and Reporting Indicators Country Baseline Reports Baseline Collection Tool (BAT) Systems Development (ICT Framework) Technical Report & Excel Database (Raw Data) Benchmark Tool Volume II: M,E&R Tools Current Phase

22. Establishing the M,E&R Framework • It has been an iterative process in that it has involved various stakeholders including NDOs, donor agencies, sector representatives, etc. • Began with RBM workshops in December 2008 and February 2009 and Pre-Board Workshops in April 2009 • Work since December 2009 has focused on Phase 5 of the process with country visits and collection of baseline data • This led to the development of a regional baseline study, country baseline reports, the regional technical report and the overall M,E&R Framework (Volume I) and Data Collection Tools for Monitoring (Volume II)

23. CDM Regional Baseline Data and Report • The baseline is critical to the M,E&R Framework and now provides a ‘starting point’ against which measurement can begin • Baseline data was collected in late 2009/early 2010 from 16 countries against an agreed set of core indicators using a Baseline Assessment Tool (BAT) Some Sample Baseline Data Follows

24. Outcome 1: Enhanced Institutional Support for CDM Program Implementation Output 1.1: National Disaster Organizations are Strengthened for Supporting CDM Implementation and a CDM Program is Developed for Implementation at the National Level Indicator 1.1.1: # of countries with drafted/enacted disaster legislation 8 of 15 countries (53.3%) state that they have a drafted or enacted legislation, 6 of these 8 (75%) state that it mandates and defines their status. This is not applicable to 7 countries.

25. Output 1.6: Capacity for Monitoring, Evaluation and Reporting Being Built NDO ratings of ability to respond to CDM-related reporting requests 14 respondents

26. Output 2.2: Utilization of Information Communication Technology (ICT) and other Infrastructure for fact-based Policy and Decision-Making is Established and Strengthened NDO rating of ICT inter-operability between CDEMA CU and countries 15 respondents

27. Output 3.1: CDM is Recognized as the Roadmap for Building Resilience and Decision-Makers in the Public and Private Sectors Understand and Take Action on Disaster Risk Management 12.5% 80% 83.3% 66.7% 13 respondents 14 respondents 13 respondents 12 respondents Whether sector disaster plans are congruent with NDP 80% 83.3% 12.5% 66.7% 13 respondents 14 respondents 13 respondents 12 respondents

28. Output 3.4: Prevention, Mitigation, Preparedness, Response, Recovery and Rehabilitation Procedures Developed and Implemented in Tourism, Health, Agriculture and Education Sectors rating of level of adequacy of institutional framework, by sector 80% 83.3% 12.5% 66.7% 13 respondents 14 respondents 13 respondents 12 respondents

29. Output 4.2: Improved Coordination and Collaboration between Community Disaster Organizations and other Research/Data Partners including Climate Change Entities for Undertaking CDM Community accessibility to hazard maps and relevance of mapping 12Respondents 80% 83.3% 12.5% 66.7% 13 respondents 14 respondents 13 respondents 12 respondents

30. Output 4.4: Standardized Holistic and Gender –Sensitive Community Methodologies for Natural and Anthropogenic Hazard Identification and Mapping, Vulnerability and Risk Assessments and Recovery and Rehabilitation Procedures Developed and Applied in Selected Communities Indicator 4.4.5: Evidence that shelter management policies & procedures address women’s protection (by country) Baseline data suggests that 8 of 15 countries (53.3%) have shelter management policies and/or procedures that address women’s protection. Indicator 4.4.6: Evidence of sex-disaggregated data collection at country level on damages, loss, etc. National data collection and reporting on damages and losses after disasters is disaggregated by sex in 9 of 15 countries (60%). One additional country noted that this kind of disaggregation was being planned and tools are in development. Sex-Disaggregated Data 80% 83.3% 12.5% 66.7% 13 respondents 14 respondents 13 respondents 12 respondents

31. Technical Report – Raw Data by Country To compliment the baseline report, a secondary technical supplementary document was developed. The contents of this report include: • Explanatory notes on data analysis and aggregation of data • Results of each indicator by output (broken down by country) as a disaggregation of data • Explanatory notes and references for each indicator relevant to the transfer of data to any web-based platform or database

32. CDM Regional Baseline Data and Report • In addition, Country Baseline Reports were also developed for each of the 16 countries covered

33. Limitations Like any baseline, there are challenges and limitations. Two of the issues that are most critical to keep in mind are: Economies of Scale - Because data presented regionally, assumes indicators speak equally across countries – they do not. Analysis limited in terms of country size, population, frequency of disasters, poverty rates, etc. Understanding of Terminology/Concepts • CDM: discourse around CDM during baseline data collection is still in process of evolution and therefore understood differently between stakeholders outside of NDO. This limited uniform responses and required an enhancement.

34. Limitations and Resolutions Understanding of Terminology/Concepts • Climate Change discourse also evolving and various levels of understanding. Since no common agreed upon criteria for integrating climate change, provision of precise data was limited and required an enhancement. • Gender: understanding of gender mainstreaming not understood and defined consistently and various levels of understanding. Since no common agreed upon criteria for gender mainstreaming, provision of precise data was limited and required an enhancement.

35. Limitations and Resolutions • Based on this process, several issues have been resolved through the final M,E&R Framework: • Indicators: certain indicators (and benchmarks) have been revised and enhanced to improve data being collected and facilitate data collection processes in the future b) Tools: based on experiences the BAT and subsequent data analysis, a revised set of user-friendly monitoring tools have been developed

36. Improvements for Monitoring of Cross-Cutting Themes - Gender • ISDR and the HFA require that countries mainstream gender in risk management. Aside from this requirement, mainstreaming of gender in disaster management and risk reduction is crucial as women and men are affected differently by disasters and hazards • The M,E&R Framework integrates gender as a cross-cutting theme across the four outcome areas through 11 gender-specific or gender-integrated indicators (e.g. integration of gender issues into policies/plans; evidence that shelter policies/plans & procedures address women’s protection, etc) • Indicators and data requirements for these, as well as others, have been revised to include criteria for more comprehensive assessment of gender in CDM monitoring

37. Cross-Cutting Themes - Gender Example of revised data requirements for gender indicators – Tool A

38. Cross-Cutting Themes – Climate Change • The CDM Strategy/Framework addresses the cross-cutting issue of Climate Variability and Change and key considerations were integrated into the outputs of the Strategy and Framework in 2007. • In the current PMF, there are a total of 15 indicators which either directly address climate change and adaptation or where there is overlap and synergy between measuring aspects related to CDM and those related to climate change • Data requirements have now been refined to ensure more comprehensive assessment of climate change elements

39. Cross-Cutting Themes – Climate Change

40. How to Monitor Using the Framework Putting it Into Practice

41. Process for Monitoring CDM • Challenges in monitoring at the regional level due to: • Multiple and diverse stakeholders with different needs, systems and processes • Lack of harmonized programming/monitoring which involves multiple monitoring requirements • Lack of existing ICT capabilities present in the region to accommodate monitoring at a regional level • As a result, two phase approach is necessary toward functional and sustainable system: • Phase 1: addresses immediate steps and processes that can be implemented in context of current reality • Phase 2: processes once an enhanced ICT system in place

42. Phased Approach to M,E&R Framework • Phase I: • Use of Excel database with all raw data (keep as is) or transfer to another chosen in-house and stand alone off the shelf software (e.g. File Maker, Access) • In early 2011, disseminate/administer data collection tools. These can be done electronically or submitted in paper form • Phase II: Expansion and enhancement of an ICT platform within specified time frame so data can be uploaded directly

43. Phase 1: Indirect Process for Data Collection

44. Steps to Monitor in Phase I

45. Monitoring – Data Collection Tools • In total there are 9 tools that have been developed – all in Word format with Active Controls in order to simplify completion of data forms

46. Monitoring – Data Collection Tools Keeping it simple: • Check Boxes: data entry persons select appropriate selection. In some cases, data entry persons will need to select more than one choice. In these instances, instructions follow the question. • Option Button: option buttons are used for all benchmarks where data entry persons can only select one choice. If a choice is changed, the option control will automatically change.

47. Monitoring – Data Collection Tools • CDEMA CU is responsible for overall monitoring and so there are more tools specific to the CU as data which is submitted to the CU is done so for analysis at the regional level by output • Tool B-1: Outputs: This tool is used for indicators where CDEMA CU is responsible for data collection directly and is developed in a similar way as the tools for NDOs and sectors • Tool B-2: Roll up from country and sectors: This tool is used for the aggregation or consolidation of data collected through the NDO, Key Sectors or other stakeholders. It is intended as a data capture sheet prior to input into a database • Tool B-3: Outcome and Impact: This tool is used to capture data of all four outcomes as well as the three indicators for impact • Tool B-4: Document checklist : This tool serves as a checklist for documents that require review to validate indicators at all levels

48. Data Collection Tools: Example Tool B-1

49. Data Tools: Example Tool A

50. Summary of Monitoring Process • From National to Regional b) From Sectoral to Regional