210 likes | 226 Views
Evaluation in US Foreign Assistance. Monitoring and Evaluation Roles, Systems, Priorities DAC Evaluation Network November 19, 2008 Paris, France Peter Davis: Office of the Director of US Foreign Assistance
E N D
Evaluation in US Foreign Assistance Monitoring and Evaluation Roles, Systems, Priorities DAC Evaluation Network November 19, 2008 Paris, France Peter Davis: Office of the Director of US Foreign Assistance Gerald Britan: US Agency for International Development Harry Carr: Millennium Challenge Corporation
Office of the Director of US Foreign Assistance (F) Monitoring and Evaluation for the US Foreign Assistance Program Meeting of the OECD/DAC Evaluation Network November 19, 2008 Paris, France Peter Davis: Lead Monitoring/ Evaluation- F: davispb@state.gov
The Office of the Director of Foreign Assistance • Established in 2006 • Responsible for coordinating USG Foreign Assistance (FA) • Developed a standard programming structure to codify FA objectives • Created a comprehensive database to track assistance across all programs, countries, and Bureaus • Developed systems to improve performance and accountability
Policies and Initiatives • Critical importance is given to monitoring and evaluation, performance management, and accountability. • Interagency coordination • Training • Development of support tools • glossary, standards, guidelines, indicators • Assistance with the development of evaluation policies for State and USAID
Country Assistance Strategy (CAS) • A new initiative to assure that longer-term, whole of government strategic planning is carried out • Will provide context for completing and reviewing 1 year Operational Plans • A short document that states overall USG foreign assistance priorities, regardless of funding source • Produced jointly by Field and Washington
New Standard Program Structure • The SPS classifies what FA is doing. It breaks down programs into tiered categories: • Program Objective (5) • Program Area (25) • Program Element (115) • Program Sub Element (364) • Standard indicators, linked to program elements, collect performance information consistently across all countries and programs
New Systems and Tools • New Strategic Framework with 5 goals • The Foreign Assistance Coordination and Tracking System (FACTS) collects and manages narrative, budget and performance information in a standard format and through a single point of entry • Development of standard indicators complemented by custom indicators • Operational Plans, Performance Reports • Emphasis on training for evaluation
Priorities • Monitoring and evaluation, performance management, transparency, and accountability • Working as a learning organization • Work with State, USAID, and other USG agencies to coordinate implementation of foreign assistance • Work on cross cutting or cross agency issues • Develop cross agency evaluations
Monitoring and Evaluation Policies and Practicesat the Millennium Challenge Corp. (MCC) Evaluation Network November 2008 Meeting Informal Session with USAID and State Department “F” November 19, 2008 Harry Carr, Managing Director for Monitoring/Evaluation:carrhc@mcc.gov
MCC’s Three Perspectives on Results Program results are assessed over three distinct time horizons: Future results are assessed – ERR Current results are monitored – M&E Plan Final results are evaluated – experimental design • To make sure that the program is logically coherent and its components are necessary and sufficient to accelerate economic growth • To calculate ex-ante the Economic Rate of Return (ERR) and poverty impact for program components • To collect performance data for better management of the program and to trigger future disbursements • To report to constituencies on progress achieved in reaching the program’s goal • To measure ex-post, in a statistically valid way, the program’s impact on growth and poverty • To provide evidence of program and activity effectiveness and • To learn lessons for future programs and test assumptions
The Program Logic of Results Goal Objectives Outcomes Outputs Processes Inputs Results Levels Results Indicators • Same for all MCA Programs: Economic Growth and Poverty Reduction • Income Outcome & Impact Level • Higher order effects of outputs on beneficiaries • Yield • Immediate effects of outputs on beneficiaries • # of farmers adopting new technology • Products and services produced • # of farmers trained in new technology • Curriculum developed • Training service provider mobilized Activity Level • Major milestones achieved • Financial, human and material resources • Budget/funding secured • Training service provider TOR released
The Monitoring and Evaluation Plan • Summary of Program and Objectives • Overview of the Program • Economic Growth and Poverty Reduction Impact • Program Logical Framework • Beneficiaries • Assumptions and Risks • Monitoring Plan • Indicators, Baselines, Targets • Data Collection Strategy • Data Quality Reviews • Evaluation Plan • Purpose • Methodology • Timeline • Organization structure and staffing
Impact Evaluation Reduces Selection Bias Selection Bias: Participants are often different than Non-participants Income With Project B Control Group D True Impact w/o Project Selection Bias C A Y1 Y2 Y3 Y4 Time
The Actors in M&E Contractors: Evaluation; Surveys; Training; etc. Implementing Entities: National Statistics Agency; Municipal Infrastructure Fund; Ministry of Agriculture etc. MCC MCA M&E Specialists M&E Director Economist Project Directors Sector Specialists Interagency Agreements: Engineering; Watershed Mgt.; Survey Statistics Grantees: Municipal Water Authorities Farm Service Centers; etc. Contractors: Evaluation; Surveys
LESSONS LEARNED • ACTIVITY LEVEL PROCESS INDICATORS MUST BE GATHERED EARLY • Key Process Milestones • NEED GREATER STANDARDIZATION: • Compact Program Logic • Indicator selection criteria • Standard performance monitoring reports • NEED CROSS-COUNTRY MCC PERFORMANCE MEASURES • Develop Core Indicators • LOCAL CAPACITY IS WEAK • Procedures guidelines and instruments • Training and technical assistance
Strengthening Evaluation at USAID Gerald Britan, Ph.D. Chief, Central Evaluation, USAID November 19, 2008 gbritan@usaid.gov Meeting of the OECD/DAC Evaluation Network Paris, France
USAID’s Evaluation Highlights • Project Evaluations (50s) • Logical Framework (late 60’s) • Central Evaluation Office (early 70s) • Impact Evaluations (late 70s) • DAC Evaluation Group (early 80s) • CDIE for KM (early 80s) • RBM Pioneer (early 90s)
USAID’S EVALUATION DECLINE • Performance Monitoring Grows (90s) • Evaluations Drop (450 to 150 by 01) • “Knowledge workers” Replace “evaluators” (02 Review) • Funding & Staff decline (03-05) • “Evaluation Initiative” Short-Lived (05) • CDIE abolished (06)
REVITALIZING EVALUATION • Mission Directors’ Conference (07) • Updating Evaluation Policy (07-08) • New central evaluation unit (08) • Strengthening technical support • Expanding evaluation training • Improving evaluation coordination • Re-engaging evaluation community
USAID’s Evaluation Priorities • Strengthen Our Evaluation Capacity • Implement a New Program of More Rigorous Impact Evaluations • Work with Development Partners on Collaborative Evaluations • Participate in Evaluation Organizations and Forums • Provide Intellectual Leadership