Elements of poverty reduction program monitoring and evaluation

1. Elements of poverty reduction program monitoring and evaluation Sophisticated development context: the Kazakh experience Ayse KudatSocial Assessment2004

2. WHAT IS OUR CHALLENGE IN KAZAKHSTAN? • START A PRPM PROCESS • DO SO BY TAKING MODEST SUSTAINABLE STEPS • AS A FIRST STEP, INTEGRATE NATIONAL AND OBLAST LEVELS PRP ELEMENTS INTO A UNIFIED SYSTEM AND SUPPORT THIS INTEGRATION WITH REAL-TIME AND GEPOGRAPHICALLY SPECIFIC COMPUTER APPLICATION • DESIGN A PROCESS OF CAPACITY BUILDING FOR PRP AND PRPM BY LINKING INPUTS, ACTIVITIES, OUTPUTS, OUTCOMES AND IMPACTS AT THE OBLAST LEVEL • INITIATE A SIMPLE SYSTEM OF PRPM COORDINATION AND CREATE PRP MONITORING WORKING GROUPS IN MEBP, MLSP AND IN THE 14 OBLASTS • ENSURE MONITORING CONTRIBUTIONS OF NGOS CURRENTLY COOPERATING WITH OBLASTS IN PRP PREPARATION AND IMPLEMENTATION • SUPPORT THIS PROCESS IN 2 PILOT OBLASTS WITH FOCAL POINTS FINANCED BY THE TA PROJECT • SUPPORT THE PARTICIPATORY PRPM GOALS OF OBLASTS BY HELPING 2 PILOT REGIONS STRENGTHEN M&E CAPACITY IN THE RAYONS---HELP THESE OBLASTS ENHANCE LOCAL M&E CAPACITY WITH MODEST RESOURCES AVAILABLE IN TA PROJECT • CREATE INSTITUTIONAL ARRANGEMENTS FOR EVALUATION • CONDUCT PILOT STUDIES FOR IMPACT ASSESSMENT AND INITIATIVE SURVEYS TO TRACK PUBLIC EXPENDITURES • ASSESS EQUITY AND EFFICIENCY IMPLICATIONS OF PUBLIC SPENDING ON PRP • EVALUATE IMPLICATIONS OF TA PROJECT ACTIVITIES FOR NATIONWIDE IMPROVEMENTS IN THE M&E OF PRPS AND PRP

3. Q: WHAT DO WE LEARNFROM INTERNATIONAL EXPERIENCE OF POVERTY REDUCTION STRATEGIES/PROGRAMS? A: PUBLIC SECTOR EXPENDITURES AND ACTIVITIES DEVOTED TO POVERTY REDUCTION PROGRAMS (PRP) CANNOT BE JUSTIFIED OR CAUSALLY LINKED TO POVERTY TARGETS UNLESS A RESULTS BASED MONITORING IS ESTABLISHED

4. Impact Results Outcome Output Activity Implementation Input In Kazakhstan we know more about outcomes than about the means of achieving them Results Monitoring Implementation Monitoring (Means and Strategies)

5. Impact Results Outcome Output Activity Implementation Input Sources of Information for PRPM TA financed qualitative surveys, Public expenditure Tracking Surveys in 2 pilot oblasts, Households surveys, qualitative surveys Public expenditure Tracking Surveys, Sector specific surveys community infrastructure inventories, Rayon M&O achievements, WSS Systems repaired/constructed Oblast PRP workshops, task forces, Policy reviews, Oblast budgets, oblast payroll, Rayon budgets, rayon payroll

6. Progress made in PRPM&E in Kazakhstan • 1. Conducting readiness assessment • 2. Agreeing on Outcomes to monitor • 3. Selecting indicators to monitor outcomes • 4. Establishing baseline • 5. Selecting realistic targets • 6. managing for results • 7. Role of evaluations • 8. Reporting findings • 9. Using findings • 10. Sustaining M&E systems

7. Much is already known • What data are collected (source) • When data are collected (frequency) • How data are collected (methodology) • Who collects the data • For whom the data are collected • Who reports the data Where revenues Are collected, Expenditures made And activities held, Outcome data Are not systematically gathered RAYONS

8. Rayon Administrations Local Univ., and civil society organizations Oblast PRP working groups, with Oblast M&E Coordinators Households CITIZEN FEEDBACK Two Pilot Oblasts Remaining Oblasts Focal Points Ministry of Labor and Social Protection MEBP Inter-Ministry Steering Committee Secretariat Monitoring Working Group Joint evaluation Group Other Line Ministries National Statistics Agency Steering Committee Recommendations to the Parliament and the Cabinet for Poverty Reduction Strategy

9. Stakeholders in KAZAK PRS monitoring • Central Ministries (Finance, Economy) • MEBP consolidate information, • Line Ministries (MLSP) • Provide oblast based data, helps evaluate results • Reporting arrangement to be structured between MEBP and other line Ministries under the TA • Universities, research institutes • Local universities contribute to training and impact data gathering for small scale impact studies • Rayon governments • Key provider of input and output data to Oblasts • National statistical agencies • Conducts reliable and large scale, nationwide household surveys representative at the oblast level • Develops rayon level capacity for collection of output data • Civil society organizations • Trade unions provide feedback information, Association of SME can contribute impact data • Parliament/Cabinet • MEBP through the Steering Committee to carry out Evaluation and provide strategy feedback to Parliament and Cabinet

10. Participation and Feedback • Civil society involvement in policy formulation exists at Oblasts and through the TA financed focal points their contributions for M&E can be structured. • Through PRPM the program planning for future PRPs (2004+) can be improved. Targets, inputs, outputs can be better defined. • Sharing PRP results with public through the Newsletter of the national NGO consortium, as well as the website of the MEBP (with links to other line Ministries) can help generate citizen feedback and ensure sharing of PRP/PRS information

11. Client Feedback • Limited resources available for all studies, including report cards, are extremely limited under the TA. • Specific citizen feedback will be generated through the NGO Newsletter • Citizen/user feedback will be assessed for 2 projects (e.g., a- SME development for the unemployed, and b- micro credit for rural areas) in 2 pilot oblasts through surveys carried out by local universities. These studies will also include budget tracking surveys • Qualitative impact studies of pioneering programs in 2 pilot oblasts (a-process of social passport issuance and targeting; and b- emigration from ecological disaster areas) • PRPM works shops in 2 pilot oblasts with rayon and select settlement representatives; participatory evaluation of PRP as an input to next steps of PRP preparation.

12. SME training for The unemployed Pre-Implementation Assessment Process Implementation- commissions Case Study of Immigrants to 2 oblasts Impact Evaluation of Health sector expenditures Meta-Evaluation Types Of Evaluation Proposed within TA project

13. Monitoring data flow In terms of institutional mechanisms, the data on poverty will flow through several different channels • From RAYONS to OBLASTS to the central level MEBP for monitoring PRP • From RAYONS to Statistical Agency for monitoring of the outcomes Sectoral information, MIS data and/or data from sample surveys conducted by different central agencies, including the MLSP to MEBP • Data from key NGOs (e.g., trade unions, association of business and farmers’ association) to Oblast governments and transmitted to MEBP • Case studies, budget tracking studies and client satisfaction studies conducted at the Oblast level and shared with MEBP

14. Real-time Monitoring and Evaluation Software Sources of Monitoring Indicators • Input, Activity and Output Indicators: Unified and re-prioritized in accordance with the National Poverty Reduction Strategy • Data obtained from the oblast level • Data obtained from the implementing line ministries • Outcome Indicators • Data obtained from the oblasts • Data obtained from the National Statistics Agency Expected Results from the Monitoring Software • Analysis of specific actions underlined at the National Poverty Reduction Strategy in terms of actual outputs provided, and poverty reduction outcomes achieved • Making these results available at the oblast level • Enabling direct linkages between resources put through specific actions and policy outcomes of these actions, assisting policy makers in designating new priorities

15. Sample Indicators – How the Software Will Work Oblasts Inputs Name of Activity: Primary school repairs Amount of financing (to be indicated) If financing not available, indicate estimated number of man-hours dedicated to this activity Activities and Outputs Number of schools undergoing repairs Estimated number of beneficiaries (ie the number of pupils in these schools) National Level Outcomes Number of primary school age children studying in schools – urban and rural Percent of pupils in schools that study on the third shift Charts and Graphs for summary information and evaluation

16. Results will be made available on a geographical system (sample indicators on education) 30 schools repaired 15,000 pupils benefited 20 schools repaired 13,000 pupils benefited 10 schools repaired 9,000 pupils benefited