1 / 29

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?. ACDI/VOCA Vision . Vision A world in which people are empowered to succeed in the global economy. Mission

teagan
Download Presentation

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?

  2. ACDI/VOCA Vision • Vision A world in which people are empowered to succeed in the global economy. • Mission To promote economic opportunities for cooperatives, enterprises and communities through the innovative application of sound business practice. • Our Work • Financial Access • Enterprise Development • Community Development • Food Security • Agribusiness

  3. Results Based Oriented • M&E systems at ACDI/VOCA have two primary purposes: accountability to stakeholders and learning to facilitate results-oriented program design and knowledge-based management. • Accountability refers to measuring project efficiency, effectiveness, relevance, and sustainability of projects, as well as transparently (disclosing) sharing findings from project evaluation to stakeholders, and using the findings to inform resource allocation and other decisions. • Learning refers to the systematic generation of knowledge that can be used to refine designs and introduce improvements into future efforts with the goal of achieving meaningful and sustainable results. M&E systems should ultimately demonstrate that ACDI/VOCA’s activities are contributing to its vision and producing sustainable and equitable development results for the communities in which it works.

  4. Results Based Guiding Principles • Integrated into program management and technical approaches; • Systematic so as to produce reliable, accurate and useful information, while also • Flexible to the complex environments in which ACDI/VOCA operates; • Participatory of stakeholders and beneficiaries; • Ethical in the collection and reporting of information about and to beneficiaries, partners and stakeholders (Confidentiality and anonymity-Do no harm); • Supportive of building local capacity of staff, partners and beneficiaries in M&E; • Foster inclusivity and equity of program interventions for both men and women and traditionally marginalized groups; and. • Results-oriented: in addition to focusing on implementation processes; ACDI/VOCA M&E systems will emphasize results by asking the “so what” questions.

  5. Capturing Performance Indicators on a Statistical Canvas (CaPISC) • CaPISC is an online database that captures the performance monitoring plans for all of ACDI/VOCA’s 70 active projects. To help ACDI/VOCA measure progress towards the goals at the project and practice area level to support our attention to accountability and learning. The system will provide ACDI/VOCA with agency-wide performance measures for: • Reporting to stakeholders and donors • Communicating with the development community • Managing regional and country programs through comparative performance measures • Measuring beneficiaries reached by AV intervention • Measuring integration of activities • Reporting on Companywide common indicators • Managing Practice Areas/ individual Project • Developing New Business (Marketing)

  6. Why Global Performance Indicators? • To assess the performance of its projects, A/V would like to use a comprehensive performance measurement framework. • ACDI/VOCA will track this progression of its performance through results reporting against indicators from the project level (predominantly outputs), to the broader program level (predominantly outcomes). • The indicators will be standardized to the greatest degree possible, thus allowing ACDI/VOCA and its partners to compare performance across the portfolio of projects.

  7. A Starting Point • Global Level Indicators • Per capita expenditures (as a proxy for income) of USG targeted beneficiaries • Number of jobs attributed to A/V implementation • Number of hectares of natural resources showing improved biophysical conditions as a result of USG assistance • Percent of beneficiaries adopting xx number of improved practices/technologies

  8. Goals of CaPISC CaPISC allows for current project reporting thus more responsive management and program adaptation. Reporting with CaPISC will allow for enhanced program and institutional learning.

  9. Uses of CaPISC – as a Results Based Management Tool Visually depict progress on project goals and indicators Determine progress of key project indicators COP Dashboards Provides tools to help you analyze your data Create reports for donors

  10. Use CaPISC to Help in New Business Development • Illustrate results over time and across projects • Show past experience and results • Demonstrate capability using indicator data • Model PMPs after similar ones in the system • Uniform terminology • Global indicators • Indicator library

  11. Challenges/Lessons Learned • It has been a long path to persuade the management to invest in this database. • It is taking some time to train project level M&E specialist on how to use and enter the data • Data quality (especially data that comes from the COs). • Recourses (human and financial)

  12. InterAction Forum 2012 Measuring Agency-Level Results CRS Beneficiary and Service Delivery Data (BSDI) May 1, 2012

  13. Beneficiary and Service Delivery Indicators (BSDI) Output Level Indicators that Measure Agency-wide Performance on CRS Fundamental Program Outputs Services Delivered Beneficiaries Served Beneficiary and Service Delivery Indicators (BSDI) Presentation

  14. Illustrative Program Area and Services Catalogue: Water and Sanitation Beneficiary and Service Delivery Indicators (BSDI) Presentation

  15. BSDI Data Flow Map Point of Service Delivery Typically data are recorded on paper by individual beneficiary Beneficiary and service delivery data are registered Data Typically data are transcribed to Excel files Regional/Partner Office Beneficiary and service delivery data are recorded and records are updated Data Project Management Office Typically data are stored in a project database Beneficiary and services delivery data are stored, maintained, analyzed and reported Data Reports On-line Agency Database (PIMS) Donor Beneficiary and Service Delivery Indicators (BSDI) Presentation

  16. BSDI Performance Products BSDI allows accurate, annual agency-wide tracking of CRS performance BSDI allows integrated technical and financial performance analysis BSDI will track “double counted” beneficiaries in a multi-sector projects Beneficiary and Service Delivery Indicators (BSDI) Presentation

  17. Mission Metrics

  18. Incoming Data Analyze Reflect/ Discuss Cycle of Learning Review Results/Learn Make Decisions Take Action

  19. Measuring Agency Level ResultsInterAction –Forum 2012May 01, 2012

  20. Purpose, key drivers and elements of Agency Level Measurement Purpose Measure progress towards multi-year strategic plan—specifically to support an overarching objective of “accountability for results” Communicate w/staff and board and on performance against our intended results for children within the context of our “Theory of Change” Key Drivers Accountability to ourselves and to others (specifically children) Need to have a tool for management purposes from the senior leadership of the organization (including the board). • Key Elements • Operational Metrics (eg, budget performance, revenue generation, brand recognition, media rating etc) • Programmatic and Policy Metrics (eg. Reach, Global Indicators around our Theory of Change) • Audience and Use • A dashboard for Senior Management Team/Board (Accountability) • Staff at all levels (Global, Regional, CO staff) – Learning, new business development • Public/Donors – public report (Accountability)

  21. Theory of ChangeHow we work to create impact for children

  22. Understanding Results The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention. RESULTS Impact Outcome Input Output Input Output Outcome Impact The financial, human, and material resource used for the development intervention. The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes. The likely or achieved short-term and medium-term effects of one or more interventions’ outputs. Positive and negative, primary and secondary long-term effect produced by a development intervention, directly or indirectly, intended or unintended Definition E.g. $ spent, staff time e.g. # health workers trained, teachers trained, e.g. children accessing life saving interventions; Policy change implemented e.g. children’s lives saved; under 5 mortality rate over district, region or country Examples REACH GLOBAL INDICATORS EVALUATIONS

  23. Reach Measures the number of children and adults who have received program input and/pr services directly and indirectly by Save the Children and its partners in the fiscal year disaggregated by theme and sub-theme area. Global Indicators A set of outcome indicators linked to Save the Children’s global outcome statements which measure achievement of outcomes. Advocacy Measurement A tool to measure progress against an advocacy objective aimed at policy change and/or implementation Components of our program/policy results monitoring system (towards our strategic goals)

  24. Example of Global Indicators • Intended Impact: Children learn and develop with age appropriate care and education • Outcome Statement: In support of MDG2, by 2015, Save the Children will have significantly contributed to getting 2 million of the hardest to reach children into school, raising the Quality of Learning Environments (QLE), and improving the learning outcomes of more children • Indicators: • Quality of LE: % of Basic Education and Early Childhood Care and Development schools / sites supported by SC that achieve 4 guiding principles

  25. Buy in and support from the senior leadership at all levels (global and country) is critical for the success of such a system. Agency dashboard is used to discuss results quarterly—informing decision making on a regular basis. Data quality and management can be overwhelming, capacity (at all levels) and resource constraints to institute a strong system at all levels. Shared global objectives has helped to establish shared metrics across the Save the Children Global Movement – however determining attribution is a challenge. Keeping the balance between a concise dashboard (with few metrics) while providing a comprehensive picture of our results for children and the complex nature of our work. Reach figures do not tell us about our program impact. But measuring outcome and impact level indicators are resource intensive. Lessons learned and key challenges

  26. Contact

More Related