1 / 22

Policy & Operations Evaluation Department (IOB )

Policy & Operations Evaluation Department (IOB ).

caelan
Download Presentation

Policy & Operations Evaluation Department (IOB )

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Policy & Operations Evaluation Department (IOB) EVALUATION FOR ACCOUNTABILITY: Measuring Development Effectiveness of NGOsKeynote Address ECSP Conference ‘Challenges for Philanthropy: Scale, Governance & Impact’Rotterdam, 22 September 2011Prof. dr. Ruerd RubenDirector Policy & Operations Evaluation (IOB)Ministry of Foreign Affairs, The Netherlands

  2. Policy & Operations Evaluation Department (IOB) Policy & Operations Evaluation (1) • Mandate: • Assessing policy effectiveness • Promoting learning from experiences • Improving evidence-based policy-making • Accountability to Parliament, Public and Partners (autonomous programming & independent reporting)

  3. Policy & Operations Evaluation Department (IOB) Policy & Operations Evaluation (2) • Activities • Impact studies on main development priorities • Analysis of the coherence in foreign policy • Thematic studies on key success factors •  Creating Transparency & Trust

  4. Policy & Operations Evaluation Department (IOB) Key Messages • Development effectiveness has to show ‘value for money’ • Showing effectiveness may enhance public trust (necessary, but not sufficient condition) • Transparency starts with autonomy & objectivity • Evaluability is key responsibility of donors and recipients • No learning without insights in quantitative impact

  5. Policy & Operations Evaluation Department (IOB) Development Effectiveness • Delivery of meaningful results • Net impact over time • Value for Money • Attribution/Contribution • Additionality

  6. Policy & Operations Evaluation Department (IOB) Evaluating Civil Society Large share of Dutch aid: 5%  20 % of ODA (+ 10% ODA through Delegated budgets) Three components: Direct Poverty Reduction (MDGs) Capacity Development (5C) Lobby & Advocacy

  7. From Input to Impact Policy & Operations Evaluation Department (IOB) Outreach = # of participants Throughput = activities within the system

  8. Policy & Operations Evaluation Department (IOB) Common Pitfalls • Strong selection bias • Spatial overlaps: externalities & free riding • Absence of baselines data • Scarce controls • Long result chains • Limited additionality

  9. Policy & Operations Evaluation Department (IOB) Goal Credible evidence with • Impact measurement: comparing: • before/after (time) • with/without (counterfactual) • Everything else is additional • (e.g. story harvesting, outcome mapping, SROI, etc) • (See: NONIE Guidelines ; 3ie) without T=0 T=1 Time

  10. Policy & Operations Evaluation Department (IOB) Difference in Difference (DiD)

  11. Policy & Operations Evaluation Department (IOB) Propensity Score Matching (PSM)

  12. Policy & Operations Evaluation Department (IOB) Evaluation Criteria • Effectiveness • Efficiency • Relevance • Coherence • Sustainability • See: OECD-DAC Evaluation Guidelines

  13. Policy & Operations Evaluation Department (IOB) Methods & Approaches Document review Policy reconstruction Robust Impact studies (DiD) Experiments (Real-time evaluation) Systematic Reviews (Campbell protocol) Thematic studies (Key success factors)

  14. Policy & Operations Evaluation Department (IOB) Worrisome Evidence ‘NGO support is highly appreciated, but no evidence can be generated to demonstrate its impact’ ‘Local capacities of NGO partners have grown substantially, but it remains unclear whether and how donor support has contributed to this’ ‘Access to services and markets for the poor have improved, mainly due to better socio-economic performance of the country’

  15. Policy & Operations Evaluation Department (IOB) Declining trust

  16. Policy & Operations Evaluation Department (IOB) Reputation & Trust Insight in development effectiveness is a necessary (but not sufficient) condition for maintaining public support “Show me” Tell me” “Tell me” “Trust me” “Trust me” “Showme”

  17. Policy & Operations Evaluation Department (IOB) Evaluation & Accountability • Guaranteeing Evaluability • Intervention theory • SMART indicators • Random sampling • Valid baselines • Comparison/control groups

  18. Policy & Operations Evaluation Department (IOB) Organizing Evaluations Independency Quality control systems Peer review Horizontal control Certification (CBF, NEN)

  19. Policy & Operations Evaluation Department (IOB) Learning from Evaluations Single, double & triple loop learning Defining innovation spaces Real-time experiments

  20. Policy & Operations Evaluation Department (IOB) Methodological Challenges Overcoming poverty thresholds Focus on Complementarities Search for Policy coherence Measuring Additionality Institutional Strength (proxies) Spillovers (catalyst effects) Externalities

  21. Policy & Operations Evaluation Department (IOB) Institutional Challenges Organizing Independent NGO Evaluation Units Upfront Focus on Evaluability Training of local (Southern) partners Web-based open data systems (see: www.ngo-database.nl) Openness on Failures & Successes Comparing NGO performance (ranking)

  22. Policy & Operations Evaluation Department (IOB) Thanks for your Attention…

More Related