Approaches to Harmonizing Public and Private Sector Project Evaluation

1. Approaches to Harmonizing Public and Private SectorProject Evaluation Kris Hallberg and Nicholas Burke, Consultants Evaluation Cooperation Group October 25, 2013

3. Other Differences Between the GPSs • Basis of the evaluation • Public sector: objectives-based • Private sector: commercial benchmarks • Attribution • Environmental and social safeguards • Definitions of rating scales and what is required for a satisfactory rating

4. Option 1: A Common ReportingFramework • Maps existing Public GPS and Private GPS criteria to standard OECD-DAC criteria • Minimizes changes to existing criteria • Some criteria are disaggregated • Some optional criteria are made mandatory • Could be used by CEDs as an option, e.g., for synthesis reporting at the aggregate level

5. Public GPS Criteria OECD-DAC Criteria Private GPS Criteria Relevance Relevance Fulfillment of Project Business Objectives Effectiveness Effectiveness Efficiency Efficiency Impact Sustainability Sustainability Overall Project Performance Rating IFI Additionality IFI Performance IFI Investment Performance Borrower Performance IFI Work Quality/Bank Handling

6. Option 2: A CommonEvaluation Methodology • Establishes a common basis of evaluation, sharpening the focus on the project’s intended development, policy, or transitionresults • Defines common criteria that are conceptually consistent, with some sub-criteria specific to public and private sector operations • Provides guidance on evidentiary requirements, analytical methods, and benchmarks for a positive rating • Clarifies how to assess and rate the results attributable to the project

7. Basis of the Evaluation • Both public and private sector operations are evaluated against a set of intended outcomes that are specific to the project • For private sector operations, “internalizes” the project’s intended PSD, transition, and regional economic impact outcomes • Intended outcomes are taken from project entry documents or reconstructed by the evaluator

9. Public Sector Private Sector Relevance Strategic Alignment Rationale for IFI Intervention Strategic Alignment Rationale for IFI Intervention • Consistency of the project’s intended results with beneficiary needs, country priorities, and IFI assistance strategy and corporate goals • Evidence-based justification for market failures and targeting • Financial and non-financial additionality (private sector) • Comparative advantage of the IFI relative to other providers (government, private sector, other donors, NGOs)

11. Public Sector Private Sector Achievement of Outputs Fulfillment of Project Business Objectives Achievement of Outputs • The extent to which the project achieved its targeted outputs (appraisal projections or performance targets) • The extent to which the project delivered on its process and business objectives (private sector) • Attribution analysis not needed

12. The project’s contribution (or expected contribution) to its intended development, policy, or transition outcomes • The project’s reach to intended beneficiaries • The project’s success in reducing or compensating for market failures • The project’s impact on all stakeholders (employees, suppliers, competitors, etc.) (private sector)

13. Outcomes are assessed against a without-project counterfactual, either using an impact evaluation or (more commonly) a theory-based approach to establish plausible causality • The rating reflects the project’s incremental contribution to observed outcomes, regardless of whether observed outcomes moved in the “right” or “wrong” direction • On an exceptional basis, “Not Rated” is possible when evidence is missing or weak

14. Rating Based on Results Attributableto the Project SME employ- ment Observed What would have happened without the project (counterfactual) Observed decrease in employ- ment Higher employment attributable to the project Counterfactual Project Start Project End

15. The project’s contribution to broad corporate goals that were not included in the project’s intended results (e.g., rural poverty reduction, shared prosperity, gender equality) • Uses a theory-based approach to establish plausible causality • The rating reflects the project’s incremental contribution to observed outcomes, regardless of whether observed outcomes moved in the “right” or “wrong” direction • “Not Rated” is a possible rating

16. Positive or negative results of the project that were not reflected in its intended outcomes (e.g., environmental damage) • Uses a theory-based approach to establish plausible causality • Unintended outcomes must be at least as well evidenced as the project’s intended outcomes • “Not Rated” is a possible rating

17. Public Sector Private Sector Efficient Use of Resources Financial Performance Economic Efficiency Implementation Efficiency IFI Investment Profitability Economic Efficiency Implementation Efficiency • The profitability of the project from all perspectives: financial stakeholders (privatesector), society at large (public and private sector), and the IFI (private sector) • The financial performance of the project/company is compared to a without-project counterfactual (private sector) • The economic efficiency of the project (CBA or CEA) is compared to a without-project counterfactual (public and private sector)

18. Public Sector Private Sector Sustainability Outcome Sustainability Commercial Sustainability Compliance with Safeguards Outcome Sustainability Compliance with Safeguards • The sustainability of outcomes achieved or expected to be achieved (public and private sector) • The commercial sustainability of the company, financial institution, and/or sub-borrowers/fund investees (mainly private sector) • The environmental and social sustainability of project results, measured by compliance with relevant safeguards (public and private sector)

19. Relevance Efficient Use of Resources Sustainability Overall Project Performance Rating

20. Public Sector Private Sector IFI Performance Quality at Entry Quality of Supervision Quality at Entry Quality of Supervision Client Performance Non-Financial Performance of the Company Gov’t. and Implementing Agency Performance • “Below the line” criteria • IFI Performance: Quality at Entry and Quality of Supervision (public and private sector) • Client Performance: Government and Implementing Agency Performance (public sector); Non-Financial Performance of the Company (private sector)