Download
1 / 20
200 likes | 352 Views
Innovations in Investment Climate Reforms: An Impact Evaluation Workshop Takeaways and Next Steps. Outline. Understanding impact evaluation Impact evaluation (IE) vs. Monitoring IE as an operational tool for reform design and results attribution Overview of methods
E N D
Innovations in Investment Climate Reforms: An Impact Evaluation WorkshopTakeaways and Next Steps
Outline • Understanding impact evaluation • Impact evaluation (IE) vs. Monitoring • IE as an operational tool for reform design and results attribution • Overview of methods • Evaluations under the IC impact program • Research questions • Key methodological points to keep in mind • Next steps
IE vs. Monitoring • Monitoring • Offers a way to track progress over time and whether projects are moving in the “right direction” • Tracks only those that are receiving the intervention • Does notmeasure if our intervention caused certain outcomes • Impact evaluation • Identifies the underlying mechanism, tells us why things happen or how to make them happen • Tracks both, those receiving the intervention (treatment group) and a group with similar characteristics as the treatment group that is not receiving the intervention (control group) • Measures the outcomes causedby our intervention
IE as an operational tool • IE can help answer the “big” questions: e.g. what is the impact of the business registration reform on job creation? Does tax reform promote formal firm creation? How do risk-focused inspections impact business growth and private investment? • BUT, is also critically important as an operational tool to help test variations in project design to maximize impact: e.g. which value added services are most effective in maximizing the impact of the registration reform on job creation? Do thresholds of eligibility for tax reforms de-courage firms to declare a higher fraction of their income? How does information on the probability of being inspected influence compliance?
IE as an operational tool: Example for business registration
IE as an operational tool: for better informed decisions • Improve the design and implementation of CIC projects Impact assessments will measure the overall impacts of interventions, test their underlying mechanisms, combinations with other interventions and sequencing. • Improve strategic decision making and resource allocation Findings on impacts will improve strategic choices on the project portfolio, and support our evidence-based approach to policy making through a better understanding of what works, what doesn’t, where, why and for how long. • Share with client countries and development partners Increased knowledge on various impacts will support the decision-making process of our donors’ related to prioritization of reform topics and funding.
Evaluations under the IC impact program • Benin Registration “Entrepreneurial” Status • Nepal Registration Automation • Bangladesh Licenses • Kenya Health Safety Inspections • Tajikistan Risk Based Tax Audits • South East Europe Trade Logistics • Kosovo Investment Policy Incentives • Romania Insolvency Reform • ECA Food Safety Standards
Key methodological points to keep in mind • Intervention take up • Multiple treatments • Sub-group analysis • High-frequency data • Sampling • “General equilibrium” effect and spillovers • Realistic timelines
Intervention take up • Low take-up may reduce chances of measuring impact. Need to think about innovative techniques to ensure high take up (using behavioral lessons). • Nepal/Benin/Bangladesh • Measuring the effect of formalization on jobs requires a large increase in registration probabilities. Worthwhile having a “treatment arm” that is designed specifically to increase take up dramatically with the objective of measuring the effects of formalization. For example, a handholding activity (basically doing everything for firms to ensure formalization, following the Malawi IE example) can be used to ensure high registration take up and to test how the act of formalization impacts business growth and jobs.. • Specify the strategy on how to target informal businesses • Romania • Relies on encouragement design. Measure if take up differs across regions to assess if some firms are less accessible to the training location. Highly indebted people are also less likely to take calls or open letters. • Intervention might vary depending on instructors and influence results (will the training course be always the same?). Also, make sure to clarify eligibility criteria.
Multiple treatments • Multiple treatments compare effects across different interventions and their combinations. The more comparisons you make, the larger the sample you need to be “confident”. Different treatments also affect various populations, and results should be interpreted accordingly. • Benin • The IE goes beyond usual business registration IEs, which focus on the effects of registration simplification (and possibly information campaigns) on registration rates, by studying the effects of different incentives packages (registration+information,+microloan,+tax, and combinations thereof) on business performance. Each package entails different take up rates and populations. The team should make sure to interpret impacts based on the characteristics of populations receiving specific treatments. • Nepal • What are the different ICT components to be tested as a part of the new e-engineering and automating registration process? Consider adding several value added services (VAS), in addition to a business directory, to test the economic benefits of such registration? • Benin/Kenya • Several treatment arms will require a large sample. • Bangladesh/Tajikistan • Consider other useful VAS for businesses/taxpayers (Tajikistan: consider measuring corruption). • Kosovo • Not feasible to test all types of incentive schemes, but could assess some variations.
Sub-group analysis • Important, but requires bigger samples. Need to stratify sample at start: male vs. female entrepreneurs, small vs. bigger businesses/health facilities, different business sectors or region etc., depending on the theory of change and what we want to test. • Benin • Targeting different segments of informal businesses is likely to matter a lot (i.e. who gets chosen to be an “entreprenant”). The team could be more open in the selection criteria at first, then analyze the impact for different subgroups to help refine and define the final targeted sub-group. • Nepal/Bangladesh • Nepal proposes rural/urban sub-groups to test how online registration system overcomes geographical barriers. Both IEs could consider including business sector and gender criteria. • Tajikistan • What are the indicators to identify “risky” businesses for auditing? Consider sub-sample analysis for some of these selected measures to improve understanding of which types of businesses are most likely to be influenced by RBA. • SEE Trade • Which sectors are likely to benefit most from the risk-based inspections? Important to consider perishable vs. non-perishable products as well as products with volatile vs. non-volatile market prices.
High-frequency data • Should be exploited whenever available: they can strengthen estimates of important indicators, complement other data sources and are less costly then collected data. They are usually available when administrative data of interest is regularly updated (trade data, daily business registration rates, bank data on loan repayments/default rates, judicial court case data, etc.) • Benin/Bangladesh • Consider exploring high-frequency data (especially Bangladesh, given the regional/municipality based roll-out) and obtain historical and future registration rates for treatment and control in the lowest aggregation possible (e. g. daily). • Kenya • Consider using high-frequency administrative data on health facility performance and satisfaction surveys at the facilities to reduce the cost burden for data collection. • SEE Trade/ECA Food safety • Will need to rely on high-frequency data and trend breaks at the times of intervention effectiveness. A potential challenge is that the time between project effectiveness and awareness by users can be fuzzy. Consider ways to sharpen this by providing an information campaign if the intervention is still to take place. Alternatively try to exploit natural variations in the types of businesses that are most likely to become aware of the intervention most immediately and consider sub-sample analysis on these groups. • ECA Food safety • Expand on the discontinuity of the FTA date. Check historical data on whether trends change before and after discontinuity (due to anticipation) or whether these are stable.
Sampling • A sampling frame to choose survey participants needs to be carefully designed to obtain credible results. Also, when determining the sample size, teams should consider several factors, but mainly outcomes of interest, the expected change in the outcomes and expected take up rates. • Benin • Relies on partner institutions. The risk is that partner institution info may vary greatly in quality. Consider doing an initial screening of lists through direct contact to ensure quality and remove ineligible or closed businesses. Also, consider using tax and other official administrative data to identify informal entrepreneurs. • Nepal • Consider using tax data to identify informal businesses. This will be a specific sub-sample of informal businesses; alternatively conduct HH-level listing in survey areas to identify businesses and compile frame. This will only consider business formalization rather than business creation since firms that do not exist cannot be listed. • SEE Trade • Matching would require a detailed sampling frame of businesses. Is this available? Alternatively, phased approach at ports. • Kosovo • Can we get a big enough sample? Investment incentives may suffer especially from low take up rates, severely limiting the scope for evaluation. If using an encouragement design, think about different ways to derive a sampling frame – for example by consulting the expected outcomes of different intervention variations with local government agencies such as Chambers of Commerce. • Specify the unit of randomization and the overall identification strategy.
“General equilibrium” effect and spillovers • Looking at the bigger picture of the project, and its effects, is important. Measuring only the impact on the target groups may miss important spillovers and complement/substitute effects of the intervention. • Kenya/Tajikistan • Inspection rates are likely to influence behavior in other health clinics and firms. Varying the “intensity” of inspection by changing the rate of inspection for neighboring centers may help to measure this, but sample size may be a challenge. • SEE Trade • How does the change in border processes influence decisions of importers/exporters to move to other border crossings? Will be important to capture regional trade as a whole to make sure any impact being observed is not just a “substitution effect”.
Realistic timelines • Align IE implementation with the intervention timeline (ex-ante IEs are usually better as they allow for tailored data collection). • Identify possible implementation bottlenecks, which might have time and cost implications. • Applies to all IEs.
Next steps • Evaluation teams • Finalize concept note (✔) and start planning for methodology note. • CIC impact team • Acts as the CIC focal point and main interface for IE teams (see Annex for the list of CIC focal points and researchers). • Provide technical support and financial assistance to evaluations in the program to advance further on their design and implementation. • Promote partnership with DIME and external experts that provide feedback to the teams and ensure implementation of the most rigorous methodologies that are feasible. • Engage in a series of follow up activities to increase understanding of most appropriate methods to use when impact evaluation are not feasible and how to use findings from one country in another. • Share knowledge on CIC impact evaluation, strengthen the community of practice, and use website as platform for information, tools, and interaction.
Milestones * Indicative deadline, may vary by IE
