Tanguy Bernard, AFD Ruth Hill, IFPRI

1. External validity:What role for short-cut impact assessment?(Mixing types of trees to see the forest) Tanguy Bernard, AFD Ruth Hill, IFPRI

2. Doing what worksSeing the forest and picking the trees • Banerjee and He (2008) • List projects that have been shown to work (with ‘internally valid’ studies). • Argue to scale them up at global level before anything else is done. • Yet, things proven effective somewhere may not be elsewhere (and reversely) • Greenberg et al. (2003) US: welfare programs have different results across sites • Attanasio et al. (2004) Mexico: Impact of Progresa 3 times larger in richer states. • Differences likely greater across countries. • Internal validity: Make sure that Δx caused Δy • External validity: Δx in similar circumstances should also lead to Δy. • Unknown: what is meant by similar circumstances. Need to know how impact varies with • Type of environments (1) • Program modalities (2) Risk: missing the forest for the tree(Bardhan, 2005)  This paper : one way to raise the number of trees (with other type of trees) • and Interactions of (1) and (2)

3. Using existing trees (1): Same objective, diverse approaches Cost per additional year of school Source: Duflo, 2009 (lesson)

4. Using existing trees (2):Same approach, various environments CCT projects And: some are pilot, others are national; some are rural, others are urban, some have strong monitoring and associated sanctions, others don’t etc.

5. Using existing trees (3):Deriving general lessons CCL: while generally consistent with human capital theory, some evidence of peer effects and time inconsistent preferences.

6. If we had more of these trees…Meta-regressions • With large enough number of RCTs, one can estimate a regression. • E : Vector of context characteristics • Pre-intervention level of outcome • Site characteristics (e.g. rural/urban, drought prone/moisture reliable) • Period characteristics (e.g. economic growth) • P : Vector of project characteristics • Targeting (e.g. gender targeting, geographical targeting) • Intensity (e.g. per capita size of project, scale: national or local) • Modalities (e.g. free vs co-payment, type of condition, who implemented) • Problem : need enough (project-level) observations to run regression • Replication problems with RCTs (public goods, resources, publications etc.)

7. There are other type of treesProject evaluations • Habicht and Vaughan (1999): • Adequacy: « Did the expected change occur » • Plausibility: « Did the program seem to have an effect above and beyond external influence »? • Probability: « Did the program have an effect with probability <x% » • Donors collect independent information on projects effectiveness • WB: 25% (~70 / year) evaluated by non-project staff (6 weeks with field mission) • EBRD: 76% projects independently evaluated since 2003 • UNDP: All projects greater than $1 million until 1999 independently evaluated • ADB: 40% independently evaluated • AFD: all projects evaluated by external consultants, in field (~70 / year) • … • Different type of trees… • Impact most often appreciated, no specific data collection • Sometimes scaled (satisfactory, non-satisfactory etc.), sometimes not. • Generally low use.

8. There are other type of treesRecent meta-evaluation

9. There are other type of treesMeasurement error problems (cf. Hyslop and Imbens, 2001) • Classical: Error is independent from true value (CME) • E.g. error due to imprecise measurement tool • Optimal prediction error (OPE(1)): error independent from reported value • Agent reporting the data is fully aware of the imprecision of his/her tool. Agent reports his/her best estimate, given his/her information set. • Critical:agent’s awareness. • If he/she is aware of not having the exact information, he/she will understand the question « what is the value of X » as « what is your best guess ». • Knowing it helps infer the type error and associated bias. • Important: correlation of errors in outcome variables and independent variables. Lower bias if no correlation

10. There are other type of treesMeasurement errors and biases If no correlation between measurement erros in dependent and independent variables Source: Hyslop & Imbens, 2001  First best: no bias  Conservative position: avoid the « away from zero »  Avoid correlation of errors b/w outcome and regressors

11. There are other type of treesWhat type of errors could we have? • Outcome variables • Mainstreaming: no bias in the projects selected to be evaluated, and raise number of observations • Comparability (across projects, across agencies): all measures carry same meaning • Credibility: independence and trained to the typical problems of impact assesment (missing data)  OPE(1) • Independent variables • Initial level through administrative statistics ( CME) • Context typologies (e.g. development domains (Chamberlin, Pender, Yu, 2005), micro regions (Torero, 2007) ( CME) • Project design: no error

12. Mixing the trees… • Use RCTs to test unbiasedness: is significant? • Weak correlation between type and project is necessary if parameters are to be identified.  A number of randomly selected projects to be evaluated with RCTs. • Use RCTs to test prediction performance of meta-analysis  Requires that a large number of RCTs be implemented as well.

13. Recommendations • Systematic evaluation of projects • Harmonize outcome indicators • Harmonize with RCTs • Train evaluator to problem of impact evaluation (missing counterfactual) and encouragment judgement-based corrections • Define ‘quality’ standards • Harmonize project-level indicators • Global dataset of environmental indicators • Centralize information

14. Conclusion • Pure statistical learning unlikely. But with somewhat of a theory of what similar means, meta evaluations can be used for tests.  One of the tools towards external validity • Short cut evidence can be used to raise the number of observations, although at cost of potential biases • Under certain conditions, biases may be well understood, and meta-evaluation results informative. • Overcoming these biases requires coordination. Thank you