Validating Ex Ante Impact Evaluation Models: An Example from Mexico

Francisco H.G. Ferreira Phillippe G. Leite Emmanuel Skoufias The World Bank PREM Learning Forum-April 22, 2008 Validating Ex Ante Impact Evaluation Models: An Example from Mexico

Introduction • Conditional Cash Transfer (CCT) programs are becoming an important element of social policy in LAC • Distinguishing characteristic of CCTs: • social accountability supported by rigorous impact evaluation (IE)

Introduction • Alternative IE designs: • Experimental design: hh randomly assigned to T and C groups, prior to implementation of program. Typically hh surveyed in baseline and for 1 or more rounds after the start of the program. • +: provide most reliable estimates (gold standard) of program impacts • -: costly, likely to be of small scale • -: large time lags involved

Introduction • Quasi-experimental designs: typically comparison/control hh are obtained ex-post (after the start of the program), attempting to equalize selection bias between treatment and control groups • +: less costly • -: lack of baseline data (and/or pre-program differences) • Overall, ex-post methods do not provide ANY information about the possible effects of the program prior to its implementation.

Introduction • Ex ante methods: simulate the effects of the program on the basis of a structural (or reduced form) model of household behavior • Easily implemented using a representative hh data set (e.g. BFL, 2003) • Expand the set of policy-relevant questions that can be addressed, e.g. useful in designing the program, size of transfer, etc. • Based on the concept of treatment and comparison/counterfactual group • However, require some strong assumptions about: • Functional form • Perfect implementation of the program • Absence of time or trend effects.

Introduction • This paper is one of the first to provide a validation test of the ex-ante evaluation methodology • Approach: Use household survey data from two CCT programs (PROGRESA in Mexico and BDH-Bono de Desarollo Humano in Ecuador) where experimental designs were employed to (ex post) evaluate program impact • Use the baseline data from each survey to apply ex-ante evaluation methods to predict program impact.

Introduction • Compare the impact predictions obtained with the ex-ante method to the impact estimates obtained using the experimental (ex-post) methods.

Some Background on PROGRESA • What is PROGRESA? • Targeted cash transfer program conditioned on families visiting health centers regularly and on children attending school regularly. • Cash transfer-alleviates short-term poverty • Human capital investment-alleviates poverty in the long-term • Started in 1998. By the end of 2004: program (renamed Oportunidades) covered nearly 5 million families, in 72,000 localities in all 31 states (budget of about US$2.5 billion). • Transfers given to mothers: 20% of hh consumption expenditure

Some Background on PROGRESA • Two-stage Selection process: • Geographic targeting (used census data to identify poor localities) • Within Village household-level targeting (village household census) • Used hh income, assets, and demographic composition to estimate the probability of being poor (Inc per cap<Standard Food basket). • Discriminant analysis applied separately by region • Discriminant score of each household compared to a threshold value (high DS=Noneligible, low DS=Eligible) • Initially 52% eligible, then revised selection process so that 78% eligible. But many of the “new poor” households did not receive benefits

Ex ante model: BFL • Why BFL instead of Attanasio, Meghir et Santiago (2005) ou Todd et Wolpin (2005)? • Simplicity since dynamic Ex ante models as AMS and TW are data intensive depending on panel data. • Is a behavioral model based on four key assumptions: • Do not model household behavioral, i.e., do not debate who makes child’s decision; • Adults are unafected by children’s choice; • Siblings interaction are ignored; • Household composition is exogeneous

Ex ante model: BFL • The model • Child’s occupational choice • (0) Not going to school; • (1) Going to school and paid work; • (2) Going to school and non-paid work

Ex ante model: BFL • The model • Child’s contribution to income in each state 0, 1 and 2

Ex ante model: BFL • The model • Child (household) i chooses the alternative that yields the highest utility

Ex ante model: BFL • The model • Child (household) i chooses the alternative that yields the highest simulated utility

Ex ante estimator • Average Intent to Treat effect (AIT) which provides an estimate of the average impact of the availability of the program to eligible households (in treatment communities) by simulating impact of the program on the sample of eligible age group of children; • Assumes good implementation of program • Attention: Ex ante model is static, i.e., no time or trend effects. • So, it is best to compare AIT (ex ante) with AIT (ex post) obtained using 2DIF (which removes the trend effect from the estimated impact) whenever is possible.

Results: PROGRESA

Conclusion • Ex Ante model analysis indicates so far that they can be very useful as well as powerful in predicting program impacts. • But work is still in progress. • Useful for simulating the design or re-design of a transfer program. • Increasing demand from governments as Panama, Jamaica and Ecuador

Validating Ex Ante Impact Evaluation Models: An Example from Mexico