International Food Policy Research Institute Pedro Olinto

1. Using non-experimental methods to evaluate a Conditional Cash Transfer program:The case of Bolsa Alimentação

2. International Food Policy Research Institute • Pedro Olinto • Bolsa Alimentação (Ministry of Health, Brazil) • Eduardo Nilson • London School of Hygiene & Tropical Medicine • Saul Morris • Emory University • Rafael Flores • Instituto Materno-Infantil de Pernambuco • Ana Claudia Figueró • Instituto de Estudos do Trabalho e Sociedade • Alinne Veiga

3. The program • Bolsa Alimentação is a Brazilian federal government program designed to reduce nutritional deficiencies and infant mortality • Beneficiaries • (i) pregnant women, • (ii) mothers breastfeeding a child up to 6 mo • (iii) children of age six months up to seven years • Beneficiary selection: • Municipal quota based on projected number malnourished children • Municipal authorities select beneficiary households • R$15 (US$6.25) /beneficiary /month (up to R$45) • Mother signs “Charter of responsibilities”

4. Evaluation design: main features • Non-experimental • Ex-post only comparison of beneficiary and non-beneficiary households • Non-beneficiary households were registered by local authorities as beneficiaries, but excluded due to: • Data transmission error • “Special characters” in the name • Discrepancy family records Bolsa Escola

5. Evaluation design: main features • Evaluation limited to municipalities • In the Northeast region of the country • With at least 40 excluded households • Where transfers had been made for 6 months • Fieldwork undertaken exactly 6 mo after first transfer in each of the 4 survey municipalities

6. Pairing of beneficiaries and excluded • Pairing separate for each class of beneficiary • Pregnant women • “Lactating mothers” (up to 6 mo post-partum) • Children • For each “would-be beneficiary”, the most similar similar actual beneficiary was identified, based on • Municipality of residence* • Sex* • Socio-economic status (based on reported income; rent; water, electricity & gas bills, family size) • Age

7. Pairing (2) • Pairing algorithm: “nearest neighbor” with calipers • Pairwise distance calculated based on Euclidean distances • Relatively importance of age and SES chosen subjectively • No replacement of beneficiaries • “Excludeds” matched in a random order • Two beneficiaries per “excluded”, to maximize power

8. Problems with the pairing • Turned out that program registers had not recorded all the young children in the hhs • Expected no. under-7s: 1.3 per hh • Revealed no. under-7s: 1.9 per hh • Individual-level pairing not so good for household-level analyses, since same beneficiary hh paired to more than one excluded hh

9. Bolsa Alimentação beneficiary households Excluded households P-value Educational level of women aged 15-49: Incomplete primary 79.0% (646/818) 79.3% (252/318) 0.92 Flooring material: Earth Floor tiles Other (mostly cement) 8.0% (57/716) 10.8% (77/716) 81.3% (582/716) 9.6% (27/282) 11.0% (31/282) 79.4% (224/282) 0.70 Water source: public network 50.5% (360/713) 50.4% (142/282) 0.97 Telephone 9.2% (66/717) 11.0% (31/282) 0.39 Family size, mean (s.d.): Total 0.0– 6.9 y 7.0 – 13.9 y 5.4 (2.1) 1.9 (0.9) 0.8 (1.0) 5.8 (2.3) 1.8 (0.9) 1.1 (1.1) 0.003 0.02 <0.001 Bolsa Escola beneficiary 26.8% (192/717) 55.7% (157/282) <0.001 Duration of exposure to Bolsa Alimentação at the time of the survey, months, mean (s.d.): 5.9 (0.3) N/A N/A Socio-economic and demographic characteristics of Bolsa Alimentação beneficiary and excluded households in the evaluation sample

10. Marginal propensity to consume foodout of Bolsa Alimentação transfers • OLS model: • R$0.70 for every R$1.00 transferred • IV model (controlling for possible endogeneity of number of beneficiaries): • R$0.57 for every R$1.00 transferred

11. Impact on dietary diversity

12. Weight-for-age Z-score Age group: Bolsa Alimentacao Difference Benef’y Excl’ded Unadjusted Adjusted 0.0 - 23.9 mo mean -0.68 -0.40 mean -0.28 -0.25 s.d. 1.11 1.15 s.e. 0.13 0.13 n 315 103 signif P=0.030 P=0.058 24.0 - 47.9 mo mean -0.75 -0.65 mean -0.10 -0.11 s.d. 1.06 1.02 s.e. 0.10 0.10 n 430 141 signif P=0.33 P=0.29 48.0 - 83.9 mo mean -0.77 -0.74 mean -0.03 -0.08 s.d. 1.04 0.90 s.e. 0.08 0.08 n 596 238 signif P=0.68 P=0.31 Total mean -0.74 -0.64 mean -0.10 -0.13 s.d. 1.06 1.00 s.e. 0.06 0.06 n 1,341 482 signif P=0.068 P=0.024

13. Longitudinal analysis

14. Associations between length of exposure to Bolsa Alimentação and weight gain in children aged 0–36 months. Panel analysis based on routinely recorded weight data: random effects model. ------------------------------------------------------------------------------ WEIGHT | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- AGE | .0835284 .0050511 16.54 0.000 .0736286 .0934283 ln(AGE) | 1.941647 .0445841 43.55 0.000 1.854264 2.029031 MALE | .5057401 .0886546 5.70 0.000 .3319804 .6794998 BA_BASELINE | -.1923509 .1034153 -1.86 0.063 -.3950412 .0103395 BA_DURATION | -.0305449 .0074744 -4.09 0.000 -.0451945 -.0158953 BE | -.0201923 .0475438 -0.42 0.671 -.1133765 .0729919 _cons | 3.187013 .1141902 27.91 0.000 2.963205 3.410822 -------------+---------------------------------------------------------------- sigma_u | 1.0507184 sigma_e | .61632495 rho | .74400885 (fraction of variance due to u_i) ------------------------------------------------------------------------------

15. Differential weight gain associated with each month of exposure to Bolsa Alimentação, estimated for children of ages 6, 12, 18, 24, 30, and 36 mo, in a random effects model that allows program effect to interact both with age and the natural log of age ------------------------------------------------------------------------------ weight | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- 6 months | -.0324806 .0137262 -2.37 0.018 -.0593834 -.0055778 ------------------------------------------------------------------------------ 12 months | -.0457275 .0095092 -4.81 0.000 -.0643653 -.0270897 ------------------------------------------------------------------------------ 18 months | -.0402928 .0084215 -4.78 0.000 -.0567987 -.0237869 ------------------------------------------------------------------------------ 24 months | -.0272094 .0088983 -3.06 0.002 -.0446498 -.009769 ------------------------------------------------------------------------------ 30 months | -.009935 .0122145 -0.81 0.416 -.0338749 .0140049 ------------------------------------------------------------------------------ 36 months | .0099903 .0176865 0.56 0.572 -.0246747 .0446553 ------------------------------------------------------------------------------

16. Lessons learnt: methodology • Very detailed understanding of beneficiary selection process resulted in robust non-experimental identification • Success critically dependent on getting into the field at the right time • Individual matching ensured that beneficiary and non-beneficiary groups were very similar • Routinely collected weight data permitted longitudinal analysis, quality much better than expected • However, lack of conventional baseline allows controversy over results to rage indefinitely!

17. Lessons learnt: results • Strikingly high proportion of transfer spent on food, suggesting good targeting • Diversity of family diet improved markedly, as in similar programs • Vaccination coverage not improved because already excellent • Weight gain effect probably illustrates dangers of linking targeting and/or program graduation to nutritional status outcomes • For nutrition, too many beneficiaries over 2 yo, damage likely to be irreversible