The Application of Quantitative Methods in Poverty Impact Evaluation

1. The Application of Quantitative Methods in Poverty Impact Evaluation Regional Seminar on Poverty Monitoring and Evaluation Nanchang, China, May 11-12, 2007 Shahid Khandker, WBI

2. 5 Non-experimental Methods to Construct Data on Counterfactual • Matching • Propensity-score matching • Difference-in-difference • Matched double difference • Instrumental variables

3. 1. Matching • Matched comparators identify counterfactual. • Match participants to non-participants from a • larger survey; • The matches are chosen on the basis of • similarities in observed characteristics (X); • This assumes no selection bias based on • unobservable heterogeneity; • Validity of matching methods depends heavily • on data quality.

4. 2. Propensity-score Matching (PSM) Propensity-score matching match on the basis of the probability of participation. • Ideally we would match on the entire vector X of observed characteristics. However, this is practically impossible. X could be huge. • Rosenbaum and Rubin: match on the basis of the propensity score = • This assumes that participation is independent of outcomes given X. If no bias given X then no bias given P(X).

5. Steps in Score Matching • Representative, highly comparable, surveys of the non-participants and participants; • (ii)Pool the 2 samples and estimate a logit/ probit model of program participation: • Predicted values are “propensity scores.” • (iii)Restrict samples to assure common support; • (iv) Failure of common support is an important • source of bias in observational studies • (Heckman et al.).

6. Density of scores for non-participants

7. (v) For each participant find a sample of non-participants that have similar propensity scores; (vi) Compare the outcome indicators. The difference is the estimate of the gain due to the program for that observation; (vii) Calculate the mean of these individual gains to obtain the average overall gain. -Various weighting schemes.

8. 3. Panel Analysis Where: =impact (“gain”); = counterfactual; = comparison group

9. Difference-In-Difference (i) if change over time for comparison group reveals counterfactual, and (ii) if baseline is uncontaminated by the program,

10. 4. Matched double difference • Matching helps control for bias in diff-in-diff. • Score match participants and non-participants • based on observed characteristics in baseline; • Then do a double difference; • This deals with observable heterogeneity in • initial conditions that can influence subsequent • changes over time.

11. 5.Instrumental Variable method • Identifying exogenous variation using a 3rd variable • Outcome regression: • D = 0,1 is our program – not random • “Instrument” (Z) influences participation, but does • not affect outcomes given participation (the • “exclusion restriction”). • This identifies the exogenous variation in • outcomes due to the program. • Treatment regression:

12. Reduced-form outcome regression: where and Instrumental variables (two-stage least squares) estimator of impact:

13. An example: Grameen Bank impact • Grameen Bank provides financial and non-financial services for low-income people; • It relies on group mechanisms to enforce contract and to reduce the impacts of capital market imperfection and asymmetric information; • Grameen Bank started off with grants and soft money for institutional development. • Is Grameen Bank cost-effective? • Assess its impact on the poor

14. Notation • C is credit demand; • Y is outcome variable (consumption, assets, employment, schooling, family planning); • Subscripts: household i; village j; male m; female f; time t; • Want to allow separate effects on outcome of male and female credit; • X represents individual or household characteristics.

15. Measuring Impacts: Cross-Sectional data using IV Credit demand equations: Cijm = Xijmβc + Zijmγc + μcjm + εcijm Cijf = Xijfβc + Zijfγc + μcjf + εcijf Outcome equation: Yij = Xij βy + Cijm δm + Cijf δf + μyj + εyij Note: the Z represent variables assumed to affect credit demand but to have no direct effect on the outcome.

16. Endogeneity Issues • Correlation among μcjm, μcjfand μyj, and among εcijm, εcijf and εyij • Estimation that ignores these correlations have endogeneity bias. • Endogeneity arises from three sources: 1) Non-random placement of credit programs; 2) Unmeasured village attributes that affect both program credit demand and household outcomes; 3) Unmeasured household attributes that affect both program credit demand and household outcomes.

17. Resolving endogeneity using an IV approach • Village-level endogeneity – resolved by village FE; • Household-level endogeneity – resolved by instrumental variables (IV); • In credit demand equation Zijm and Zijf represent instrumental variables; • Selecting Zij variables is difficult; Possible solution: identification using quasi-experimental approach.

18. Quasi-experimental survey design • Households are sampled from program and non-program villages; • Both eligible and non-eligible households are sampled from both types of villages; • Both participants and non-participants are sampled from eligible households. Identification conditions: • Exogenous land-holding; • Gender-based program design.

19. Quasi-experimental Design (contd.) • Exogenous land-holding criteria: - Only households owning up to 0.5 acre of land qualify for program participation. In practice there is some deviation from this cutoff. • Gender-based program participation criteria: - Male members of qualifying households cannot participate in program if village does not have a male program group. - Female members of qualifying households cannot participate in program if village does not have a female program group.

20. Quasi-experimental Survey Design:Construction of Zij variables Male choice=1 if household has up to 0.5 acre of land and village has male credit group =0 otherwise Female choice=1 if household has up to 0.5 acre of land and village has female credit group =0 otherwise Male and female choice variables are interacted with household characteristics to form Zij Variables.

21. Comparing Results among Alternative Models: Log-log impacts of GB women’s credit on HH per capita consumption

22. Panel Data Analysis: Rationale • Results based on cross-sectional data may not be robust; • Impacts based on cross-sectional data may be short-term; • Cross-sectional data analysis cannot separate credit effects from non-credit effects; • Panel data analysis can assess whether credit has increasing or diminishing returns; • Panel data analysis can assess whether diseconomies or market saturation exists.

23. Concerns with Panel Data Household attrition: • Not a problem if it is random; • Efforts should be taken to minimize it. Households split-off: • As many component households as possible should be traced; • Component households are logically combined.

24. Estimation with Panel Data – Resolving Endogeneity • Household FE resolves both household- and village-level endogeneity. • So IV (quasi-experimental) method is not required for resolving endogeneity. Outcome equation: Yijt = Xijt βy + Cijmt δm + Cijft δf + μyj + εyijt After differencing between two time-points: ΔYij = ΔXij βy + ΔCijmδm + ΔCijftδf + Δεyij

25. FE-IV method over FE method • Durban-Hausman-Wu test was conducted to see if the estimated coefficients of FE or FE-IV model are significantly different. • Results suggest that changes in credit volume used in the FE method are somewhat determined by the time-varying heterogeneity or the measurement errors in credit variables. That means, the FE method may not be as reliable as one would expect. • For comparison, both results are presented.

26. Comparing Results Among Alternative Models: Log-log impacts of women’s current credit on HH per capita consumption

27. Propensity Score method • Run a program participation equation • Predict propensity score for each household • Match non-participant household with participant household • Take the difference of outcomes of interest between participant and non-participant • Weighted mean difference is the average program effect • Use both cross-sectional and panel data analysis

28. Results of PSM with Cross-sectional Data Set: Average treatment effect of female borrowing 1991/92 1998/98 panel • Per capita expenditure: -140.9 455.8* 587.7* • Moderate poverty: 2.30 -5.40* -7.46* • Extreme poverty: 0.30 -6.70* -14.84* Note: * refers to significance level of 10% or better; results shown based on nearest neighbor matching.

29. Conclusions • No method is perfect • Alternative methods are desired to verify results