Qualitative and Quantitative Poverty Appraisal: Maximizing Complementarities, Minimizing Tradeoffs

1. Qualitative and Quantitative Poverty Appraisal:Maximizing Complementarities, Minimizing Tradeoffs Chris Barrett Cornell University August 16, 2003 IAAE Learning Workshop Durban, South Africa

2. Good background resources:- Ravi Kanbur, ed.,Q-Squared: Combining Qualitative and Quantitative Methods In Poverty Appraisal(Delhi: Permanent Black, 2003)- Biju Rao and Michael Woolcock chapter in forthcoming IMF/World Bank Tool Kit on Evaluation Methods

3. Significant recent progress in both qualitative (QUAL) and quantitative (QUANT) methods: - rapid rise of participatory appraisal (PA) methods - emergence of widespread, nationally representative household survey data, including panels.

4. Are QUAL and QUANT complements or substitutes?There is considerable conflict among the practitioners of each … does that mean the methods necessarily conflict too???Importance of being self-reflexive and self-critical

5. Dimensions of QUAL-QUANT Difference:Be clear about focus of question:(1) Data collection methods(2) Data types(3) Data analysis methods(4) Audience

6. Qual-Quant DimensionsData collection methods Analytical Coverage General Specific Census Random Sample Surveys PRA Autobiography Passive Active Population Involvement in Research

7. Qual-Quant Dimensions “Qualitative”“Quantitative” Data types: Categorical Ordinal Cardinal Each data collection method can yield both non-numerical and numerical data types

8. Qual-Quant Dimensions “Qualitative”“Quantitative” Data analysis methods: Inductive Deductive Related to the specific-general data collection methods distinction, there’s often (not always) a difference in analysis methods.

9. Qual-Quant Dimensions Audience: Local community Global/national policymakers QUAL researchers often worry out loud about local empowerment and the intrinsic value of the research process. QUANT types tend to worry about “big picture” or “take home” messages, about “speaking truth to power”

10. Key advantages of QUAL approaches: Allow more immediate probing in response to unanticipated results (adaptability) More nuanced and textured for complex, unmeasurable concepts (e.g., power, opportunity, security) Let subjects speak for themselves More oriented toward uncovering processes/mecahnisms

11. Key advantages of QUANT approaches: Use of sampling frames and randomization reduces inferential bias: coincidence and causality Uniformity/structure in design/definitions fosters replicability over time (longitudinal analysis) and across samples (comparative analysis) Far easier aggregability – few scaling up problems

12. Myths about QUAL-QUANT differences • One more/less extractive than the other (“ethical superiority”) • One more/less contextual than the other (“historical superiority”) • One inherently numerical/non-numerical (“statistical superiority”) (4) One more “rigorous” than the other (“scientific superiority”) Bad practice is bad practice, whatever the method... Key question: when and how is good practice within one strand still wanting? How can the other fill the blanks?

13. Mixing methods When brought together, QUAL and QUANT rarely have similar status, especially in policy discourse, where aggregability and the “illusion of precision” commonly dominate. Improve analysis by mixing the two … taking the “con” out of econometrics, generalizing beyond the “part” of participatory methods

14. Why mix methods? • QUAL can improve QUANT by: • Improving survey/instrument design. Data are social products, so need to understand source • Improving specification of formal models • Improving statistical inference • Identifying suitable instrumental variables, exclusionary restrictions, etc. • Shedding light on outliers (“It helps to have had tea with an outlier” – Biju Rao) • Highlighting likely sources of measurement error (the “Chai stall” error – Ron Herring) • Breathing life into otherwise abstract numbers

15. Why mix methods? • QUANT can improve QUAL by: • Reducing researcher-induced bias by structure and replicability • Facilitating comparability • Facilitating aggregability • Broadening the audience for results • Fostering more precise criteria for demonstrating causal relationships

16. Different methods of mixing “Sequential mixing” or “classical integration” Practitioners of each method do their best with their own tools on same problem, sometimes taking outputs from one as intermediate inputs to another. Then triangulate to get an integrated result.

17. Sequential mixing Example: Understanding welfare transitions Step 1: Panel survey data collection to construct transition matrices and change measures. Poort Nonpoort Poort+1 Step 2: Draw several households from each of 6 cells in matrix Nonpoort+1 and do detailed oral histories. Why? Capture omitted variables, check transitions, 2nd method of inference, problem of identifying thresholds econometrically, value of stories for policy audiences.

18. Different methods of mixing “Simultaneous mixing” or “Bayesian integration” Iterative approach to using one method to inform another, then back to the first, etc., keeping multiple methods interactive throughout the research process to update researchers’ priors continuously. Feedback loop yields a homeostatic research mechanism: “ethnography” precedes “participatory” which in turn precedes “survey” in dictionary … and the in field, too! Ongoing, creative tension between methods helps ensure originality, robustness and relevance of results

19. Different methods of mixing Example: Pastoral Risk Management (PARIMA) project based on multidisiplinary integration: (a) What does it mean to poor or vulnerable in this setting? How does this vary across individuals, households, communities and time? [asking the right questions or the right people at right time?] (b) Derivative from (a), are we measuring the correct variables and in the right manner? (c) Is our inference consistent (i) across methods (a test of robustness) and (i) with local understandings of the problem(s) (a test of relevance)?

20. Tools developed/employed - Participatory risk mapping (Smith et al. WD 2000, JDS 2001) to identify relevant threats (e.g., human health) open-ended, spatially-explicit, pseudo-cardinal - Quarterly repeated surveys with open-ended sections and mixed modules: (i) complex property rights; climate forecasting, resource conflict; land use history; livelihoods strategies, etc. (ii) complementarity at multiples levels of analysis and different methods (e.g., livestock marketing with data from households, markets and traders)

21. Example: Participatory risk maps of rainfall and drought risk (Smith et al. 2001 JDS)

22. Walking On Both Legs Development scholars and practitioners increasingly recognize the complementarity between qualitative and quantitative methods. There are big gains to be enjoyed from relatively small movements along the QUAL-QUANT axes in any of several dimensions. Tradeoffs grow, however, so multidisciplinary mixing seems best, whether sequential or simultaneous, to take advantage of inherent complementarities from diversity of methods.

23. Walking On Both Legs But much remains to be done … Need work on (i) vocabulary (ii) field methods (iii) data cross-referencing (iv) fostering respectful dialogue