Outline of today’s talk

Dana BurdeVisiting Assistant ProfessorDepartment of Humanities and Social SciencesNew York Universitydana.burde@nyu.eduPanacea or Straitjacket? A Qualitative Researcher Reflects on a Randomized Trial in Afghanistan

Randomized Trial Assessing Community-Based Schools in AfghanistanCo-Investigator (for the large-scale randomized trial): Leigh Linden, Assistant Professor, Economics Department and School of International and Public Affairs, Columbia UniversityAgency studied: Catholic Relief ServicesResearch assistants: Matt Hoover, Dr. Saeed Mahmoodi, Nathan Falkner, Nafi Olomi, Nicole Rigg, Amy Kapit. Funders (from the inception of the study): Columbia University Institute for Social and Economic Research & Policy ($15,000); Weikart Family Foundation ($75,000); United States Institute of Peace ($40,000); Spencer Foundation ($40,000—pilot; $483,000—large scale study); National Science Foundation ($100,000).

Outline of today’s talk • Brief overview of arguments: Heated pro/con debate between qualitative researchers and quantitative researchers regarding the uses and effectiveness of randomized trials • Study design: Description of the randomized trial of community-based schools in Afghanistan and the ways in which qualitative study components complemented the quantitative design • Discussion: Design, execution, ethics, and research culture—how these issues affected our randomized trial

The Pro/Con Debate Those in favor of randomized trials say they provide: • Evidence-based findings that show more robust results (than other kinds of studies) • Answers to key questions in education research This position is epitomized by this quotation: “…[randomized controlled] experiments provide a better warrant [for causal conclusions]…than any other method. So if experiments can be conducted in schools they should be. Not to use them requires a very strong justification” (Cook, 2003, p. 117).

The Pro/Con Debate Those who oppose randomized trials accuse them of: • Weakness in the research design and execution of the study; • Ethical shortcomings; and • Promoting a culture of privileging quantitative methods over qualitative. This position is epitomized by this quotation: “…there are formidable limits to the randomized control approach—limits which the proponents of this methodology have not sufficiently recognized. To achieve ‘rigor’ in education research, we need to adopt a more realistic grasp of the problems that plague attempts to implement research results as well as a greater openness to other approaches that offer crucial insights into social processes in organizations. For these purposes, qualitative methods, including participant observation and in-depth interviews, are likely to be the most promising” (Lareau, forthcoming, p. 146).

Argument Based on our study in Afghanistan, I argue that: • Randomized trials are excellent tools for assessing program impact and are underused to study education programs conducted by international NGOs • Strong design and execution of place-based randomized trials can overcome some weaknesses identified by critics • Questions raised regarding ethics can be addressed and should not preclude using randomized trials • In agreement with the critics, I argue randomized trials should not eclipse other forms of research; they are not appropriate for every impact study; qualitative methods can address their weaknesses and complement randomized trials to great effect

Overview of Study The study: • investigates whether community-based education programs improve child welfare by comparing villages that receive schools to villages that have not yet received them and by comparing children in these villages. • examines attendance/enrollment, academic achievement, child labor, and social benefits (protection, social networks). Phases: • Qualitative pilot study: Tested field procedures 2005-2006 • Randomized trial 2007-2008: Worked with a large international US-based NGO, Catholic Relief Services (CRS), to randomly assign their community-based schools to eligible villages (i.e., villages that had none previously) • Complementary qualitative case study of program

Program: Partnership for Advancing Community Education in Afghanistan (PACE-A) • PACE-A is a five-year, $24 million USAID-funded program: • meant to “expand quality learning and life opportunities for marginalized communities and their children in Afghanistan” (PACE Summary, 2006, p. 1) • Consortium of 4 NGOs tasked with providing hundreds of community-based schools in 20 provinces across Afghanistan • The CRS portion of the program was intended to reach approximately 100 villages by the end of 2008 • Schools were to be phased in over several years

CRS Community-Based Schools CRS selects districts and communities according to: • Security; Availability of teachers; Level of community interest; Community willingness to mobilize resources (teachers’ salaries); Support from the Ministry of Education (agreement); Community willingness to provide space for the school (room in a home or mosque). • CRS provides training to teachers, materials for the classroom (government curriculum), and regular monitoring to track progress over time.

Study Hypotheses Hypothesis 1 Attendance/Enrollment: Children living in a village that receives a CRS school will be more likely to enroll in school and will attend more regularly than children in other villages. Hypothesis 2 Learning: Children who live in villages that receive a CRS school will perform better on academic achievement tests on math and language skills than children in the villages that do not receive a CRS school. Hypothesis 3 Indirect Social Outcomes: Children who live in villages that receive a CRS school will report higher levels of social integration, higher levels of security, and higher levels of safety than children in other villages.

Original Study Design

Actual Study Design

Qualitative component Case study of the CRS program to complement the RT and learn more about: • program implementation; • place, i.e., country, province, districts, and villages in which we are collecting data; • institutions that are part of our study or influence it: i.e., households, families, schools (government and CBS), mosques, SMCs, village shuras, ministry of education. • individuals that are part of our study or influence it, i.e., children, parents/heads of households, teachers, arbobs, CRS education staff members, CRS managers, PACE-A directors

Map of Afghanistan

Ghor Province, Afghanistan

Ghor Province

Discussion: Design Criticism: Addressing only narrow questions misses significant variables in education Our study: “Schools” constitute a significant variable Criticism: Extent to which variables can be manipulated accurately, e.g., networks Our study: Intent was not to manipulate networks, but rather to measure the impact of schools on networks, and other secondary effects Limitation: We can say that the program did not have an impact on friendship networks, but need qualitative methods to say why

Discussion: Execution Criticism: Adherence to treatment; “contamination” of control by treatment Difficulties our study faced in execution: • Persuading CRS to carry out a randomized trial • Avoiding contamination • Preserving the integrity of the randomization: • Communication breakdowns (sample size decreased) • Inclement weather (delayed start to survey) • Security (attacks on staff, tribal warfare, etc. sample size decreased to 31 villages) • Hiring and training survey staff

Discussion: Ethics Criticism: Unethical to deny treatment Ethical issues our study faced: • Sample selection: “Needs assessment” versus randomized selection • Temporary denial of treatment: Program was intended to be phased in over time, regardless • Program benefits not clear given the basic program intervention

Discussion: Ethics Additional ethical issues our study faced: • Protection for the researchers was a far greater problem than the protection of subjects • IRB was silent on this point • In fact, IRB protections for subjects put researchers at risk: • Confidentiality • Signed documents

Discussion: Research Culture A culture of preference for randomized trials pervades among some research circles. Qualitative methods are invaluable in addressing weaknesses of randomized trials in several ways: • A strong qualitative pilot study enhances the randomized study design and increases the probability of successful program intervention as well as successful execution of the study. • Qualitative methods used in conjunction with a randomized trial counteract the tendency of randomized trials to focus on narrow cause and effect relationships. • Complementary qualitative case study and ethnographic methods provide more confidence in quantitative data analysis, interpretation, and provide additional detail to explain findings that randomized trials cannot.

Conclusion Randomized trials -- crucial method to test multiple empirical questions across the social sciences. Critics have valid concerns -- randomized trials should not be allowed to eclipse qualitative methods. As in all research, ethical questions raised by randomized trials must be taken seriously. Properly designed and executed, randomized trials can produce robust and significant data even in the most difficult circumstances.

Outline of today’s talk