1 / 23

Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial

Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial. Presentation by Manoj Mohanan, Duke University. Commercial Break! A word about COHESIVE-India. COHESIVE-India : Collaboration for Health Systems Impact Evaluation in India.

wilmer
Download Presentation

Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation Designs in the IMATCHINE project: Regression DiscontinuityCluster Randomized Trial COHESIVE-INDIA Presentation by Manoj Mohanan, Duke University

  2. Commercial Break! A word about COHESIVE-India • COHESIVE-India: Collaboration for Health Systems Impact Evaluation in India. • Jerry La Forgia (World Bank) • Grant Miller (Stanford U. & NBER) • Manoj Mohanan (Duke U.) • Marcos Vera-Hernandez (U. College London & IFS) • Focus on evaluation in health sector using a combination of quasi-experimental and experimental methods • Provide critical input into the design of policies and interventions, to provide rigorous evidence on how to improve performance as part of evaluation. • Collaborating with SAMBODHI and DFID-India on IMATCHINE project COHESIVE-INDIA

  3. Key Question in Impact Evaluation • What is the counterfactual? What would have happened if this program did not exist? • Identifying CAUSAL effects is the central to impact evaluation • Two common fallacies in making causal inference: • Cum hoc ergo propter hoc • Post hoc ergo propter hoc

  4. Cum hoc ergo propter hoc Source: http://ssgreenberg.name/PoliticsBlog/2009/04/03/diversion-highway-fatalities-and-lemons/

  5. The post hoc fallacy Observe people on the street @ 9AM to predict weather? In medicine, there is a disease progression, hence you can use temporal changes to make causal claims. In social sciences, less so.

  6. Visualizing problems in evaluation Participants Counter factual OUTCOMES --------------------------- TIME ------------------------------------- This is why the Before-After method is sometimes called The “Counterfeit Counterfactual” method!!

  7. Outline of Today’s Presentation • Projects being evaluated in the IMATCHINE project • Gujarat: (Regression Discontinuity) • ChiranjeeviYojana (CY) • Karnataka: (Experimental Evaluation) • ThayiBhagyaYojana (TBY) COHESIVE-INDIA

  8. Gujarat: ChiranjeeviYojana (CY) • Introduced in 2005 • Response to acute shortage of OBGYNS in public sector • Leveraging presence of private providers in rural areas • Pays approx Rs. 1700 to accredited provider per delivery • Eligibility: BPL card holder or BPL eligible (~23% of population; total pop 55 million) COHESIVE-INDIA

  9. More on CY • First introduced in 5 backwarddistricts 2005-2007 and then rolled out across the state Jan 2007-08 onwards • B/w 2005 – Feb 2008, CY had covered over 165,000 deliveries provided by 852 providers • Claims: (in 2009) • Has increased institutional births from a national average of 57% to over 80% • Has reduced MMR & IMR • Won WSJ Innovations Award & is now widely looked upon as the “model” COHESIVE-INDIA

  10. Previous ‘Evaluations’ of CY Source: Mavalankar, D. et al. 2009. Saving mothers and newborns through an innovative partnership with private sector obstetricians: Chiranjeevi scheme of Gujarat, India. International Journal of Gynecology and Obstetrics 107: 271–276. • Have typically used data from CY facilities to extrapolate estimated utilization and health benefits in the population COHESIVE-INDIA

  11. Gujarat: Regression Discontinuity Evaluation Design • The program uses the BPL line as the eligibility criteria, which allows a quasi-experimental RD design • Since the BPL score is continuous, households immediately next to each other across the BPL line are comparably similar to each other • Discontinuity in program eligibility across the BPL line allow us to test for differences in outcomes that can be attributed to the program COHESIVE-INDIA

  12. Gujarat: RD Design …contd…Two main challenges with BPL criteria: • Lots of other state run programs use BPL eligibility (such as JSY, food subsidies),: • SOLUTION: We use a “Difference-in-Difference” framework by relying on timing of introduction and expansion of the program to 5 districts in 2005 and all over the state in 2007 to try to identify program effects of CY. • Manipulation of BPL criteria, resulting in misclassification • SOLUTION: We rely on a “Fuzzy” regression discontinuity strategy, where we calculate the “true” eligibility and then instrument for CY participation using simulated eligibility COHESIVE-INDIA

  13. Gujarat:RD Analysis • First Stage: (instrumenting for CY Participation) • Second Stage: (Change in avg. outcomes) • Key outcomes: • Rates of institutional delivery • Study is not powered to detect effects on IMR or MMR, but we will collect data on these measures anyway in addition to measures of morbidity that are more common. Also will collect data on HH characteristics COHESIVE-INDIA

  14. Gujarat: RD Analysis using the Geographic Spread • Accounting for CY roll out in time and space (2nd difference estimate) • Second Stage: (change on avg. outcomes : same equation accounting for roll out) COHESIVE-INDIA

  15. BREAK B/W RD & CRT COHESIVE-INDIA

  16. Experimental Methods • The Randomized Controlled Trial • Clearly, the gold standard • An obvious solution to the ProgEval problem, although not always practical: • Assigns treatment in a manner that is unrelated to outcomes • Two important steps in randomized evaluation • Randomly selecting potential participants from population • Randomly assign treatment to the group • The RCT in IMATCHINE Project…

  17. Karnataka: ThayiBhagyaYojana (TBY) • New Conditional Cash Transfer program of Rs. 1000 for women who prefer to give birth in the private sector • Two components • (1) Prospective evaluation • Since the program uses a BPL eligibility like the CY program, we will use a RD based method, combined with a difference-in-difference • We are working with the Govt. of KN to implement a baseline survey COHESIVE-INDIA

  18. KN : Experimental Evaluation of Provider Incentives (2nd component) • Cluster randomized trial of incentives for providers to estimate effect of incentives for improvements in process measures v/s outcome measures • Arm 1: Provider incentives evaluated based on performance on quality of care indicators • Arm 2: Provider incentives evaluated based on improvement of MCH outcomes in catchment area population • Arm 3: Control group, with no incentives COHESIVE-INDIA

  19. KN Incentives Experiment: Design Issues • 180 rural clusters, in 3 groups of 60 each • Each cluster is approximately at the level of the HOBLI (called kasba in N. India) • On average: 3 OBGYN providers in each cluster • Sample for the study: approx 550 doctors and 18,000 households (100 women who have a baby in catchment area of each cluster) COHESIVE-INDIA

  20. Key challenge from the RCT perspective • Recall the two important steps in randomized evaluation • Randomly selecting potential participants from population • Randomly assign treatment to the group • The second one is relatively easy – just write a STATA code • The first one is the big challenge • Need to identify providers whom we can include in the study • Both a conceptual and logistic challenge • Need to define eligibility based on objective of the experiment COHESIVE-INDIA

  21. KN Incentives Experiment: Design Issues …contd… • There are three key issues related to measurement in this study: • Definition of clusters and catchment area • Measurement of process measures of quality and health outcomes in a population • Identification of women who have had a baby and interviewing them in time COHESIVE-INDIA

  22. Measurement Issues – contd.. • Some things to keep in mind: • Sample Sizes in Cluster Randomized Trials … the devil called Intra-Cluster Correlation • Careful attention to defining eligible population – mapping is VERY effort and resource intensive, but has HUGE pay offs • How practical is it to do RCTs? • “Politically Robust Randomization” – Gary King et al. • Policy Relevance…. • Very important topic, but we don’t have time for this today. • Working in conjunction w state governments to ensure buy-in and policy impact. • Findings from our research in 2011. COHESIVE-INDIA

  23. Thanks • 3ie • Government of Gujarat • Government of Karnataka • DFID • World Bank • My colleagues at Sambodhi • For further details on project contact: • Ms. Manveen Kohli, Project Manager, IMATCHINE, imatchine@gmail.com COHESIVE-INDIA

More Related