Individual vs. Group Randomized Trials

1. Individual vs. Group Randomized Trials Jens Ludwig University of Chicago, Brookings Institution and NBER

2. Three things to consider • Realizing randomization • Power • Nature of the intervention

3. Realizing randomization • “Should I randomize at the individual or the group level?”

4. Realizing randomization • “Should I randomize at the individual or the group level?” • Sort of like asking: • Should I take my $30 million Powerball prize all at once, or spread it out in installments?

5. Why are so many American public schools not performing better? • Four major hypotheses: • Inadequate resources • Inefficient production technologies (curriculum, etc.) • Unmotivated teachers • Unmotivated students • Education researchers usually can only randomize what they can pay for • Severely limits which of these we can study with randomized experimental designs of any sort

6. Keep our eye on the prize • Our real goal has to be to convince governments to do more randomization • Tennessee STAR • Progressa • Even Roland Fryer w/ Eli Broad at his back is taking on just a modest slice (pay for grades) • First consideration: Choose unit to randomize you need to in order to be able to randomize

7. Me: “Can we randomly assign the intervention?” City official in very large Midwestern city: The rhetoric of randomization

8. Me: “Can we randomly assign the intervention?” City official in very large Midwestern city: “No way.” The rhetoric of randomization

9. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” City official in very large Midwestern city: “No way.” The rhetoric of randomization

10. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” The rhetoric of randomization

11. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” The rhetoric of randomization

12. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” City official: “Good question.” The rhetoric of randomization

13. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” Me: “Could we flip a coin, which would be the fair thing to do?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” City official: “Good question.” The rhetoric of randomization

14. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” Me: “Could we flip a coin, which would be the fair thing to do?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” City official: “Good question.” City official: “Ah, now I get it.” The rhetoric of randomization

15. The rhetoric of randomization • Never use term “randomized experiment”

16. The rhetoric of randomization • Never use term “randomized experiment” • Acceptable talking points: • “Pilot program” • “Excess demand” • “Fair, random lotteries” • If there would be a natural unit for doing “regular” pilot program, randomize that • I.e., we’d just be implementing an unusually informative pilot program • For many education interventions would seem to argue for group randomization (ex: pay for grades)

17. Second consideration:Power • There is the standard statistics version: • More power from 1,000 kids distributed across 1,000 schools than 1,000 kids distributed across < 1,000 schools • Due to non-independence of student observations within schools

18. Then there is the real-world version of this issue • We live in a resource-constrained world • Are there economies of scale in data collection? • Cluster randomization could reduce data collection costs for same reason that population surveys use two-phase sampling • Are there economies of scale in delivering the intervention itself?

19. For a given budget, cluster randomization could in principle generate more power • Imagine rapidly declining marginal costs of “treating” and studying kids w/in a school • Suppose school-based self-administered student and teacher surveys plus use of student administrative school records • Suppose we have a DARE-like intervention (“Don’t do drugs kids, look what happened to me!”)

20. Power considerations in a resource-constrained environment • It’s conceivable you could get more power out of a clustered sample of given size N, even with non-trivial intra-class correlations (ICCs) • Significant information requirements: • Need to know ICCs for your sample & outcome • Need school system to help you think about average cost & marginal cost schedules for intervention • Need very good survey subcontractor to help you think about economies of scale in data collection

21. Power considerations • On the other hand, you can run out of observational units quickly in clustered experiments • Imagine randomizing schools: • Even in Chicago, “just” 116 high schools, 483 elementary schools • Imagine half of schools meet your eligibility criteria, then half of principals agree to cooperate in experiment, then you randomize half to T and C • That would be 14 treatment high schools, 14 controls • Or 60 treatment elementary schools, 60 controls

22. Nature of the intervention • Previous two issues are shamelessly pragmatic • Individual vs. group choice could also hinge on substantive considerations

23. Spillover effects • Stable unit treatment value (SUTVA): • Your treatment effect is independent of how many others get treated • If violated, then your treatment effect estimates generates only to situations with similar take-up rates • Sometimes you want to study intervention independent of these spillovers, while sometimes spillovers key part of treatment

24. Spillover effects example:Moving to Opportunity (MTO) • Initial concerns: • Generic SUTVA concern • Groups of public housing families moving in to new areas might generate backlash • Also didn’t want families to recreate any unproductive baseline social ties • New concerns: • Families lose access to productive social ties, so should we have randomized in groups?

25. Spillover effects example number 2 • Roland Fryer, paying kids for grades in Chicago, DC, NYC • Wants to change whole school climate around academic achievement • Generic adolescent (American?) anti-intellectualism • Plus specific “acting white” concerns • Only by paying everyone (or offering to pay everyone) can you change peer norms (or try to change peer norms)

26. Interventions as public goods • MTO suggests neighborhood safety key factor for parental mental health • Maybe for kids, too • Could also affect learning in other ways, too • If you reduce crime in neighborhood, every kid in neighborhood will benefit • This is sort of another way of talking about economies of scale in providing intervention

27. Bottom lines • Clustered experiments might help realizing randomization • Power considerations complex in a resource-constrained environment • Research community needs to develop infrastructure to meet informational requirements for decisions • Substantive considerations about role of spillovers and “public good” interventions • ‘Tis better to have randomized at the wrong level than to have never randomized at all?