Recruiting Sites and Participants for Random Assignment Studies

1. Recruiting Sites and Participants for Random Assignment Studies IES/NCER Summer Institute June 25, 2007 Fred Doolittle

2. Overview of the Session Topics • What is the purpose of site and participant selection? • The art of recruiting sites for random assignment studies • Topics to address • Building the foundation for longer term success of the study • Addressing the tough issues in advance • Assuring cooperation in study implementation

3. The Goal of a “Fair Test” of the Intervention/Program • Some elements of the “fair test” • Sites that serve the intended target group • Sites that can operate the intervention reasonably well • Sites that can provide the intended service contrast • A research design that can produce credible findings on impacts and can be implemented • A data collection plan that preserves the good impact design and documents differences in services and outcomes at the various stages of the theory of action • An analysis strategy and report process that produces useful findings • In various ways, all of these involve the “sites”

4. Topics for a conversation • The process of recruiting sites • Embedding a random assignment design and data collection into normal program operations • Asking the tough questions • Real understanding of the design • Political issues • Ethical concerns • Working to achieve long-term cooperation in study implementation • Lack of crossovers and withdrawals • Data collection

5. Reading the Site Recruitment Context

6. Factors to Consider • How many sites do you need? • What restrictions are there on your choices? • Geographic- need to be close to you, need to be a specific places because of funder, need to test in specific kinds of places, need to be representative • Programmatic- • do sites need to be doing something on which you build • not doing something so get service contrast • Size – need to be big enough to provide the desired sample of “clusters”, teachers and/or kids

7. Factors (cont.) • Are you offering something new or wanting to study existing programs? • Does what you offer fit into their existing plans programmatically and operationally so easy sell? • After-school example • Reading Professional Development example • Is the existing program long established and popular? • Is it oversubscribed so there is “scarcity?” • How have selection decisions been made? • Does it have outcome-based performance standards? • Others????

8. Recruiting Framework • What is the benefit-cost balance in participating from the perspective of sites? • Possible Benefits- the potential benefits for kids because of new services, staff professional development, the visibility of being in a study, site-level findings, etc • Possible Costs – the need to devote management time to the intervention and study, a divergence from local priorities, hassle of integrating the intervention into local operations, controversy of random assignment, operational hassles of random assignment, the hassle of data collection, etc.

9. Responding to the Site Recruitment Context • The context affects the sequence of topics • The more it looks like a tough sell, the more you lead with benefits of participating • The more it looks like an easy sell, the more you move quickly through the benefits and into the roles and responsibilities • In either case: • The goal is to get folks interested in participating so they can help you solve the inevitable problems, find ways to lessen the costs, and identify locally-relevant benefits • Always include the benefits in any material because it may be passed around to new folks who need to understand why to bother with this.

10. Examples of Strategies • Building site “recruitment” into special program funding decisions • Still need engagement process • Selecting “representative” sites • When might this be appropriate? • What is the pitch? • Using data bases to identify prospects that fit the profile and then making “cold call” contact • Data bases may not include key criteria • With whom to make the contact? • What is the pitch? • Working through service provider or funding networks to gain access • Pros and cons • What is the pitch? • Substantial outreach to build demand and stimulate “applications” • What is the balance between benefits and obligations in initial outreach? • How orchestrate the application and selection process?

11. What to Expect When Make Contact In districts/schools/programs: • Everyone is overburdened and stressed • Many things are changing simultaneously • A new intervention is of greater interest than the research • Getting chosen for a study and/or getting something free might be regarded as significant--but not for long • Evaluation results –either overall or site-specific - often do not matter much because way off in the future

12. Recruiting Process • What is the point of entry? • The pros and cons of starting “high” in an organization • Recovering from an initial rejection • First dates (i.e., meetings) are pivotal • Make a “good impression” by bringing the right info and people • Get the right people from the site in the room • Build relationships from the outset • Be the buyer and the seller—explain the benefits but ask the tough questions relatively early • Try to understand their perspective • Be willing to dig in to details

13. “Embedding” Random Assignment into Normal Operations

14. Why Does This Matter? • Your goal is to mesh the research procedures with their operations • If you do this well, it significantly reduces the cost to a site of participating • If you get them interested enough in the possible benefits, they will start helping you figure this out

15. Understanding Normal Program Operations • How does the “sample” usually get to the place (program, school, classroom, etc) where you want to introduce the intervention and do random assignment? • Ask managers and line staff • Focus on understanding the usual way and any exceptions, alternative routes in, etc. • Understand the timing of the steps and the information that is available at each step • Understand the frequency of later “corrections” and how done • Can be because of mobility of students, late hires, turnover • Cannot “fix” this after random assignment

16. Understanding Normal Program Operations (cont.) • For the kids: • Do kids apply? If so how and when? • If not, how are appropriate kids identified? Who is involved? Counselors, other teachers, referral agencies? Do they expect all they refer to get in? • Who decides which kids are “accepted”? • When are these decisions made? • What data is normally available or collected on the kids at various stages? • What percent of those selected actually participate?

17. Understanding Normal Program Operations (cont.) • For the staff: • How and when are staff assignments made? • What is the usual skill set of staff? • What other responsibilities do these staff have that might affect your ability to train on intervention? • Are there other constraints, for example union rules? • Matching kids with staff and services • How and when is this normally done? • Is there a tradition of special cases – parents, staff requesting changes? • Is there a tradition of deciding some kids “just have to have” this staff person, program services, etc. and these are handled outside the usual process?

18. Normal Program Operations (cont.) • Ask about information and time available at key stages • What info is available on kids at different stages – to see if eligible/appropriate for intervention or to serve as baseline data? • Is there time to introduce informed consent and baseline data collection prior to random assignment? • When will any research delays be a problem – because scheduling complicated, because teachers/kids/parents need to know what is up, etc. ?

19. Examples • Individual random assignment • After-school services • Upward Bound • School level random assignment • Elementary school teacher professional development in reading through summer institutes and coaching • Middle school teacher professional development in math through summer institutes and coaching

20. Focusing on Service Receipt • Two parts to this are key because the service contrast is what drives impacts • Program group: Want strong implementation of the intervention to intended kids, with intended participation • Worry about mobility of kids and turnover of staff • Control group: Want clear service contrast for the control group • Not necessarily no service • But clearly different services

21. Picking the Point of Random Assignment • How close to the start of services do you put the lottery? • Choice affects: • the question you address, • the sample, • the baseline data you will have, and • service participation for program and control group • The tradeoff: • “Late” means greater participation rate for the program group but also more hassle for staff, harder to plan, more disappointment if not selected, and -usually - more motivated control group so more services for controls • “Early” means the opposite

22. Examples • Individual level random assignment • KIPP middle school evaluation • Mentoring programs • Cluster random assignment • The previous professional development examples

23. Setting Random Assignment Ratios • This can be part of the site recruitment discussion • A balanced design is best for power, but not necessary • The power drop-off is not major until move past 2:1 • An unbalanced design can make a big difference to sites and help address ethical concerns • A very unbalanced design with many sites is an option if data collection costs are manageable • Avoiding empty program slots is important • “Overbook” somewhat the program group • Use a non-research waiting list to fill empty slots • In extreme, can set program group size to fill available program slots and the control group is the rest

24. Handling the Sensitive Cases • Much better if do this prior to random assignment • Tell programs if absolutely necessary they have a few “chips” they can use prior to the lottery for cases where they could not live with control group designation • They apply to use a chip for a person or cluster before random assignment • Affects generalizability but not internal validity • In cluster random assignment, can include some “extra” non-study staff from program clusters in program services • Sometimes have to accept a “crossover” to save a study • Small numbers in a large sample do not matter a lot

25. Don’t let post-lottery procedures undo the “randomness” • Post-random assignment allocation of kids to clusters • Compare school-level random assignment to teacher-level random assignment • In the latter, important to get commitment of kids to clusters or a clear process to do it before conduct random assignment

26. Avoid Undoing Randomness • Data collection problems • Low rates for sample • Differential rates across research groups • Build tracking into design • Try to mesh data collection with normal participation in services • Create incentives • Cover the costs at a minimum • Provide positive incentives- there are OMB limits • Special incentives for control sample? • Clusters • Individuals • Pay for a local study liaison to coordinate data collection

27. Asking the Tough Questions Early

28. Tough Questions to Ask • Do you really want to do this or is someone forcing you? • Do you really understand random assignment? • Are you committed to using a lottery to decide who gets the intervention and who does not? • Do you really understand your roles and responsibilities related to: • Research procedures • Data collection • Implementation of the intervention

29. More Tough Things to Ask • Who could object to the study procedures and are you prepared to confront these objections and stay the course? • unions, parents, governing board, participant referral sources • Can you devote the staff time needed to manage the study and the intervention?

30. Even More Tough Questions to Ask • Are you about to (likely to) do something else that will affect our ability to pull off the study? • Change local priorities so not interested? • Introduce some other similar intervention so not a service contrast? • Lay off staff so cannot staff it? • Study champion is about to retire or take a new job?

31. Still More Tough Questions • Are there lurking ethical concerns? • Uncertainty about real scarcity? • Need to recruit more to have a control group? • Local certainty the intervention really works so should not deny access? • Local certainty can identify those most in need who will benefit? • Sense of coercion despite informed consent? • ????

32. Negotiating and Closing the Deal

33. Negotiating a Formal Agreement • A formal, detailed agreement is important • It should specify respective roles and responsibilities about both program and research activities, timelines, costs • The agreement must be cleared directly with all relevant decision makers • The head person must read and sign • This takes time but worth the effort in the long run

34. Negotiating and setting the stage (cont.) • Once signed an agreement is gradually forgotten by many • Therefore, one must have at least one properly placed internal champion to push forward the study • The champion must be tended and kept engaged via ongoing communication • Other relationships also matter; make the rounds periodically