1 / 39

Sequential Experimentation: A Different Approach to Prevention Research

Sequential Experimentation: A Different Approach to Prevention Research. Annual Meeting of the Society for Prevention Research Washington, DC June, 2003. Linda M. Collins The Methodology Center and Department of Human Development & Family Studies Penn State

Download Presentation

Sequential Experimentation: A Different Approach to Prevention Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Sequential Experimentation: A Different Approach to Prevention Research Annual Meeting of the Society for Prevention Research Washington, DC June, 2003

  2. Linda M. Collins The Methodology Center and Department of Human Development & Family Studies Penn State Susan A. Murphy Department of Statistics and Institute for Social Research University of Michigan Vijay Nair Departments of Statistics and Industrial & Operations Engineering University of Michigan

  3. Overview • Why we are proposing a change in how we conduct prevention research • Introduction to sequential experimentation • Advantages we see to sequential experimentation • Considerations and open questions

  4. Two broad scientific issues facing prevention science

  5. How can we improve our understanding of the mechanisms underlying prevention? • In early days of prevention, the field struggled to establish THAT prevention worked • Now, the time is right to begin understanding HOW, WHY and FOR WHOM prevention works • This will allow us to fine-tune, even optimize, interventions

  6. How can we hasten the building of a coherent body of knowledge about prevention? • The field has moved forward in important ways – but slowly • Solid, replicable results are mixed with unreplicable and contradictory findings • We need to speed up the accumulation of consistent, replicable findings

  7. We argue that a new approach is needed in order to address these issues. But what is the difficulty with the current approach?

  8. Two prevention program evaluation research agendas • Answering the bottom-line question of whether a program is effective • Informing theory and design of prevention programs

  9. Two prevention program evaluation research agendas, one approach • We try to address both agendas with large, expensive prevention trials involving • lots of subjects • power for detecting treatment effects • lots of variables • post hoc analyses with the hope of informing theory and figuring out how the program worked (or why it didn’t work)

  10. Two prevention research program evaluation research agendas, one approach • This approach tries to serve both agendas simultaneously, but is it the BEST way to serve either one?

  11. How can we best inform theory? • Post-hoc or secondary analyses can be very misleading • e.g. post-hoc “dosage received” analyses • Best way to inform theory: series of randomized experiments

  12. How can we best answer questions about program effectiveness? • Even sound post-hoc analyses cannot help improve the prevention trial that generated the data • Best way to maximize program effectiveness: obtain data to inform the prevention trial BEFORE it is conducted

  13. We propose a different approach: sequential experimentation

  14. Overview • Why we are proposing a change in how we conduct prevention research • Introduction to sequential experimentation • Advantages we see to sequential experimentation • Considerations and open questions

  15. A different approach: Sequential experimentation • Emerged from agriculture and engineering • Philosophy: • scientific research is an iterative process of deduction and induction • constant interplay among theory, experimentation, data analysis

  16. A different approach: Sequential experimentation • In engineering • theory suggests many possible treatment combinations that would require a prototype • not practical to build ALL prototypes – each one is resource-intensive

  17. A different approach: Sequential experimentation • In prevention • Theory suggests many candidate components and combinations of components that could make up a preventive intervention • Not practical to bring EVERY possible combination of components to a prevention trial – each trial is resource-intensive

  18. The objective of sequential experimentation • Sort through the space of all possible treatment combinations and dosages in an efficient way, to select programs and dosages most likely to be successful

  19. Sequential experimentation: A sequence of three phases • Screening of candidate program components • Refining of program components • Confirming the effectiveness of a refined program

  20. Screening phase • Purpose: to sort through candidate components to decide which are effective and should be investigated further • Starting point: candidate intervention components suggested by theory • e.g. normative education, resistance training, parental component • Examine each via a series of randomized controlled experiments

  21. Screening phase • Components identified as effective in the screening phase are tentatively selected for inclusion in the intervention. • Next, go to refining phase.

  22. Refining phase • Purpose: identify interactions, investigate blocking variables and putative moderators, refine dosage • e.g.: • examine whether normative education and parental component interact • examine effects of gender • determine optimal number of resistance training lessons • Starting point: components identified as effective in the screening phase

  23. Refining phase • Series of randomized experiments on components that passed the screening phase • Depending on results, may go back to screening, or on to confirming phase

  24. Confirming phase • Purpose: construct and test preventive intervention • Starting point: active program components, appropriate dosages, and important moderators identified during screening and refining phases

  25. Confirming phase • Develop a refined intervention, consisting of only active, effective components, delivered in appropriate amounts • A full randomized prevention trial testing the refined intervention

  26. A few key points about sequential experimentation • Theory plays a critically important role, particularly in the screening and refining phases • Selection of candidate components • Informing the designs used

  27. A few key points about sequential experimentation • Extensive use made of designs such as balanced fractional factorials and response surfaces • These designs use theory to inform choices about where to concentrate resources • Much more efficient than full factorial designs • Number of cells and sample size requirements can be greatly reduced, WHILE MAINTAINING STATISTICAL POWER

  28. A few key points about sequential experimentation • The screening and refining phases are NOT pilot studies • They are randomized, controlled studies

  29. A few key points about sequential experimentation • Type II errors (overlooking active prevention component) may be worse here than Type I errors (mistakenly selecting inactive component) • Type I error rate greater than usual p<.05 may be used

  30. Overview • Why we are proposing a change in how we conduct prevention research • Introduction to sequential experimentation • Advantages we see to sequential experimentation • Considerations and open questions

  31. Advantages we see to the sequential experimentation approach • Screening, refining phases address “how, why, and for whom” questions

  32. Advantages we see to the sequential experimentation approach • Results of screening and refining phases will be replicable and will lead to a coherent body of knowledge

  33. Advantages we see to the sequential experimentation approach • Programs that go to prevention trials have been refined; the idea is to produce larger effect sizes • increased statistical power • increased public health benefits

  34. Advantages we see to the sequential experimentation approach • It may allow for more creative risk-taking • it is possible to try a brand-new approach in the screening phase without the investment of an entire prevention trial

  35. Advantages we see to the sequential experimentation approach • Less likely to run into unanticipated problems • Thus less likely to be presented with the familiar dilemma: • Midway through the prevention trial we realize the program needs revision, but we don’t want to change the program and thereby make the evaluation difficult

  36. Overview • Why we are proposing a change in how we conduct prevention research • Introduction to sequential experimentation • Advantages we see to sequential experimentation • Considerations and open questions

  37. Considerations and open questions • In most cases, the screening and refining phases can occur in a five-year funding cycle • The confirming phase can occur in a second five-year funding cycle • The support of NIH program staff will be needed

  38. Considerations and open questions • Hierarchical data • we are working on this • Steady stream of research subjects needed • series of experiments may take place over several years • possibility of cohort effects

  39. Considerations and open questions • You can think of many others!

More Related