Quasi-Experimental Designs 101: What Works?

1. Quasi-Experimental Designs 101:What Works? The Need To Know Team January 31 – February 1, 2005 Patricia J. Martens PhD

2. Tic Tac Toe anyone? Outline • Reviewing X’s and O’s • Quasi-experimental time series designs with comparison groups • The Population Health Research Data Repository: what data do we have? • Brainstorming ideas

3. Key features of study designs • Artificial manipulation? (experimental or observational) Experimental: • Are the groups randomly assigned to receive or not receive the intervention? (randomized controlled trial) • Are the groups selected to be as similar as possible, not randomly? (quasi-experimental comparison groups)

4. Research Design Schema

5. Key Features of Study Designs • Observational: • Information collected concurrently or over a time period? (cross-sectional or longitudinal) • If over a time period, i.e. longitudinal, do you go from exposure to disease (cohort) or from disease back in time to examine exposures (case-control)? • Do you start now and go forward (prospective), or do you have a “cohort” somewhere in the past and you follow them forward (historical prospective)?

6. Research Design Schema

7. Study design: observational • Cross-sectional studies • studying all factors at once - both the hypothesized explanatory and outcome variables • Prospective studies • going forward in time, following a cohort and observing the effect of exposure to a future outcome • Case-control studies • going backwards in time from the cases/controls to look at differential exposures

8. Research Design Schema

9. Study design: “What Works” proposal • Randomized Controlled (Clinical) Trial • designing a specific intervention and randomly assigning people to receive it or not to receive it • Quasi-experimental • using a comparison group which is not randomly assigned • Each RHA is a comparison group • A quasi-experimental time series with many comparison groups (all other RHAs in the province) • Diagrammed and described by Campbell & Stanley (1963)

10. Let’s play X’s and O’s X is an intervention O is an outcome measure X O

11. Let’s play X’s and O’s O X O

12. Let’s play X’s and O’s O X O O O

13. Let’s play X’s and O’s R means randomly assigned R O X O R O O (pretest-posttest control group design)

14. Let’s play X’s and O’s _ _ _ _ means not randomly assigned (quasi-experimental comparison) O X O - - - - - - - - O O

15. Let’s play X’s and O’s O X O - - - - - - - - O O quasi-experimental pretest- posttest design (non-randomized control group) (non-equivalent pretest-posttest comparison group design)

16. Hospital BFHI Compliance Scores 40 site Arborg intervention Pine Falls 30 Split-unit anova: p=0.0009 BFHI Compliance 20 control Ten Steps and WHO Code each assigned 4 points, for total compliance of 44 10 0 1 2 Martens 2001 Time (8 month interval) Examples of a quasi-experimental pretest- posttest comparison group study to determine effectiveness of hospital policy/education program

17. Let’s play X’s and O’s O O X O O Time series (quasi experiments)

18. Breastfeeding Initiation 1992-97 1.0 0.9 CHN at conference, uses new techniques to 0.8 address prenatal feeding 1994 Breastfeeding study: intent ? 0.7 pregnant women interviewed * ? PC Training 0.6 ? begun proportion initiating breastfeeding 0.5 0.4 ? ? ? 0.3 CHN hired Video and breastfeedng booklet completed, used in 0.2 individual prenatal * p<0.05, one-tailed, adjusted for birth weight and parity instruction by CHN 0.1 0.0 1992 1993 1994 1995 1996 1997 year Martens 2002 Example of a quasi-experimental time series to determine effectiveness of a community-based breastfeeding strategy

19. Let’s play X’s and O’s Time series (quasi experiment with comparison group) O O X O O - - - - - - - - - - - - - - - O O O O

20. Example of a quasi-experimental time series with comparison groups to determine effectiveness of a regional teen pregnancy reduction program From CIHR proposal submission September 2004

21. Additions of small amounts of phosphorus to one section of ELA Lake 226 caused surface blooms of blue-green algae, and vividly demonstrated the importance of phosphate as a cause of excessive algal growth or eutrophication. This experiment spurred legislation controlling the input of phosphorus to many water bodies. A demonstration of the work of Dr. David Schindler and the Experimental Lakes project in NW Ontario http://www.umanitoba.ca/institutes/fisheries/eutro.html

22. Study design: Low internal validity • Anecdote/case study • Pre-experimental • just doing a pretest and posttest on one group and seeing its effect • Cross-sectional • a snapshot in time: can’t tell which comes first, but only that they are “associated”

23. Study design: medium internal validity • Time series; Time series with qualitative layer • looking over time to see change, with information about when interventions occurred in the time frame • Case-control • going backwards in time from the cases/controls to look at different exposures to possible risk factors • Observational (prospective) • going forward in time, observing the effect of exposure on a cohort to a future outcome

24. Study design: high internal validity • Randomized Controlled (clinical) Trials, RCT • designing a specific intervention and randomly assigning people to receive it or not to receive it • following people to observe the outcome of interest • Quasi-experimental comparison group studies • using a comparison group which is not randomly assigned, but very similar at onset

25. High Randomized Controlled Trials RCT Quasi-experimental comparison group studies Time series with comparison Observational (prospective) Case-control Time series with qualitative layer Internal validity Cross-sectional Pre-experimental Anecdote/case study Low

26. “There is nothing so useless as doing efficiently that which should not be done in the first place.” Peter Drucker

27. Family Services Education Immunization Hospital Medical Home Care Nursing Home Pharmaceuticals Provider Cost Vital Statistics MCHP’s … “paperclips”“Population Health Research Data Repository” Population-Based Health Registry Census Data EA/DA level National surveys

28. Brainstorming: “What Works” proposal • Pick (a) a policy; and (b) a program • Think of something that your region has done in the past, somewhere between 1997 and the present (hopefully, with a few years of data AFTER the onset of this) • What OUTCOME measures would you think this would impact? • Think of what you would expect to see if this intervention was “working” • Are there specific target groups to which this intervention applies? (e.g. teens, people living in a certain district of your region?) • What measures of this intervention would be available through the Repository data? • Brainstorm and report! (see sheet for recording)