C2 Training: May 9 – 10, 2011

1. C2 Training: May 9 – 10, 2011 Introduction to Systematic Reviews

2. Systematic Review Methods • Systematic reviews are a form of research • secondary observations • in which studies are the unit of analysis • Follow basic steps in the research process • Aim to minimize bias and error • But SRs are not immune to bias and error (not a panacea)

3. Stages of a research synthesis (Cooper, 1982) The seminal article outlining five stages of a research review is by Harris Cooper: Cooper, H.M. (1982). Scientific guidelines for conducting integrative research reviews. Review of Educational Research, 52, 291-302. Cooper, H. M. (2009). Research synthesis and meta-analysis: A step-by-step approach. Thousand Oaks, CA: Sage. 3

4. Stages of a research synthesis (Cooper, 1982) Problem formulation Clarifying your questions and writing a protocol Set explicit inclusion/exclusion criteria Data collection Literature search Information-gathering from studies Data evaluation Criteria for including and excluding studies Assessing study quality 4

5. Stages of a research synthesis (Cooper, 1982) Data analysis and interpretation Integrating the effects from collected studies Interpreting analysis results Report preparation Narrative, statistics, graphs, tables 5

6. What’s required? • A team with • Substantive expertise • Methodological expertise • Statistical expertise • Information retrieval expertise • Time and money • SRs are labor intensive • $50-$150K depending on scope, complexity, and number of studies

7. Protocols for SRs • A detailed protocol (plan) for the SR should be developed and made available to readers (Higgins & Green, 2008; Moher et al., 2009) • Protocols increase transparency, limit ad hoc decisions • The review process is iterative and plans may change during the process • The final report should document and explain changes made (deviations from the protocol)

8. Stages of a research synthesis Most of the work involved in conducting a review is not spent in statistical analysis. The scientific contribution of the final product is dependent on all stages of the review and not just the statistical analysis stage. 8

9. An example of the stages of a review We will illustrate each of the stages by considering the C2 systematic review on Multisystemic Therapy (MST): Littell, J.H., Popa, M., & Forsythe, B. (2005). Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10 – 17. (Campbell version) 9

10. Stage 1 Problem Formulation 10

11. Problem formulations • Conceptual issues arise in attempts to combine results of studies that vary (to some degree) in their methods, treatments, samples, outcome measures • Apples, oranges, and other fruits • Parallel problems in studies of individuals (no 2 people are identical and there is no “average” person) • Given variation in primary research • What should be included in a synthesis? • How should we synthesize results?

12. SRs vary in scope • Specific, narrow questions • Useful for testing effects of specific treatments • Broad, global questions • Useful for generating new knowledge • Identify common elements of effective programs (Lipsey, 2008) • build better intervention theories to guide program development and evaluation design (Lipsey, 1997)

13. Scope of Systematic Reviews • Not limited to questions about effects • Can address trends, epidemiology, accuracy of diagnostic and prognostic tests • Not limited to randomized controlled trials or quantitative data • Qualitative synthesis (e.g., meta-ethnography, narrative analysis of qualitative research reports) • Mixed/multiple methods synthesis (e.g., Thomas, Harden, et al. on progams to combat childhood obesity)

14. What kinds of research questions can be asked in a systematic review? • Questions about intervention effects: • What are the effects of x intervention on y outcomes for z populations/problem? • Variations on this theme (e.g., differences in effects of interventions x1 vs x2)

15. Kinds of research questions (continued) • Questions about associations • How does x1 relate to x2 for population z? (direction and strength of correlation) • Variations on this theme (e.g., differences in relation of x1 and x2 between populations z1 and z2 ) • Diagnostic/Prognostic questions • Which test (A vs. B) is a better predictor of y? • Which test (A vs. B) is a better predictor of y for z1 vs. z2 populations?

16. Problem formulation To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families. 16

17. Problem formulation What research evidence will be relevant to the problem or hypothesis of interest in the synthesis? Studies where youth (age 10-17) with social, emotional, and/or behavioral problems were randomised to licensed MST programs or other conditions (usual services or alternative treatments). 17

18. Problem formulation Define the (a) variables and (b) relationships of interest so that relevant and irrelevant studies can be distinguished. Multisystemic Therapy (MST) is a multi-faceted, short-term, home- and community-based intervention for families of youth with severe psychosocial and behavioral problems. Based on social ecological and family systems theories, and on research on the causes and correlates ofserious antisocial behavior in youth (Henggeler 1998, Henggeler 2002a), MST is designed to address complex psychosocial problems and provide alternatives to out-of-home placement of children and youth. 18

19. Problem formulation (continued) • Measures of behavioral, psychosocial, and family outcomes were examined. • Behavioral outcomes included antisocial behavior (as measured by arrest or conviction of a criminal offense), drug use (self-reports and drug tests), and school attendance. • Psychosocial outcomes included measures of psychiatric symptoms (on standardized scales),school performance (teacher reports), peer relations (self-reports and parent or teacher reports),and self esteem. • To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

20. Step two: Establish study inclusion and exclusion criteria • The PICOS framework: • Population/Participants (problems/conditions) • Interventions (if applicable) • Comparison group (e.g., absolute vs. relative effects, counterfactual conditions) • Outcomes (primary and secondary outcomes, acceptable outcome measures) • Study Design (should be fit for purpose) Geographic area, time, language, other criteria

21. Participants: Who is included in the sample of subjects? • Why 10 - 17 only? • Exclude special populations? • Exclude studies with only males or females? • Include families of these children? • What is the rationale for the choice?

22. Interventions: How do we define MST?Comparisons: What is the comparison? • What kinds of programs are considered MST? • What types of treatments are the “right” comparisons to MST?

23. What are the Outcomes? • Research question states: • Placement in out-of-home living arrangements • Crime and delinquency • Behavioral outcomes • Psychosocial outcomes

24. Study Designs: What types of study designs are relevant for our review? • If the research question is the assessment of the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and social outcomes for youth and families, what designs are most appropriate?

25. What studies are relevant? • Will we attempt to generalize to all studies done? • Will we look only at “high quality” studies? • Will we look at both published and unpublished? • Will we set a time period for the studies? • Will we look at only randomized controlled trials or will we include quasi-experimental studies? Why?

26. Study eligibility criteria • Usually focus on inclusion criteria, with few exclusion criteria • Studies should not be excluded based on: • Sample size (or statistical power considerations) • Publication status

27. What kinds of studies should be included? Garbage in, garbage out • Synthesis of invalid studies produces invalid conclusions • What constitutes credible evidence? • Depends on the question • Need to set clear standards • Based on methodological principles • Relevant for topic and context

28. Small group exercise: Problem formulation • Form a working group of 2 – 5 people and develop a preliminary research question for a comprehensive review • Elect a recorder for the group who will report back to the whole • Each group will share the research question, and any challenges that they will face in studying this question • See the first exercise on problem formulation