Systematic Review Module 3: Study Eligibility Criteria

1. Systematic Review Module 3: Study Eligibility Criteria Melissa McPheeters, PhD, MPH Associate Director Vanderbilt University Evidence-based Practice Center

2. Learning Objectives • To understand the role of selection criteria in framing a systematic review • To know when and how to set selection criteria • To understand the effect of selection criteria on interpretation of a review

3. CER Process Overview

4. Study Selection Criteria • Function the same in systematic reviews as in primary research • Should reflect the analytic framework and key questions • Are powerful tools for widening or narrowing the scope of a review • Provide information to determine whether reviews can be compared or combined

5. Some Example Criteria • Adult, community-dwelling females • Study of a screening tool for depression • United States only • Hospital-based studies only • N > 200 • Randomized controlled trials

6. Using Broad Criteria • Can be as broad as all related studies • Helpful for exploring “what is known” • May result in too much literature to feasibly review or disparate literature that cannot be compared

7. Using Narrow Criteria • May return too little literature • Can result in inability to answer the intended question • Helpful in culling homogenous literature • Can reduce size of the literature to a manageable scope

8. Bias in this Context • Distortion of the estimate of effect that comes from how studies are selected for inclusion • Affects the applicability or “external validity” of the review itself

9. Examples of Bias in this Context • Included studies may not have been conducted in the patient population whose care the review is intended to affect • e.g., the use of studies of twin pregnancies in a review of preterm labor management for low-risk women • Selection criteria may be set to include more of a certain study type that either overestimates or underestimates effectiveness

10. Selecting Criteria • Review study goals • Assess analytic framework and key questions • Tie criteria to PICOTS • Set criteria before beginning abstract review

11. Basic Questions • What is the relevant population? • What is the intervention of interest? • To what exposure is the intervention being compared? • What outcomes are relevant? • Should time to outcome be limited? • In what setting should the results be applicable?

12. Exercise 1 • What would you do if you were asked to review the literature on transition support for adolescents with autism spectrum disorders (ASD) entering adulthood? • Before seeing the key questions, consider the categories of criteria that we will want to apply.

13. Apply PICOTS • Population—condition, disease severity and stage, comorbidities, patient demographics • Intervention—dosage, frequency, method of administration • Comparators—placebo, usual care, or active control • Outcomes—health outcomes, morbidity, mortality, quality of life (QoL) • Timing—Duration of followup • Setting—Primary, specialty, in-patient, cointerventions

14. PICOTS • Population • Intervention • Comparators • Outcomes • Timing • Setting • What constitutes an adolescent? What constitutes a diagnosis of ASD? • How is transition support defined? • Do we compare to no transition support or directly compare types of support? • What are the goals for adolescents with ASD as they transition to adulthood? Should they be individually focused? • How quickly should the outcomes be apparent? • Is transition support provided in multiple settings, such as schools, clinics, and the community?

15. What Would You Do with… • A paper that was about “individuals over age 10”? • A paper that was about an intervention for individuals with a range of developmental disabilities? • Or, conversely, a paper that was specifically about children with Asperger’s syndrome but not other ASDs?

16. Example of a Narrow Scope • What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?

17. Implications of a Question with Narrow Scope • Efficacy: RCTs only • Low risk: no prior preterm birth • No. of studies: 11

18. Overactive Bladder Study • Management of OAB among women • Considerations • OAB is a fairly difficult condition to define • Treatments include pharmacologic, behavioral, complementary and alternative medicine (CAM) and procedural—and each area includes very different types of studies • Study of OAB is often combined with other types of urological conditions, such as stress incontinence or prostate issues

19. Exercise 2 • Set two criteria and consider the potential implications • Minimum study size • Gender of study participants

20. Study Size • 50 at study start • Implications • Excluded for size only: 79 • Excluded for N < 20: 36 • Excluded for N 20 to 29: 23 • Excluded for N 30 to 39: 8 • Excluded for N 40 to 49: 12

21. Gender • Studies had to include at least 75% women • This decision was based on expert opinion and the size and scope of the literature • 40 studies were excluded with less than 75% women • 27 additional studies would have been excluded had the review been limited to studies of only women

22. Other Considerations • What study designs should be included? • Include foreign studies? Other languages? Studies conducted in specific parts of the world? • Include “grey” or “fugitive” literature?

23. Types of Studies? • Limit to RCTs? • Include observational studies? • If so, which kinds? • What is the value of a case series? • What counts as a case series?

24. Example of Maternal-fetal Surgery Technical Brief • Included case series with N ≥ 2 • Only 3 of 169 studies were RCTs, and 122 were case series • Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups

25. Observational Studies • Types • Cohorts (with comparisons) • Case controls • Case series • Registries/databases

26. Observational Studies • Well-done observational studies can address issues of applicability and the need for longer-term outcomes if they: • Include more representative patient populations • Have relevant comparators • Report more meaningful clinical outcomes over longer time frames • Observational studies may be a better source of information about harms

27. Foreign Literature • Positive findings may be more likely to be published in high-profile journals published in English • Therefore, to include only English-language journals may overestimate the positive effect of an intervention • Empirically, the bias associated with limiting one’s review to English has been shown to be small (Moher et al., 2000; Gregoire et al., 1995)

28. Systematic Review on Cesarean Delivery • Systematic review on outcomes of cesarean delivery on maternal request • Conducted for the National Institutes of Health-Office of Medical Applications of Research State-of-the-Science conference

29. Exercise 3 • Define the appropriate population group and comparator. • What other limitations would you put on included literature?

30. The Challenge • No evidence on outcomes of CDMR vs. other modes of delivery • Urgent need for actionable evidence • Need to recognize and account for confounders 

31. Solution • Expand search to include proxies • Weight rungs of evidence to account for confounding • Highest rung: Trials of breech delivery, but only for maternal outcomes • Next rung: Planned cesarean vs. planned vaginal • Lowest rung: Comparisons of maternal and neonatal outcomes from actual modes of delivery

32. Summary • Selection criteria are essential for setting the scope of the review • They should be tied to the analytic framework, key questions, and PICOTS • When properly applied, selection criteria can reduce bias and support the applicability of the review