1 / 78

Completing a Campbell Systematic Review

Completing a Campbell Systematic Review . Paul Montgomery Joanne Yaffe David Albright. Society for Social Work and Research Washington, DC January 15, 2012. Mission:

trinh
Download Presentation

Completing a Campbell Systematic Review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Completing a Campbell Systematic Review Paul Montgomery Joanne Yaffe David Albright Society for Social Work and Research Washington, DC January 15, 2012

  2. Mission: The Campbell Collaboration (C2) helps people make well-informed decisions by preparing, maintaining and disseminating systematic reviews in education, crime and justice, and social welfare. • Who are we?The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions. Campbell is based on voluntary cooperation among researchers of a variety of backgrounds.

  3. What is a systematic review? • The purpose of a systematic review is to sum up the best available research on a specific question. This is done by synthesizing the results of several studies. • A systematic review uses transparent procedures to find, evaluate and synthesize the results of relevant research. Procedures are explicitly defined in advance, in order to ensure that the exercise is transparent and can be replicated. This practice is also designed to minimize bias. • Studies included in a review are screened for quality, so that the findings of a large number of studies can be combined. Peer review is a key part of the process; qualified independent researchers control the author's methods and results.

  4. Why systematic reviews? • Evidence-based practice demands high quality evidence on which to base practice and policy decisions. • It is impossible for most social work practitioners to read all relevant research on a given topic. • Even if they could, many of the available studies are of low quality. • Studies accessible in social work journals are biased in favor of positive findings. • “Through critical exploration, evaluation, and synthesis the systematic review separates the insignificant, unsound or redundant deadwood from the salient and critical studies that are worthy of reflection.”

  5. Additional Rationale for Systematic Reviews: Decision Making • Practitioners need SRs to keep up with the literature in their field • Policy-makers need them to make sensible decisions • Researchers need them to identify gaps and establish protocols for future research.

  6. Additional Rationale for Systematic Reviews: Efficiency • A review is usually less costly and time-consuming than embarking on a new study (although an SR may conclude that new studies are called for). • Reviews can also suggest or even establish that further research in a given area is not called for or even unethical. • Continuously updated reviews can shorten the time between research discoveries and ‘real world’ implementation of effective diagnostic or treatment strategies.

  7. Additional Rationale for Systematic Reviews: Generalizability • Diversity of multiple reviewed studies provides an interpretive context not available in any one study…. studies addressing similar questions often use different eligibility criteria for participants, different definitions of the ‘problem’, different variations of an intervention, different study designs, etc.

  8. Additional Rationale for Systematic Reviews: Examining Consistency of Relationships • Are effects in the same direction, and of the same general magnitudes, given the variation in study protocols? • Does consistency exist between studies of the same intervention or different dosages/intensities/classes of the same intervention?

  9. Additional Rationale for Systematic Reviews: Examining Inconsistency of Relationships • If an intervention strategy is effective in one setting and not in another, or among some people and not among others, this is important! • Furthermore, whether findings from a single study stand alone for any reason (uniqueness of study population, study quality or outcome measure) should be explored…

  10. Additional Rationale for Systematic Reviews: Power • Quantitative SRs have the advantage of increased power. The advantage of increasing power is particularly relevant to conditions of relatively low event rates or when small effects are being assessed.

  11. Additional Rationale for Systematic Reviews: Precision • A measure of the likelihood of random errors in the results of a study, meta-analysis or measurement. • Confidence intervals around the estimate of effect from each study are a measure of precision. • The weight given to the results of each study in a meta-analysis (typically the inverse of the variance of the estimate of effect) is a measure of precision (i.e. the degree to which a study influences the overall estimate of effect in a meta-analysis is determined by the precision of its estimate of effect).

  12. Additional Rationale for Systematic Reviews: Accuracy • …or at least an improved reflection of reality…. Traditional (non-systematic) reviews can be criticized as biased and haphazard… subject to idiosyncrasies of the reviewer… • SRs and meta-analyses using scientific principles which aim to reduce random and systematic errors are superior.

  13. A Campbell systematic review must have: • Clear inclusion/ exclusion criteria • An explicit search strategy • Systematic coding and analysis of included studies • Meta-analysis (where possible)

  14. How do Campbell systematic reviews differ from other systematic reviews? • Campbell reviews must include a systematic search for unpublished reports (to avoid publication bias). • Campbell reviews are usually international in scope. • A protocol (project plan) for the review is developed in advance and undergoes peer review. • Study inclusion and coding decisions are accomplished by at least two reviewers who work independently and compare results. • Campbell reviews undergo peer review and editorial review.

  15. The Campbell Library of Systematic Reviews • Campbell reviews are published in the Campbell Library of Systematic Reviews, and are freely accessible online. Campbell systematic reviews can be updated as further relevant information emerges.

  16. Completing a Campbell Review

  17. Major Steps for a Campbell Systematic Review Title Registration Protocol Development Protocol Publication Review Development Review Publication Periodic Updates

  18. Campbell Systematic Review • Select a topic and formulate a focused question • Register the title • Prepare the protocol (Campbell Editorial Board & external peer review) • Electronic publication of protocol in The Campbell Library • Perform a comprehensive literature search • Prepare the Review (Editorial Board & external peer review) • Electronic publication of completed Review in the Campbell Library • Respond to comments/criticism • Keep the review up-to-date

  19. Title Registration • Reserve the topic • Do your homework! • Components • Title: Intervention for Problem in Population • E.g.: Does Eye Movement Desensitization and Reprocessing (EMDR) for Posttraumatic Stress Disorder (PTSD) in Combat Veterans • Background and objective of the review • Define the population • Define the intervention(s) • Outcome(s) • Methodology • Review team • Roles and responsibilities • Potential conflicts of interest • Support • Funding • Preliminary timeframe

  20. Protocol • Roadmap for Review • Components • Background • Objective of the review • Methods • Acknowledgements • References • Appendices • Contribution of authors • Declaration of Interest • Sources of Support

  21. Background • Description of the condition • Description of the intervention • How the intervention might work • Why it is important to do this review

  22. Objective of the Review • Examples: • The primary objective is to complete a SR of experimental and quasi-experimental studies of EMDR for combat veterans with PTSD • The secondary objective is to synthesize the results of these studies to asses the effect of EMDR on reducing PTSD in combat veterans

  23. Methods • Criteria for considering studies for the review • Types of studies (e.g., experimental, quasi-experimental) • Types of participants (e.g., military combat veterans) • Types of interventions (i.e., EMD/EMDR) • Types of outcomes (e.g., level of PTSD symptoms via structured diagnostic interview or self-report questionnaire)

  24. Methods Search methods for identification of studies Electronic searches (e.g., EMBASE, MEDLINE, PILOTS) Search terms Searching other resources (e.g., correspondence, grey literature, hand-searching)

  25. Methods • Data Collection and analysis • Selection of studies (stage 1 and 2) • Data extraction and management (stage 3) • Assessment of risk of bias within included studies (e.g., sequence generation, incomplete outcome data) • Measures of treatment effect (how we will handle dichotomous and continuous data) • Unit of analysis issues • Dealing with missing data and incomplete data • Assessment of heterogeneity (e.g., Q-statistic, I² statistic) • Assessment of publication bias

  26. Methods • Data Synthesis • Subgroup analysis, moderator analysis and investigation of heterogeneity • Sensitivity analysis

  27. Appendices • Examples: • Screening: Stage 1 • Screening: Stage 2 • Extraction

  28. Dr Paul Montgomery University of Oxford Dr Jane Dennis, School for Policy Studies University of Bristol

  29. Validity • Did the review address a clearly focussed question? • Were the right sort of studies selected? • Was the search strategy explicit and comprehensive? • Did the reviewers assess the quality of the identified studies?

  30. 1. Question formulation: Are non-drug CBT interventions effective in resolving the sleep problems of people aged over 60?

  31. Validity • Did the review address a clearly focussed question? • Were the right sorts of studies selected? • Was the search strategy explicit and comprehensive? • Did the reviewers assess the quality of the identified studies?

  32. 2. Inclusion Criteria: • Types of studies: Randomised controlled trials • Types of participants • Over 60s • diagnosed with sleep problems via standardised measure • screened to exclude dementia, depression and sleep apnoea or secondary insomnia (sleep disturbance caused by a psychiatric or medical disorder)

  33. Sleep-CBT reviewOutcome measures: • Sleep onset latency (SOL) • Wake after sleep onset (WASO) • Total wake time (TWT) • Sleep duration (total) • Early morning wakening • Sleep efficiency • Self-report of sleep satisfaction • Scales related to sleep, eg the Pittsburgh Sleep Quality Index (PSQI); Stanford Sleepiness Scale • Quality of life Outcomes were divided, where possible, into immediate post-treatment, medium term (3-12 months) and long-term (more than 12 months).

  34. Validity • Did the review address a clearly focussed question? • Were the right sort of studies selected? • Was the search strategy explicit and comprehensive? • Did the reviewers assess the quality of the identified studies?

  35. 3. Search strategy: Potential hazards: • Publication Bias • Tower of Babel Bias • Uncritical use of electronic databases

  36. Publication Bias “the tendency of investigators, reviewers and editors to differentially submit or accept manuscripts for publication on the direction or strength of the study findings.” Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-2753

  37. Publication bias • Prospective • Registries of all trials in progress • Retrospective • Trial amnesty • Negotiation • Funnel plot

  38. Unpublished data • Controversial • Unpublished data may not be a full or representative sample (Cook 1993) • Publication is no guarantee of scientific quality (Oxman 1991) Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-2753 Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al. Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.

  39. Tower of Babel Bias • Studies that find a treatment effect are more likely to be published in English-language journals. • Opposing studies may be published in non-English-language journals. Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol. 1995;48:159-163

  40. Maintenance treatment review Search Strategy: • Electronic databases • Reference checking • Hand-searching • Personal communications

  41. Search terms involving • sleep problems • the older adult • psychological, physical, phototherapeutic, treatments

  42. Validity • Did the review address a clearly focussed question? • Were the right sort of studies selected? • Was the search strategy explicit and comprehensive? • Did the reviewers assess the quality of the identified studies?

  43. 4. Assessment of study quality: • Should be routine • Dependent on sort of studies included • Use of rating scales with fixed cut-offs potentially misleading

  44. Broad search results:2722 hits, of which 2486 were eliminated at once Of the 236 remaining, 79 were potential trials

  45. Importance: • Were the results similar from study to study? • What is the overall result of the review? • How precise are the results?

  46. Importance: • Were the results similar from study to study? • What is the overall result of the review? • How precise are the results?

  47. 1. HeterogeneityCombining apples and oranges? Sources of Heterogeneity: • Differences in study participants • Different comparisons • Different duration of follow-up • Different outcome measures • Differences in methodological quality

  48. Testing for heterogeneity: • Look at plots of results • Formal tests of homogeneity .

  49. Importance: • Were the results similar from study to study? • What is the overall result of the review? • How precise are the results?

More Related