1 / 37

Assessing study quality for a systematic review

Assessing study quality for a systematic review. Trudy Bekkering , PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research Katholieke Universiteit Leuven, Belgium. Why is it so important?.

ranae
Download Presentation

Assessing study quality for a systematic review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing study quality for a systematic review Trudy Bekkering, PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research KatholiekeUniversiteit Leuven, Belgium

  2. Why is it so important? • Meta-analysis aims to increase precision • Meta-analysis of studies with bias in results gives very precise but wrong results • Garbage in, garbage out

  3. Bias versus imprecision Bias: • A systematic error in the results or the inferences • Methodologicalflaw • Overestimation or underestimation

  4. Bias versus imprecision BIAS Idealstudy

  5. Bias versus imprecision Imprecision: • A random error in the results • Sample variation • Direction of error is random

  6. Bias versus imprecision IMPRECISION Ideal study

  7. Bias versus imprecision BIAS + IMPRECISION

  8. Bias versus imprecision BIAS ? IMPRECISION?

  9. Risk of bias versus bias • Clearempiricalevidencethatparticularflaws in study design can lead to bias. • Usuallyimpossibletoknowtowhatextentbiaseshave affected the results. • Keyconsideration = should the resultsbebelieved

  10. Tools for assessing study quality Scales: discouraged Checklist: between 3 and 57 items Cochrane tool: “domain based” Depends on study design

  11. Tool for RCTs: Cochrane tool Risk of bias on 6 domaines: • Sequencegeneration • Allocationconcealment • Blinding • Incomplete outcome data • Selectivereporting • Other sources of bias

  12. Risk of which biases?

  13. How do you assess?

  14. Sequencegeneration

  15. Allocationconcealment

  16. Blinding of intervention • Participants (patients, clients) • Care providers (doctors, nurses, teachers …) • Outcome assessors

  17. Blinding of intervention

  18. Incomplete outcome data “Attrition” (drop-out): no data • Withdrawal • Do notattend follow-up appointment • Failure to complete questionnaire / diaries • Cannotbelocated (lost to follow-up) • Decisionby investigator tocease follow-up • Data or records are lost

  19. Incomplete outcome data “Exclusion”: data available, but excluded from analysis • Participants found to be ineligible after enrolment • An “as treated” (or per-protocol) analysis:participants are only included if they received the intended intervention in accordance with the protocol

  20. Assessing risk of bias

  21. Selectiveoutcomereporting • = selection of a subset of the variables recorded for inclusion in publication, • on the basis of the results • For example: • Omission of non-significant outcomes • Choice of data for an outcome (e.g. osteoporosis) • Choice of analysis (e.g. blood pressure) • Reporting of subsets of data (e.g. sepsis) • Under-reporting of data (e.g. only “not significant”)

  22. Presentation in your review “Risk of bias graph”

  23. “Risk of bias summary”

  24. Assessing risk of bias in NRS • Selection bias (how was groupallocation?) • Performance bias (blinding, fidelity of interventions) • Attrition bias (completeness of sample & follow-up) • Reporting bias (selectiveoutcomereporting) • Confoundingandadjustment

  25. Confounding ? ? Imbalance in prognostic factors Comparison intervention - control Intervention versus control Difference in outcome

  26. Confounding Confounding factor Association between 2 factors Presence of risk factor Occurrence of outcome

  27. Confounding & adjustment • At the stage of protocol: list potential confounding factors • Identify the factors the authors have considered and omitted • Assess balance between groups at baseline • What did authors do to control for confounders (matching, restricting to subgroups, stratification, regression modelling)

  28. Tool for NRS Downs and Black instrument (J Epidemiol Community Health 1998;52:377-84) 27 items: • Reporting (10) • Externalvalidity (applicability) (3) • Internalvalidity - bias (7) • Internalvalidity – confounding (6) • Power (1)

  29. Downs and Black instrumenthttp://www.nccmt.ca/ registry/view/eng/9.html

  30. Tool for NRS Newcastle-Ottawa Scale (NOS) (Wells 2008) 8 items covering 3 perspectives: • Selection of studygroups • Comparibility of groups • Ascertainement of exposure (case-control) or outcome (cohort)

  31. http://www.ohri.ca/programs/ clinical_epidemiology/oxford.asp

  32. Diagnostic studies QUADAS tool Whiting BMC Medical Research Methodology 2003;3:25 Cochrane version: 11 items (out of 14 original) Diagnostic Test Accuracy Working Group: handbook http://srdta.cochrane.org/ handbook-dta-reviews

  33. Other risk of bias assessment tools SIGN (Scottish Intercollegiate Guidelines Network) http://www.sign.ac.uk/methodology/ checklists.html

  34. In summary • Risk of bias assessment is essential for systematic reviews • For RCT: use the Cochranetool • For NRS: • Higher risk of bias (selection bias & reporting bias) • Use the appropriate tool toassess risk of bias • Considerhowpotentialconfounders are addressed

More Related