How to Critically Appraise Literature

1. How to Critically Appraise Literature James M. Bolton MD Associate Professor, University of Manitoba Departments of Psychiatry, Psychology, Community Health Sciences Adjunct Scientist, Manitoba Centre for Health Policy Medical Director, Crisis Response Centre, WRHA Summer Rounds, University of Manitoba June 28, 2016

2. Disclosures • Relationships with commercial interests: • Grants/Research Support: • Canadian Institutes of Health Research • New Investigator Award (113589) • Brain and Behavior Research Foundation (NARSAD) • Young Investigator Award • Speakers Bureau/Honoraria: Nil • Consulting Fees: Nil • Other: Nil

3. Objectives Learn key questions to ask when reviewing a scientific article Understand the importance of bias

4. Ten Key Questions in Critical Appraisal Is the study relevant? Does the study add anything new? What type of research question is being asked? Is the study design appropriate for the research question? Did the study methods address sources of bias? Was the study performed according to original protocol? Did it test a stated hypothesis? Were statistical analyses performed correctly? Do the data justify the conclusions? Are there any conflicts of interest? • Young JM & Solomon MJ. Nat ClinPractGastroenterolHepatol. 2009

5. Question #1So what? • Is the study important? • The best done study is irrelevant if it does not address an important question • What to look for • The introduction • Authors need to convince a reviewer that the topic/area is important to health, the public, etc.

6. Question #2Does it add anything new? • What to look for • The introduction • Literature review – how thorough is the review, how well is it summarized? • The reviewer should have a clear idea that this is not only something new but also not simply an incremental addition to previous knowledge • You may need to do a quick literature search • Different journals have different thresholds

7. Is the Study Design Appropriate? • Relates to the research question • Efficacy of an intervention: RCT • Frequency of events/risk factors: observational studies

8. Hierarchy of Evidence Systematic reviews and meta-analyses of RCTs RCTs Cohort studies Case-control studies Cross sectional surveys Case reports

9. Do the Methods Address Bias? • Bias • When the results of a study deviate from the truth • When bias is due to the methodology of a study (vs. random chance) – this is called systematic bias

10. Pannucci CJ & Wilkins EG. PlastReconstuct Surg. 2010

11. Addressing Bias: Example 1 • Study: Examining rates of mental disorders in parents bereaved by MVC death of their child • Method: Parents surveyed in a bereavement support group

12. Addressing Bias: Example 1 • Examining rates of mental disorders in parents bereaved by MVC death of their child • Method: Parents surveyed in a bereavement support group • Potential Bias: Selection Bias • Attendees in bereavement groups have been shown to be healthier than non-attendees • Solution: conduct a population-based study of bereaved parents using administrative data

13. Addressing Bias: Example 2 Study: Is cancer associated with suicide? Design: Using CancerCare registries, examining all people in Manitoba who were newly diagnosed with cancer, and examining suicide rates within 1 year Statistical analysis: unadjusted logistic regression models that examined the association with cancer diagnosis and suicide

14. Addressing Bias: Example 2 • Potential Bias: Confounding • Mental disorder are associated with both the predictor variable (cancer) and outcome variable (suicide) • Solution: multivariate regression that adjusts for mental disorders

15. Do the Data Justify the Conclusions? • Meta-analysis of antidepressant trials based on clinical study reports • Main outcomes: mortality and suicidality • Results: • Mortality: Odds ratio 1.28; 95% CI 0.40-4.06 • Suicidality: Odds ratio 1.21; 95% CI 0.84-1.74

16. Do the Data Justify the Conclusions? • Meta-analysis of antidepressant trials based on clinical study reports • Main outcomes: mortality and suicidality • Results: • Mortality: Odds ratio 1.28; 95% CI 0.40-4.06 • Suicidality: Odds ratio 1.21; 95% CI 0.84-1.74 • Conclusions: • “We suggest minimal use of antidepressants in children, adolescents, and young adults, as the serious harms seem to be greater.” • Sharma T et al. BMJ. 2016

17. CASPCritical Appraisal Skills Program casp-uk.net Has downloadable checklists that you can use for any type of study

18. Ten Key Questions in Critical Appraisal Is the study relevant? Does the study add anything new? What type of research question is being asked? Is the study design appropriate for the research question? Did the study methods address sources of bias? Was the study performed according to original protocol? Did it test a stated hypothesis? Were statistical analyses performed correctly? Do the data justify the conclusions? Are there any conflicts of interest? • Young JM & Solomon MJ. Nat ClinPractGastroenterolHepatol. 2009

19. Suggestions • Go toward the fear – don’t avoid! • Participate in journal clubs • Take opportunities to review for journals • Learn what top researchers are working on • See comments by other reviewers • If an option, participate in grant review committees • Remember: You don’t need to be a stats wizard! • The core of critical appraisal is based on the importance of the study and the approach to the question • Clinical knowledge goes a long way