Critical Appraisal & Literature review

1. Critical Appraisal & Literature review James M. Bolton MD Professor of Psychiatry, Psychology, Community Health Sciences, U of M Adjunct Scientist, Manitoba Centre for Health Policy Medical Director, Crisis Response Centre & HSC Emergency Psychiatry, WRHA Director of Research, department of psychiatry Research Summer school July 4, 2018

2. Disclosures • Relationships with commercial interests: • Grants/Research Support: • Canadian Institutes of Health Research • Brain and Behavior Research Foundation (NARSAD) • Young Investigator Award • Speakers Bureau/Honoraria: Nil • Consulting Fees: Nil • Other: Nil

3. Objectives • Learn how to identify and avoid bias in critical appraisal • Appreciate the difference between good and bad studies • Learn key questions to ask when reviewing a scientific article • Understand how to develop and convey the rationale for your study • The importance of a comprehensive and targeted literature review

4. PICO(T) • A useful approach to developing your research question, guiding your literature review, or critically appraising a study. • P Patient, problem, or population • I Intervention • C Comparison, control, comparator • O Outcome • T Time duration

5. Ten Key Questions in Critical Appraisal • Is the study relevant? • Does the study add anything new? • What type of research question is being asked? • Is the study design appropriate for the research question? • Did the study methods address sources of bias? • Was the study performed according to original protocol? • Did it test a stated hypothesis? • Were statistical analyses performed correctly? • Do the data justify the conclusions? • Are there any conflicts of interest? • Young JM & Solomon MJ. Nat ClinPractGastroenterolHepatol. 2009

6. Question #1So what? • Is the study important? • The best done study is irrelevant if it does not address an important question • What to look for • The introduction • Authors need to convince a reviewer that the topic/area is important to health, the public, etc. • For your study • Focus on current diseases of interest, new procedures, novel technologies

7. Question #2Does it add anything new? • What to look for • The introduction • Literature review – how thorough is the review, how well is it summarized? • The reviewer should have a clear idea that this is not only something new but also not simply an incremental addition to previous knowledge • You may need to do a quick literature search • For your study • How many studies on your topic exist? What are their limitations? What questions have not yet been answered?

8. Is the Study Design Appropriate? • Relates to the research question • Efficacy of an intervention: RCT • Frequency of events/risk factors: observational studies

9. Hierarchy of Evidence • Systematic reviews and meta-analyses of RCTs • RCTs • Cohort studies • Case-control studies • Cross sectional surveys • Case reports

10. Do the Methods Address Bias? • Bias • When the results of a study deviate from the truth • When bias is due to the methodology of a study (vs. random chance) – this is called systematic bias

11. Pannucci CJ & Wilkins EG. PlastReconstuct Surg. 2010

12. Addressing Bias: Example 1 • Study: Examining rates of mental disorders in parents bereaved by MVC death of their child • Method: Parents surveyed in a bereavement support group

13. Addressing Bias: Example 1 • Examining rates of mental disorders in parents bereaved by MVC death of their child • Method: Parents surveyed in a bereavement support group • Potential Bias: Selection Bias • Attendees in bereavement groups have been shown to be healthier than non-attendees • Solution: conduct a population-based study of bereaved parents using administrative data

14. Addressing Bias: Example 2 • Study: Is cancer associated with suicide? • Design: Using CancerCare registries, examining all people in Manitoba who were newly diagnosed with cancer, and examining suicide rates within 1 year • Statistical analysis: unadjusted logistic regression models that examined the association with cancer diagnosis and suicide

15. Addressing Bias: Example 2 • Potential Bias: Confounding • Mental disorder are associated with both the predictor variable (cancer) and outcome variable (suicide) • Solution: multivariate regression that adjusts for mental disorders

16. Do the Data Justify the Conclusions? • Meta-analysis of antidepressant trials based on clinical study reports • Main outcomes: mortality and suicidality • Results: • Mortality: Odds ratio 1.28; 95% CI 0.40-4.06 • Suicidality: Odds ratio 1.21; 95% CI 0.84-1.74

17. Do the Data Justify the Conclusions? • Meta-analysis of antidepressant trials based on clinical study reports • Main outcomes: mortality and suicidality • Results: • Mortality: Odds ratio 1.28; 95% CI 0.40-4.06 • Suicidality: Odds ratio 1.21; 95% CI 0.84-1.74 • Conclusions: • “We suggest minimal use of antidepressants in children, adolescents, and young adults, as the serious harms seem to be greater.” • Sharma T et al. BMJ. 2016

18. CASPCritical Appraisal Skills Program • casp-uk.net • Has downloadable checklists that you can use for any type of study

19. Ten Key Questions in Critical Appraisal • Is the study relevant? • Does the study add anything new? • What type of research question is being asked? • Is the study design appropriate for the research question? • Did the study methods address sources of bias? • Was the study performed according to original protocol? • Did it test a stated hypothesis? • Were statistical analyses performed correctly? • Do the data justify the conclusions? • Are there any conflicts of interest? • Young JM & Solomon MJ. Nat ClinPractGastroenterolHepatol. 2009

20. Suggestions • Go toward the fear – don’t avoid! • Participate in journal clubs • Take opportunities to review for journals • Learn what top researchers are working on • See comments by other reviewers • If an option, participate in grant review committees • Remember: You don’t need to be a stats wizard! • The core of critical appraisal is based on the importance of the study and the approach to the question • Clinical knowledge goes a long way