Disclosures

1. A professional association Perspective on Systematic Reviews and their Utility for Guideline Development Joel Yager, M.D., Laura J. Fochtmann, M.D., M.S.,Robert Kunkle, M.A., Robert M. Plovnick, M.D., M.S.Executive Committee for Practice Guidelines American Psychiatric Association

2. Disclosures • The authors are with the Practice Guidelines Project of the American Psychiatric Association • Dr. Yager is Professor of Psychiatry, University of Colorado School of Medicine • Dr. Fochtmann is Professor of Psychiatry, School of Medicine, SUNY Stony Brook • Mr. Kunkle is Director, Practice Guidelines, APA • Dr. Plovnick is Director, Quality Care, APA • None of the authors has any financial disclosures pertinent to this presentation

3. Available APA Guidelines

4. Major Themes • Positive aspects of rigorous systematic reviews • Limitations, Challenges and Dangers of guidelines based on limited high quality evidence • Opportunities for AHRQ to help professional associations to develop trustworthy guidelines

5. Positive aspects of rigorous systematic reviews • Where robust evidence bases exit, systematic reviews are worth the effort and cost. • Such reviews will help develop trustworthy guidelines in the following ways:

6. Rigorous systematic reviews can help • equilibrate ratings regarding strength of evidence and, in turn, strength of recommendations, across guidelines • differentiate strong from weak recommendations • determine treatment priorities • reduce vague, underspecified recommendations • reduce biases resulting from informal group dynamics, differential power status of “experts”, and potential competing interests • increase transparency regarding evidence-base to guideline users • meet IOM standards

7. But, let’s face it… relying primarily on systematic evidence-based reviews poses…… • Limitations • Challenges • Dangers…

8. Limitations of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development • For many important clinical questions there’s not much high quality evidence out there. • Example from eating disorders

9. APA Guideline Recommendation Level [I] : 118 Level [II] : 57 Level [III] : 21 British NICE Recommendation Grade A: 3 Grade B: 13 Grade C: 85 A Sidebar on Evidence-Based Medicine:Strength of Recommendations inEating Disorders Practice Guideline

10. National Institute for Clinical Excellence (NICE)Recommendations

11. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials (BMJ 327 : 1459 Published 18 December 2003)

12. Limitations • The more precise the PICO-TS question with regard to patient population characteristics, comparative treatments, and meaningful outcomes, the less high quality evidence is available. • Think co-occurring medical and psychiatric comorbidities, patient preferences, prior longitudinal illness course, and treatment experience.

13. Evidence for PICO-TS questions? • For patients with paranoid schizophrenia who have developed tardive dyskinesia on first generation antipsychotic medications and who suffer from obesity and type II diabetes mellitus, which of the following treatments (clozapine vs. aripiprazole vs. olanzapine vs. perphenazine) will result in the most desirable outcomes with respect to psychiatric symptoms, psychiatric impairment, medical morbidity and mortality? • For currently depressed patients with bipolar disorder type I, with co-occurring alcohol and marijuana dependence, which of the following combinations of medications and psychosocial interventions will result in the most desirable outcomes with respect to psychiatric symptoms, psychiatric impairment, medical morbidity and mortality?

14. Limitations • When you factor in patient/family preferences and access to/costs of care, don’t expect to find much. • There will always be lots of wiggle room in framing assumptions and interpreting results. • Patients and practitioners seek clinical guidance for complex questions where randomized controlled trials are unlikely to ever be conducted. Are we supposed to ignore those needs?

15. Limitations • Common clinical questions may vary depending upon the level of generalist/specialist/subspecialist using the guidelines. • AHRQ funded EBM reviews may not address situations where lack of access limits clinicians’ abilities to offer best practices.

16. Limitations • AHRQ reports may focus on particular treatments (e.g. second generation antipsychotics) rather than on comparisons needed for choosing a treatment approach best suited for an individual patient.

17. Challengesof Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development • AHRQ evidence reports may cover only a fraction of a professional societies' guideline development efforts. Do we have “trustworthy” guidelines and “good enough” guidelines? • How are professional associations expected to pay for guideline development processes where AHRQ is not funding an EBC effort and/or where high quality evidence is not available?

18. Challenges: Determining when the juice is worth the squeeze. • What degree of pre-screening should be conducted and what type of evidence should be required before an EBC or PA heads off on detailed review? • For example, should a minimum of two high quality RCTs that specifically address a PICO-TS question be required before embarking on a detailed search? • How closely to the precise PICO questions should these studies be expected to adhere?

19. Dangers of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development • Systematic reviews may ignore or minimize important systematic biases, e.g. clinical trials design biases. • The IOM Systematic Review recommendations are admittedly riddled with strong recommendations for which there is no evidence base! A cynic might think that COI was at play.

20. Dangers of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development • “Without an assessment of hard, irrefutable measures of clinical decision-making that include individual preferences for treatment, decisions about the appropriateness of clinical treatments and variations of care cannot be made.” • Livingston EH, McNutt RA. The Hazards of Evidence-Based Medicine: Assessing Variations in Care. JAMA. 2011;306(7):762-763. doi: 10.1001/jama.2011.1181

21. Opportunities for AHRQ to help Professional Associations Developing Practice Guidelines • Establish and disseminate criteria for trustworthy pre-screening searches to determine potential value of full-scale systematic reviews. • Fund grants to professional associations and other guideline developing groups for conducting pre-screening searches.

22. Opportunities for AHRQ to help Professional Associations Developing Practice Guidelines • When extensive systematic reviews are conducted: • Organize and distribute evidence tables to enable guideline developers to use them to address clinically different questions than those for which they were initially created. • Require EBC evidence tables to link to the PubMed abstracts for all related RCTs. • Require EBC evidence tables to contain clinically relevant calculations, effect sizes, NNT, NNH, as well as proportion of individuals experiencing particular adverse effects and global outcomes.

23. Opportunities for AHRQ to help Professional Associations Developing Practice Guidelines • More broadly: • Work with other agencies to require investigators to post data in a manner suitable for incorporation into formal medical decision analyses. • Work with other agencies to require investigators to post individual subject or group descriptive data (means, standard deviations) split according to key factors such as age, sex, race/ethnicity even when the numbers are too small to analyze otherwise.

24. Opportunities for AHRQ to help Professional Associations and Other Groups Developing Practice Guidelines • For circumstances when only low-quality (or no) data is available: • Help develop acceptable standardized methods for acquiring expert consensus • Help develop policies and procedures authorizing the development of trustworthy guidelines based on available evidence plus expert consensus.

25. Thanks for Listeningjoel.yager@ucdenver.edu