Using Audit and Feedback to Improve Quality of Care

1. Using Audit and Feedback to Improve Quality of Care December 7-8, 2012

2. What are we talking about here? A definition of AuditandFeedback: summary of clinical performance over a specified period of time andprovision to health care provider(s)/organization(s) NOT: reminders at the point of care NOT: relay of individual, patient-specific clinical data between providers

3. Why Talk about Feedback? “the limiting factor to Self-Directed Learning for practicing physicians may be their limited ability to accurately self-assess” Davis et al. JAMA 2006 Sep 6;296(9):1094-1102. “You can’t manage what you can’t measure”

4. Feedback and Behaviour Change New behaviours Performance Feedback Performance Feedback Old behaviours

6. INTRODUCTIONS Objective To develop a research agenda to improve the effectiveness of audit and feedback interventions to reliably improve quality of care. Goals: • Establish the foundation for a program of research aimed at improving the effectiveness of AF interventions 1a) prioritize elements of AF design that should be tested 1b) identify settings that provide opportunities for testing AF design 1c) utilize methodology to efficiently test and improve upon AF interventions 2. Develop a knowledge translation plan: 2a) integrated collaboration with clinical, administrative, policy stakeholders 2b) disseminating the results of the meeting 2c) developing a wiki-based platform to support ongoing collaboration 2d) establishing plans for new operating grants that will carry out the agenda developed in the meeting

7. Agenda

8. Results from the Cochrane Review • What do we know about the impact of AF? • Cochrane Review and Meta-regression • Cumulative Meta-analysis (not yet published)

9. Findings of 2006 Cochrane Review • 88 comparisons from 72 studies • dichotomous outcomes median adjusted risk difference of compliance with desired practice = 0.05 (IQR 0.03 to 0.11) • “Intensity of audit and feedback might also help to explain variation in the absolute effect (p = 0.04).” • “Intensive”(individual recipients) AND ((verbal format)OR (a supervisor or senior colleague as the source)) AND (moderate or prolonged feedback) • “Non-intensive” ((group feedback) NOT (from a supervisor or senior colleague)) OR ((individual feedback) AND (writ-ten format) AND (containing information about costs or numbers of tests without personal incentives)) • “Moderately intensive”(any other combination of characteristics than described in Intensive or Non-intensive group). Jamtvedt G et al. Cochrane Database of Systematic Reviews 2006, Issue 2. Art. No.: CD000259. DOI: 10.1002/14651858.CD000259.pub2.

10. Unclear how to “do” AF “an unreliable approach to quality improvement until we learn how and when it works best” Foy R. et al. BMC Health Services Research, 2005;5, 50.

11. 2012 Update: Changes in Review Methodology ANALYSIS OF HETEROGENEITY Meta-regression • Format (verbal; written; both; unclear) • Source (supervisor or senior colleague; review organization or employer; investigators; unclear) • Frequency (weekly; monthly; less than monthly; one-time) • Instruction for improvement (explicit goal; action plan; both; neither) • Direction of change required (increase behaviour; decrease; mix/unclear) • Recipient (physician; other health professional) • Baseline performance (continuous) • Risk of bias (high; unclear; low) Plus exploratory analyses: • Targeted behaviour (prescribing; test ordering; dm/cvd)

12. 2012 Update: Study Characteristics

13. Characteristics of Included Studies: Features of Feedback Design

15. Meta-Regression

16. Meta-Regression - Exploratory

17. Summary • AF is effective • One quarter of studies find relatively large effect (>16% aRD) • AF may be more effective when: • baseline performance is low, • the source is a supervisor or senior colleague, • delivered both verbally and written, • provided more than once, • and includes both explicit targets and an action plan • Targeted behavior plays an important role

18. Growing Literature; Stagnant Science?

19. Cumulative Meta-Analysis

20. Feedback and Behaviour Change New behaviours Performance Feedback Performance Feedback Old behaviours

21. Questions? Clinical: noah.ivers@wchospital.ca Personal: noahivers@gmail.com

22. Agenda

23. Role of Theory • What can it offer?

24. Illustrative Examples • Good, bad, and ugly

25. Moving forward Tension between theory and practice?

26. Agenda

27. Small groups: part 1 • Building a complete list of potential effect modifiers • What are the things to consider when thinking about doing audit and feedback interventions?

28. Small groups: part 2 • Prioritizing effect modifiers for testing • What are the top five things that we need to know in order to sort out how to make audit and feedback more effective?

29. Agenda

31. Agenda

32. Agenda

33. What is the impact of engaging the recipient in design of AF • What is the impact of adding to AF

34. 5 Research Questions • What is impact of engaging the recipient in design implementation vs. engaged in none of them • What is the impact of adding to AF: • Incentives/penalties – financial, CME credit + licensing • Tools/practise aid – eg. Clinical decision tool… • Practise redesign, coaches, facilitation, mentorship • Skill of the person providing feedback make a significant difference • Take the top quintile of AF studies and replicate their interventions • The 5 most important aspects of AF to study are: • Frequency, individual vs. group, evidence based, in person vs electronic delivery, number of targets • Developing strategies for replicating and implementing successful intervention in other settings, looking at

35. Dinner 6.30pm Empire Grill 47 Clarence Street (in Byward Market) (About 20 min walk)