Ron Li, MD Clinical Informatics Fellow Stanford University School of Medicine ronl@stanford

1. A “bottom up” approach to curating electronic order sets • S22: Identifying Medication Risk Ron Li, MD Clinical Informatics Fellow Stanford University School of Medicine ronl@stanford.edu

2. Disclosure • I and my spouse/partner have no relevant relationships with commercial interests to disclose. AMIA 2017 | amia.org

3. Understanding how clinicians deliver care is an important piece of the Learning Health Care System Institute of Medicine. Best Care at Lower Cost. (2012)

4. Automatically generated ethnographic data

5. Digital trace data in healthcare

6. Adverse drug events are responsible for up to 770,000 inpatient injuries and deaths annually, most of which are from prescribing errors. These adverse events affect nearly 5% hospitalized patients in the United States. AHRQ Patient Safety Network. Computerized provider order entry. AHRQ PSNet Patient Safety Primer 2015 (2015). Available at: https://psnet.ahrq.gov/primers/primer/6/.

7. Unintended consequences of clinical decision support (CDS) on ordering medications Clinical Decision Support How is medication ordering affected? Clinical decision making Workflow

8. Errors mediated by automation bias • Users tend to over-accept and over-rely on computer output “as a heuristic replacement of vigilant information seeking and processing, which can lead to both errors of omission and commission. • Error of commission: a CDS generates an incorrect recommendation, prompting clinician to order a medication he otherwise would not have ordered • Error of omission: a CDS fails to alert a clinician of a patient’s risk for severe sepsis. The clinician, who is accustomed to relying on the CDS alert, does not do as careful of an assessment and misses the diagnosis. • Goddard, K., Roudsari, A. & Wyatt, J. C. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J. Am. Med. Informatics Assoc.19, 121–127 (2012).

10. How does automation bias affect the use of the electronic order sets? • Electronic order sets are present in more than 80% of inpatient EHRs and are important for standardizing/increasing efficiency of workflow and clinical decision making. • Does automation bias cause order sets to also promote incorrect mediation ordering?

11. Order sets are currently curated based on the ad hoc insights of a small group of people: expert opinion, anecdotal experiences, guidelines, incident reports. This “top down” approach may result in a inaccurate and biased understanding of how order sets are affecting clinician ordering behavior, resulting in sub-optimally curated order sets. Aim:to develop a “bottom up” approach using EHR audit trail data to systematically capture ordering errors from order sets

12. Methods • All inpatient medication orders from order sets at Stanford Hospital were identified from 2008-2017. • Frequency of medication prescribing errors (in the form of near misses) estimated by frequency of orders discontinued within 30 minutes of ordering (“dw30m”)1. • Medication prescribing error rate calculated for each order 1) when part of any order set and 2) when part of a specific order-order set pair. • Medication error rates are compared to the error rates for when the medication was ordered not from an order set (relative risk). • Koppel, R. et al. Identifying and Quantifying Medication Errors : Evaluation of Rapidly Discontinued Medication Orders Submitted to a Computerized Physician Order Entry System. J. Am. Med. Informatics Assoc.15, 461–465 (2008).

13. Inpatient medication orders at Stanford Hospital (2008-2017)

14. Medication error rates per medication when ordered from order set • After excluding medications with the bottom 25th percentile ordering frequencies, there were 519 unique medications with at least one order discontinued within 30 minutes.

16. Error rates tend to be lower with order sets, but higher for a subset of medications Relative Risk < 1 Relative Risk >1

17. Identifying the medication-order set pairs most prone to prescribing errors • After excluding order sets and medication-order set pairs with the bottom 25th percentile ordering frequencies, there were 2678 unique medication-order set pairs with at least one order discontinued within 30 minutes

19. Relative Risk >1 Relative Risk < 1

20. Data driven order set curation

21. Data-driven order set curation

22. Discussion • Medication prescribing errors from order sets are relatively uncommon, but are high (as high as 29%) for a subset of medications. • Most medications have lower prescribing error rates when ordered from an order set, but a subset of medications are more likely to be prescribed in error when from order sets. • These data can be used to target medications and order sets for further investigation during the order set review process. • Limitations: results are unadjusted for differences between order set/non-order set medication orders and are from a single institution

23. Acknowledgements • Jonathan Chen, MD PhD • Christopher Sharp, MD • Jason Wang • Connie Taylor, RN • Stanford Clinical Informatics Fellowship • Stanford Department of Medicine • Stanford Division of Biomedical Informatics Research • Stanford Research IT

24. AMIA is the professional home for more than 5,400 informatics professionals, representing frontline clinicians, researchers, public health experts and educators who bring meaning to data, manage information and generate new knowledge across the research and healthcare enterprise. AMIA 2017 | amia.org

25. Thank you! Email me at: ronl@stanford.edu