Reducing Provider Cognitive Workload in CPOE Use: Optimizing Order Sets

1. Reducing Provider Cognitive Workload in CPOE Use: Optimizing Order Sets Yiye Zhang Rema Padman, PhD James E. Levin*,MD, PhD The H. John Heinz III College Carnegie Mellon University, Pittsburgh, PA, USA yiyez@andrew.cmu.edu; rpadman@cmu.edu MedInfo2013, Copenhagen, Denmark *Dr. James E. Levin passed away on February 11, 2013. We are greatly indebted to his vision, contributions and support that made this study possible.

2. Introduction • Significant healthcare delivery challenges in the U.S. and worldwide • Cost, quality, safety, efficiency, satisfaction • 1999 landmark Institute of Medicine report indicated that 44,000 to 98,000 Americans die each year from medical errors1 • Medication errors are a major component of these errors2 • Potential of healthcare information technology (HIT) • Traditional paper prescription prone to errors due to poor legibility and miscommunication during patient transfers • Computerized provider order entry (CPOE), a core feature of the electronic health record (EHR) system, has been recommended to mitigate errors in inpatient orders • Institute of Medicine. (1999). To Err is Human: Building a Safer Health System. Retrieved March 28, 2004, from http://www.iom.edu/ • Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA 2001;285(16):2114–20.

3. Computerized Provider Order Entry (CPOE) • CPOE systems are software applications designed to enhance patient safety by allowing clinicians to enter inpatient orders electronically; Used by one third of US hospitals1 • CPOE systems have been shown to improve patient care through better order legibility, reduced rule violations, improved clinician compliance with best practices, and advanced clinical decision support features2 • Within CPOE, order sets allows clinicians to place multiple, relevant orders for each patient with fewer mouse clicks, thus the creation of order sets is an important prerequisite to successful CPOE implementation and use • HIMSS Analytics: Healthcare IT Data, Research, and Analysis. http://www.himssanalytics.org/hc_providers/emr_adoption.asp. • Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. Computerized physician order entry and medication errors in a pediatric critical care unit. Pediatrics. 2004 Jan;113(1 Pt 1):59-63.

4. Order Sets • Collection of individual orders commonly entered as an aggregate for a specific clinical purpose or procedure • Typically developed by clinical experts in a generic format • Support clinicians in high risk situations by serving as expert-recommended guidelines, reducing prescribing time by eliminating unnecessary duplication of work, and increasing clinician compliance with the current best practices1 • 1. Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of pre-constructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):322-9.

5. Challenges with Order Set Usage • Large Variability in Order Set Usage1 • Difficult to maintain order set content and combinations up-to-date with current best practices • Lack of involvement in order set development by physicians who are familiar with both the guidelines as well as the actual practice • Providers switch to ‘a la carte’ orders instead of ordering from order set, potentially resulting in unsafe and inefficient ordering process • Poorly designed order sets contribute negatively to treatment quality by exposing users to excessive mouse clicks (physical cost) and cognitive workload (cognitive cost) • 1. Zhang Y, Levin JE, Padman R. Data-driven Order Set Generation and Evaluation in the Pediatric Environment. AMIA Symposium Proc. 2012.

6. Physical and Cognitive Costs One order set All a la carte • “poor usability--such as poorly designed screens, hard-to-navigate files, conflicting warning messages, and need for excessive keystrokes or mouse clicks--adversely affects clinical efficiency and data quality” - a recent report from Agency for Healthcare Research and Quality (AHRQ)1 • There is a need to design features of CPOE according to “human factor best practices.”2,3 Physical/cognitive cost Optimal number of order sets Number of order sets • 1: Schumacher RM, Lowry SZ. NIST Guide to the Process Approach for Improving the Usability of Electronic Health Records. 2010. • 2. Wright P, Lickorish A, Milroy R. Remembering While Mousing: The Cognitive Costs of Mouse Clicks. SIGCHI Bulletin. 1994. • 3. HorskyJ, Kaufman DR, Oppenheim ML, Patel VL. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. J Biomed Inform 2003;36(1–2):4–22.

7. Research Question • Can the development of order setsbe automated using historical ordering data to learn new order sets that are evidence-based, up-to-date with current best practices, and incur least physical/cognitive costs?

8. Study Setting • Children’s Hospital of Pittsburgh (CHP) of UPMC, a HIMSS level 7 pediatric facility • Since October 2002, all inpatient orders at CHP have been entered directly into the CHP eRecord (Cerner Millenium™) • Over 12,000 pediatric patients admitted each year • Over 10 million order actions in total • On average, a patient at CHP is hospitalized for 5.5 days, and during that time, 36 unique individuals create 871 order actions • ~ 2000 departmental, in-house order sets

9. Sample Appendectomy Orders Order set being utilized A la Carte being utilized

10. Distribution of Orders: Appendectomy Minor Solid blue: a la carte, dotted yellow: order set

11. Optimization and Clustering Models Minimize Cognitive Click Cost (CCC) Subject to 1) ‘Default option’ choice constraints 2) Cluster formation constraints 3) Time interval constraints Approach: • Order set development from order items

12. Eliciting Cognitive Costs • CCC with expert estimate (CCCE): Expert input • CCC based on survey result (CCCS): Survey of 15 subjects including physicians and nurses • Each survey contains 6 questions with sub-questions, asking subjects to estimate the time it takes them to perform tasks while placing orders with large, mid-size, and small order sets

13. Approach: Order Set Development • Determine optimal time interval and number of order sets within the time interval that minimize MCC/CCC • Cluster orders using bisecting K-means clustering within each time interval • Map new order set assignment back to historical treatment data to evaluate goodness of clustering using MCC/CCC and coverage rate

14. Ex. Fixed patient, time, and order Time interval 1, 2,…., n ON if more than 80% patients use; OFF otherwise Order Set 2 Order Set 1 Order D Order A Order E Order B Order C Such that CCC can be lowered !

15. Results: Significant Reduction in CCC and Increase in Coverage Rate ***: p-value less than 0.01, **: p-value less than 0.05, *: p-value less than 0.1

16. Closer Look: Appendectomy Minor - CCCE • 12 order sets used per patient on average in training set • 6 order sets used per patient on average in test set

17. Sample Case: Under Current Order Set CCCE = 20.3, CCCS = 76, number of actual mouse clicks (MC) = 15

18. Sample Case: Under New Order Sets CCCE = 12.9 (36.4% drop ), CCCS = 23.1 (69.6% drop), MC = 13 (15.4% drop)

19. Conclusions • Order Set development based on data-driven approaches is promising • Can be generalized for not only CHP order sets but also for order sets in other settings with different workflows

20. Limitations and Challenges • Large variations in ordering patterns • Influence on usage by the current order sets • Rare combinations of orders need to be addressed separately in a data-driven approach • Constant CCC weights assumption • Incorporation of new scientific evidence

21. Future Work • Develop new approaches and extend/test current methods on other diagnoses and in other settings1 • Implemented an order set development platform and tested on pneumonia patients • Incorporate alternate methods using heuristic optimization • Evaluation by physicians on the usability and clinical validity of newly created order sets • Currently looking for interested institutions to partner on the clinical evaluation studies 1: Zhang Y, Padman R, Levin JE. Data-driven Order Set Development Using Tabu Search. Heinz College Working Paper, Carnegie Mellon University, Pittsburgh, PA, May 2013.

22. Relevant Publications • Zhang Y, Padman R, Levin JE. Clustering Methods for Data-driven Order Set Development in the Pediatric Environment. INFORMS 2012 DM-HI Workshop Proc., October 2012 • Zhang Y, Levin JE, Padman R. Data-driven Order Set Generation and Evaluation in the Pediatric Environment. AMIA Symposium Proc., November 2012. • Zhang Y, Levin JE, Padman R. Toward Order Set Optimization Using Click Cost Criteria in the Pediatric Environment. HICSS-46 Proc., January 2013. • Zhang Y, Padman R, Levin JE. Data-driven Order Set Development in the Pediatric Environment: Toward Safer and More Efficient Patient Care. Heinz College Working Paper, Carnegie Mellon University, Pittsburgh, PA, December 2012.

23. Thank you!Questions?