AHRQ State and Regional Demonstration Project Evaluation:

1. AHRQ State and Regional Demonstration Project Evaluation: Barbeque, Blues, Beneficial Technology Kevin B. Johnson, MD, MS Associate Professor, Biomedical Informatics, Vanderbilt University Medical Center Nashville, Tennessee

2. Project Overview

3. Project Drivers • Incomplete information increases admission rate and ED LOS • Poor communication impacts ED efficiency • Less patient data at the point of care impacts the rate of test ordering • Less patient data at the point of care impacts clinical outcomes

4. Data Exchange Has HUGE Potential ROI If data is exchanged across all facilities within the three-county region, the overall savings has potential to reach $48.1 million. Notes: 1 – Core healthcare entities include: Baptist Memphis, Le Bonheur Children’s Hospital, Methodist University Hospital, The Regional Medical Center (The MED), Saint Francis Hospital, St. Jude Children’s Research Hospital, Shelby County/Health Loop, UTMG, LabCorp, Memphis Managed Care-TLC, Omnicare

5. Learn,Collaborate,Design Get the Model right Build the Team ID the settings Implement Outcomes Research System Implementation and Evaluation Qualitative Research Qualitative Research

6. Key Aspects of Value Proposition • Qualitative Information • Costs • System usability • System use and utility • Clinical value (patient outcomes) • Dollars saved in care delivery process • Workflow efficiency gains

7. Qualitative Questions • Usability (focus groups in ED) • 1 month and 1 year after go-live • Barriers to implementing infrastructure (cognitive artifacts) • Evaluated in year 4 • Drivers for adoption (interviews of governing board and ED staff) • Evaluated in year 5

8. Costs • Personnel • Training • Community Meetings • Sales • Legal agreements • Organizational development • Equipment • Software development • Site-specific customizations and costs

9. Assessing Usability: The Questionnaire for User Interaction Satisfaction (QUIS) is a tool developed by a multi-disciplinary team of researchers in the Human-Computer Interaction Lab (HCIL) at the University of Maryland at College Park. The QUIS was designed to assess users' subjective satisfaction with specific aspects of the human-computer interface. The QUIS team successfully addressed the reliability and validity problems found in other satisfaction measures, creating a measure that is highly reliable across many types of interfaces. Questionnaire for User Interface Satisfaction

10. QUIS Details • Six scales • Eleven interface factors • Screen • Terminology/system feedback • learning factors • system capabilities • technical manuals • internet access • on-line tutorials, multimedia, voice recognition, virtual environments, and software installation

12. System Usability • Will conduct usability testing of SPL • Vanderbilt as pilot site for face validity and modifying QUIS • Will modify accordingly • Will survey Memphis ED attendings and nursing staff 1 month after go live and again 6 months later

13. System Usage and Epidemiology • Help desk use • Provider enrollment • Patient enrollment (RHIO in versus RHIO out) • Usage statistics • Latency • Downtime

14. Content Quality • Accuracy • Missing data • Categorization errors

15. Disease-specific Hypotheses • Improved neonatal GBBS management • Improved asthma controller med use • Improved ACE/ARB use in CHF • Improved immunization rates (flu, s.pneumo) • ?Others

16. ED Administrative Outcomes • Reduce inpatient admissions • Decreased duplicate testing (radiology and lab) • Decreased ED Expenses • Workflow efficiency • Costs per visit

17. Workflow change • Activity-based costing • Model construction at Vanderbilt • Model validation in Memphis • Use model to construct activity matrices in EDs under study • Assess how activity matrices change pre and 1 year post implementation

18. Model Construction: Data Collection • Trained observers will document • Key transition points in information flow: • Eliciting prior medical history • Triage and treatment processes • Disposition/discharge from ED • Data Elements • Activity performed • Agent (RN, MD, Clerk, etc.) • Start-Stop times (hh:mm:ss)

19. Sample of Activity-Based Data

20. Activity-Based Estimates (Aggregate)

21. Data Sources Outcome of interest Record Accessed During Study No RHIO record Accessed Patient with Data in vaults Patient without Data in vaults

22. Using the Vault as the Primary Data Source for Outcomes = LOS of all encounters in vaults (before go live) Baseline LOS LOS of all encounters in vault whose records were not accessed LOS of all encounters in vault whose records were accessed = vs Change in LOS

23. Rollout stable on off Clinical Outcomes Methodology • Pre-post • Easy to implement • Will not impact rollout or clinic flow • Sensitive to existing trends

24. Rollout stable off Other Approaches • Assign times of day randomly to downtime status • Assign patients randomly to control group (no data for them) • Assign retrieval events randomly to control (i.e., no result) retrievals

25. Covariate Analysis • ED (site) characteristics survey to be completed by ED Administration • Readiness survey to be completed by ED administration and clinical leadership

26. IRB Approach: Five Approvals • Activity-based costing (approved) • Usability, readiness and demographic survey (letters of cooperation) • Baseline data for administrative measures and activity costing • System content quality • Disease-specific hypotheses

27. Thanks!