Evaluation of electronic medical records - a clinical task perspective Presentation of thesis

1. ? Error Norwegian University ofScience and Technology, Trondheim Evaluation of electronic medical records- a clinical task perspectivePresentation of thesis Hallvard Lærum

2. The Kvalis Project • Quality assurance of electronic medical records in hospitals • Funded by The Research Council of Norway • Interdisciplinary, three PhD fellows • Sociology: Gro Underland • Informatics: Gunnar Ellingsen • Medicine: Hallvard Lærum

3. Evaluation of electronic medical records Descriptive aspects What is the level of functionality of current EMR systems? Where are they found? Explorative aspects How may the effects of various EMR systems on physicians’ clinical practice in various hospitals be evaluated? Investigations How does the method perform in various contexts? Reliability and validity studies

4. Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University The 3rd Scandinavian University Course in medical informatics Building a researchnetwork Establishing contact with research centers internationally Keeping up a website Establishing contact with hospitals Arranging monthly seminars Establishing contact with EMR vendors Arranging meetings

5. Descriptive aspects of the thesis EMR systems are very common in Norwegian hospitals (80% of hospital beds as of 2002) No systematic evaluations of Norwegian EMRs published by 1999 The systems are essentially document-based, covering the narrative parts of the medical records, lab data and radiology reports.

6. Explorative aspects of the thesis 1 Considerations for development of an evaluation method • Applicable to various EMR systems and hospitals • Not tied to a single system or hospital • Carried out using limited resources • Fast results • Used in operational reviews, incorporating organizational issues • Laboratory simulations less feasible • Involve measurement close to the site of impact • Involving how clinical work is done

7. Explorative aspects 2 The first task inventory • Direct observations, 40 h • 20 h St.Olavs Hospital, 20h RiTø (Gunnar E.) • 7h transcribed verbatim • Basis for hierarchical task analysis • Noticed that the PCs were not much in use • The first task inventory: 104 tasks • Too large for practical use • Task wording rather technical • Covert tasks not included (e.g. cognitive tasks)

8. Video observations – Flow of information • Information seeking behaviour • Looking for incidents where needed information was not found • Tagging of video recordings • Stimulated recall* not sucessful • Too time-consuming (and boring!) for an average clinician • Recruitment problems • Prohibitively resource intensive • 9 patients, 4.5 h video recordings, two physicians • Noticed that PCs were not much in use * Kushniruk & Patel 1995 OntoLog Courtesy of Jon Heggland, IDI NTNU

9. ! Simplification of the task list • Task list to be used in questionnaires and interview guides • Considerations for the process • Emphasis on information-related tasks essential for patient care • Wording adapted to clinicians’ way of speaking • Covert tasks were included • Tasks suitable for most specialties were retained • Tasks representing negligible work were deleted • Gorman’s five information needs were taken into account • Tasks should be supported by functionality found in current EMR systems or in those expected in the near future • Multiple iterations by work group (2 physicians, 2 informaticians)

10. A list of relevant tasks for the clinician

11. The first Questionnaire was compiled • Translated sections (two-way translation validation involving native english speaking translator) • End User Computing Satisfaction scale (Doll & Torkzadeh) • Global user satisfaction questions (SGUS, Anderson & Aydin) • Computer literacy (Brown & Coney) • New sections • Frequency of PC /EMR use for the 23 clinical tasks • Basic availability of computers Pilot study (n=22)

12. Investigations in this thesis • National survey, 2001 • Local survey in Arendal, 2002 • Validation and reliability studies, 2003

13. The national survey Feb 2001 • Survey • 314 physicians in 32 hospital units in 19 hospitals • 227 (72.3%) responded after one reminder (135 respondents were reminded) • Telephone interviews with key IT personnel in each hospital • What clinical tasks could be performed using the EMR system? • A set of minimal functionality requirements, incl. requirements for integration of external systems

14. Missingresponses! The section covering PC/EMR use

15. 0 % 50 % 100 % 50 % 100 % 50 % 100 % Implemented functions DIPS DocuLive Infomedix • There is a considerable difference in implemented functions between systems General Medical knowledge Aggregated data Supplementary investigations Actions Information to patient Communication and verification 1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge 8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations Percent of respondents offered minimal functionality for the given task

16. 0 % 50 % 100 % 50 % 100 % 50 % 100 % Use EMR Use both EMR and other software Type of program not stated Implementation Use and implementation of EMR systems DIPS DocuLive Infomedix A lot of implemented functionality appears not to be used General Medical knowledge Aggregated data Supplementary investigations Actions Patient info Communication and flow of information 1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge 8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms 19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations Percent of respondents offered functionality/ using it Cut-off: Respondents answering ”half of the time” or better are users

17. 0 % 50 % 100 % 50 % 100 % 50 % 100 % Use EMR Use both EMR and other software Type of program not stated Implementation Use and implementation of EMR systems DIPS DocuLive Infomedix Some physicians enter daily notes themselves General Medical knowledge Aggregated data Supplementary investigations Actions Patient info Communication and flow of information 1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes (i.e. progress notes) 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge 8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms 19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations Percent of respondents offered functionality/ using it

18. 0 % 50 % 100 % 50 % 100 % 50 % 100 % Use EMR Use both EMR and other software Type of program not stated Implementation Use and implementation of EMR systems DIPS DocuLive Infomedix Obtaining lab data (and other results) is popular, order entry is not General Medical knowledge Aggregated data Supplementary investigations Actions Patient info Communication and flow of information 1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge 8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms 19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations Percent of respondents offered functionality/ using it

19. 0 % 50 % 100 % 50 % 100 % 50 % 100 % Use EMR Use both EMR and other software Type of program not stated Implementation Use and implementation of EMR systems DIPS DocuLive Infomedix Big surprise: Nobody is using the EMR system to write prescriptions or complete sick-leave forms General Medical knowledge Aggregated data Supplementary investigations Actions Patient info Communication and flow of information 1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge 8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms 19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations Percent of respondents offered functionality/ using it

20. Why this lack of EMR use? – 1Answers provided by the questionnaire is limited • Low computer literacy? • Con: The physicians had at least a basic knowledge of computers (average score 72.2 ±1.6, 92% owned a computer) • Pro: Specific EMR system training may be needed I wish the EMR could be more like WordStar Where’s the PC when I need it? • Lack of available computers? • Con: Some PCs were available: 93% a computer in their office, 97% had a computer available in other rooms used for clinical work. • Pro: Clinicians’ work is not stationary. No hospitals have yet implemented mobile computing, and no hospitals can afford a PC in every room (personal communications)

21. Why the lack of EMR use? - 2 • Low usability of the EMR system as a whole? • Prescriptions: Navigating to the prescription module, finding the correct medication, typing dosage, administration route and package size and printing on the correct type of paper may represent more work than doing it by hand. • Sick-leave forms: Slow system response times (12-15 sec) I know my pre- scriptions by heart! I’d like to find all the information in one place Order entry is a nurse’s job! • The EMR is not complete • Pro: Until all relevant clinical information is found in the EMR, it will serve a secondary role. • Pro: The functionality of current EMR systems is limited Typing is a secretary’s job! • Resistance to new work roles? • Pro: The usage patterns found in the national study are conform to traditional work roles. • Pro: Work role issues were the most prevalent theme in the answers to the open-ended questions in the validity study (Paper 4). Follow-up studies in each hospital are necessary

22. Local study of EMR system in Aust-Agder Hospital, Arendal(Paper 2 and 3) • The paper-based medical record is scanned and obliterated! • Questionnaire rev.2 (versions for medical secretaries and nurses developed separately) • 70 of 80 physicians (88%) • 79 of 85 medical secretaries (93%) • 172 of 235 nurses (73%) • Interviews • 8-12 representatives of each profession, 0.5-2 h interviews

23. The EMR in Aust-Agder Hospital

24. 2nd revision of the questionnaire

25. Physician’s use of the regular EMR at Aust-Agder Hospital Much higher frequency of EMR use than in other hospitals having the same system

26. The physicians reported a less frequent use of the scanned document images

27. Physicians: Change in ease of performing the clinical tasks Most tasks related to information retrieval are reportedly easier using the system, but 33% of internists found task 1 and 2 more difficult

28. User satisfaction of physicians The physicians are relatively satisfied with the regular EMR, but not with the use of the scanned document images

29. Validation and reliability studies3rd revision of questionnaire • Test-retest reliability study • 37 of 96 physicians (39%) from three hospitals having different EMR systems completed the questionnaire two times. ”Test” questionnaire: 55.2% (52 of 96), ”Retest” questionnaire. 71% (37/52). • Content validity study – interviews • 10 physicians, 1 h interviews • Relevance of tasks, estimation of accuracy of answers to task-oriented questions, themes in open-ended questions. • Criterion validity using data from local and national studies • Criteria • General information retrieval vs. tasks involving information retrieval • Overall work performance vs.task performance • Task performance vs. user satisfaction • Majority of tasks covered, median correlation coefficient 0.445-0.513

30. Test-retest reliability: Weighted kappa was generally high Problematic tasks were related to functionality not available locally, and problems in discering the EMR from other software

31. Themes appearing in the interviews • Work role issues (8/10 physicians) – resistance to doing ”clerikal” tasks • Tasks 10 and 19 • The third method would be the "ask-the-nurse" method. This is convenient, though, then I may do other things. [In the future] It could be that it will be so easy to do it, that I could do it myself…if it's really easy, a completely negligible task. But if it takes some time..if I have to wait or something..then I feel that it should be a medical secretary's task, at least in a hospital.(respiratory diseases) • Various wording problems (7/10 physicians) • Tasks 16, 4 and 21 • I don't understand what you mean with "directly"…write orders on the [order entry form], request or order an operation…one other [example] is requesting treatment by physiotherapist (orthopedy) • Questions regarding use of non-existent functionality (7/10 physicians) • Task 3 • Some questions are difficult to answer, as we can't log on [to the EMR system] and find results from X-ray investigations (plastic surgery) • Distinguishing EMR from other software or media (6/10 physicians) • Task 4 • Is [the separate lab system] regarded as a part of [the EMR system]? (neurology)

32. Considerations of self-reported use • Sources of error in self-reported use • Telescoping and other memory-related effects • Implicit expectations of the questionnaire • Willingness to respond • Strategic responses • Other • There seem to be more reasons for the user to report too high frequency of use than too low. • Use of the EMR system may not necessarily benefit the patient • EMR system used the wrong way • Unintended effects of using the EMR system • Logical errors in the EMR system

33. Application of the task list and the questionnaire • The questionnaire may be used for screening, focusing the evaluation effort and providing a basis for further exploration • Separating reported use from non-use • The combination of EMR use and task performance may identifiy problematic aspects of the EMR system. • The questionnaire should always be combined with a qualitative study to investigate the ”why” (and validate the findings)

34. Thank you!

35. Extra information

36. National study: Inclusion of hospitals • Inclusion of hospitals • The EMR should at least contain medical narratives (admission reports, progress notes, discharge reports), directly or indirectly updated by and electronically available to the physicians. (minimal functional requirement for task 1)

37. National study: Lessons learned in this survey: • Missing responses • ”Use EMR/Other program” questions frequently not answered (median 10%, IQR 5-15%) • Two hospital units (8 physicians) had to be excluded post hoc • 1: Minimal requirements for EMR implementation not met • 2: Two EMR systems in use simultaneously • Error discovered one year after publication • Task not supported after all in one hospital unit (11 physicians) • Lab results not integrated in the EMR in this unit, available in stand-alone system only. • Correction sent to BMJ (paper 1) Nov 2002.

38. Local study: User satisfaction of physicians, nurses and medical secretaries The medical secretaries are most satisfied, the physicians least satisfied with the system

39. Local study: Use of hospital information system for individual tasks

40. Local study: Results of interviews • All professions found that the patient data were more accessible when stored electronically. • Physicians (internists) found searching in scanned multiple documents time-consuming and difficult • The medical secretaries found that generation, handling, fetching and delivery of paper documents and logistics of paper-based patient records had diminished dramatically. • Nurses were still using pen and paper documenting their activities.

41. Changes in rev.3 of the questionnaire • Added ”escape” choices in task-related questions • Task not relevant for me • Task not possible using the local EMR system

42. Validation: Interviews 1: Accuracy Discordant interpretation for 1/10 in task 6, ”Obtain information on investigation or treatment procedures”

43. validation: Interviews 2: Relevance of tasks Task 8, ”Produce data reviews for specific patient groups” was not found relevant by a majority of the physicians. (not part of one’s job/ infrequently performed)