Anticipating and Satisfying Clinician Information Needs: The Infobutton Manager Project

1. Anticipating and Satisfying Clinician Information Needs:The Infobutton Manager Project James J. Cimino, M.D. Department of (Bio)Medical Informatics Weekly Seminar March 6, 2003

2. Studying Clinician Information • Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med 1985 Oct;103(4):596-9 • Other studies: Osheroff, Forsythe, Gorman, Ely… • Types of needs • Patient-specific • General knowledge • Institutional • Resources • Medical record • Colleagues • Journals • Books

3. Do Clinicians Satisfy their Needs? • Needs are underestimated • Information needs may not be recognized • Resources are outdated or unavailable • Computers not used • Resolution may be deferred • Unsatisfied information needs are a potential source for errors

4. Needs Related to CIS Use? • Very small part of the clinician’s day • Very important part of the day • New information stimulates needs • Decision-making moments • Opportunity to resolve needs (using a computer) • Context might predict needs • Context might help resolve needs • Needs in this setting have not been studied

5. Context-Dependent Information Needs ! ? Age Sex Role Training Task Data Institution Context

6. Hypothesis: • Information needs are stereotypical • Information needs can be predicted based on context • Appropriate resource can be identified (automatically or in advance) • Resource can be queried automatically • Information from context can be used for query • “Infobuttons”

7. A Brief History of Infobuttons • 1991: “Medline Button” translated ICD9 to MeSH and performed Medline searches • 1994: World Wide Web • 1994: First CPMC Web-based CIS • 1996: WebCIS • 1996: Infobuttons link microbiology and medication data to PubMed and Micromedex • 1997: Infobuttons for patients (PatCIS) • 2002: Handheld, wireless infobuttons (PalmCIS)

8. Infobutton Manager Project • Study information needs of CIS users • Characterize needs based on context • Develop way to anticipate needs • Develop ways to resolve needs • Deploy anticipation/solution • Evaluate

9. Infobutton Manager Team Members • Jim Cimino • Jianhua Li • Sue Bakken • Vimla Patel • Mark Graham • Leanne Currie • Mureen Allen • Anne-Marie Ramierez • Your name here?

10. Methods • Determine information needs • WebCIS log files • WebCIS user questionnaires • Observational studies • Characterize information needs • Build Infobutton Manager (IM) • Build infobuttons • Populate IM database • Integrate IM with WebCIS • Evaluation

11. Insert WebCIS Demo Here

12. WebCIS Log File Methods • Health Resources page in WebCIS • Infobuttons in WebCIS • Problems: • Which user selected which resource? • What was the user was doing? • Solution: look at previous activity at same IP address

13. WebCIS Log File Records Mar 2 16:16:46 webcis3-i syslog: |WebCIS|ciminoj|156.111.145.61|3131313 |lab^2003-03-02-05.19.00.819650|view Mar 2 16:18:55 webcis3-i syslog: |WebCIS|ciminoj|156.111.145.61|3131313 |healthresource|view Mar 2 16:18:58 webcis3-i syslog: |Resources|(nobody)|156.111.145.61|| |url^http://salk.cpmc.columbia.edu:70/mdxdocs

14. WebCIS Log Files Results • Six months of log files • 2,607 users (1,034 per month) used a resource • 2931 IPs (2008 “156.x.x.x”) • User types: Housestaff: 51% Attendings: 34% Nurses: 5% Other: 10%

15. WebCIS Log Files Results 38,763 uses of resources 14,036 outside WebCIS (2339/month) 24,727 within WebCIS (4,104/month; 4/user) 19,913 uses of 24 resources (3,319/month) 4,814 uses of 3 infobuttons (802/month)

16. WebCIS Log Files Results Popular resources: Inside Outside Micromedex: 56.5% 51.7% Ovid: 7.4% 8.2% NYP Formulary 7.2% 7.1% Harrison’s 5.1% 5.1% PubMed 5.1% 5.1% Health Journals 5.0% 8.2% Medline Plus 3.3% 3.6% ICD9-CM 3.3% 2.9% Infobuttons Pharmacy/Micromedex 92.9% - Microbiology/PubMed 6.1% - Pharmacy/PubMed 1.0% -

17. WebCIS Log Files Results Health Resources: 19,913 times in 105 contexts: Lab results: 52.1% Radiology results: 9.0% Clinical notes: 7.4% Pathology results: 2.8% Visit list: 2.1% Pharmacy: 2.0% Discharge summary: 1.9% 80.4% Infobuttons: 4,814 times in 2 contexts: Pharmacy: 93.9% Lab results (Micro): 6.1% 100%

18. WebCIS Log Files Results Context: Lab Rad Note Card Path Vis Phar DSum MDX 53% 9% 3% 3% 3% 2% 2% 2% Ovid 49% 9% 6% 2% 3% 1% 1% 1% Formul. 52% 8% 6% 3% 3% 1% 3% 2% Harrison 49% 10% 5% 3% 4% 1% 1% 3% PubMed 48% 10% 7% 3% 5% 1% 3% 1% Journals 43% 8% 7% 6% 5% 2% 2% 1% Med+ 50% 8% 6% 3% 2% 4% 2% 1% ICD9 37% 12% 10% 2% 3% 2% 1% 2% Infobuttons MDX - - - - - - 100% - PubMed 86% - - - - - 14% -

19. WebCIS Log Files Results Context: Lab Rad Note Card Path Vis Phar DSum MDX 56% 57% 61% 59% 49% 42% 9% 62% Ovid 7% 7% 8% 5% 8% 4% 1% 6% Formulary 7% 6% 6% 6% 7% 3% 1% 7% Harrisons 5% 6% 4% 5% 6% 3% 1% 7% PubMed 5% 5% 5% 4% 8% 2% 1% 3% Journals 4% 4% 5% 9% 8% 3% 1% 3% Med+ 3% 3% 2% 3% 3% 32% 1% 2% ICD9 1% 4% 4% 2% 3% 2% .2% 3% Infobuttons MDX - - - - - - 84% - PubMed 3% - - - - - 2% -

20. WebCIS Log Files Results User: Housestaff Attending Nurse Other MDX 62% 61% 42% 27% Ovid 6% 7% >1% 1% Formulary 6% 5% 3% 5% Harrison 4% 3% 4% 3% PubMed 4% 5% 1% 1% Journals 3% 0% 3% 1% Med+ 2% 2% 3% 2% ICD9 3% 5% 1% 1% Statref 1% 1% 1% 3% PIER 0% 2% 3% 1% Infobuttons MDX 5% 6% 42% 53% PubMed 2% 1% 2% 1%

21. WebCIS Usage Logs Results • Take home messages: • Micromedex is a popular resource • Ovid edges out PubMed • Lab tests are a popular context for resources • No correlation between context and resource • Nurses and pharmacists like the Micromeex infobutton for microbiology • Still needed: • Context/resource/user type interactions • Concept-oriented contexts • Questions: • How much is based on knowledge of resources? • What are the questions in each context?

22. WebCIS User Questionnaires • Health resources page outside vs. inside WebCIS • Resources used, how often, how useful, anecdotes • Infobuttons used, how often, how useful, anecdotes • Other resources used

23. WebCIS User Questionnaire Results 1055 users in January 300 randomly selected 250 with valid e-mail addresses surveyed 42 responded 21 attendings 19 housestaff 1 nurse 1 pharmacist

24. WebCIS User Questionnaire Results How often do you access: Never<MonthMonthlyWeeklyDaily Attend 2 1 3 5 9 House 3 11 5 Nurse 1 Pharm 1 Attend 11 8 2 House 14 3 1 1 Nurse 1 Pharm 1 Resource Infobut

25. WebCIS User Questionnaire Results Why do you access: Patient QsGenl KnowOther Attend 20 12 3 House 18 10 1 Nurse 1 Pharm 1 1 Attend 10 House 4 Nurse Pharm 1 Resource Infobut

26. WebCIS User Questionnaire Results How do you access: NoneWC-LoginWC-MenuBothOther Attend 2 4 7 8 12 House 0 8 8 3 17 Nurse 1 Pharm 1 1 Resource

27. WebCIS User Questionnaire Results What resources and how useful: Very Somewhat Not _ AHAHAH MMDX 17 17 0 2 1 0 Ovid 10 11 1 2 0 0 PubMed 12 7 2 3 0 0 Medline+ 4 4 0 1 1 0 FullText 8 7 2 2 0 0 Harrisons 7 4 3 9 1 0 PIER 2 4 3 1 1 0 Formulary 2 3 2 1 1 0 Health J 6 9 0 1 0 1 Science J 3 4 0 1 1 0 STATref 1 2 6 0 1 1 ICD9 1 2 5 0 0 1

28. WebCIS User Questionnaire Results How useful are infobuttons: Very Somewhat Not _ AHAHAH Micro Org 3 3 5 1 2 0 Micro Sens 7 3 2 3 0 0 Pharmacy 6 1 0 2 0 0

29. WebCIS User Questionnaire Results How does self-reporting correlate with log files? 100 80 60 40 20 0 A A A A A H A A A A A H H H A H H H H A A A H H H A H H H H H A H A A H A A H Never <Monthly Monthly Weekly Daily Health Resources N P

30. WebCIS User Questionnaire Results How does self-reporting correlate with log files? 25 20 15 10 5 0 A H A H A P A H H A H H A A A H H H H H H A H H A A A H A H A N A A A A A H A H A Never <Monthly Monthly Weekly Daily Infobuttons

31. WebCIS User Questionnaire Results • Anecdotes: • Avoid errors (drug interactions) • Identify correct doses • Identify pills • Prepare publications • Teaching • Understand abbreviations (sensitivity tests) • Other desired resources: • Up To Date • NEJM • MDConsult • Books

32. WebCIS User Questionnaire Results • Resources are used and liked • Resources are used within WebCIS • Resources are used to answer patient-specific questions • Use of resources may be overestimated (can’t tell for sure) • Infobuttons are used and liked • Additional resources requested • Questions: • How knowledgeable are users about resources? • What are the context-specific questions

33. Observational Studies • Connect portable usability lab to clinical workstation • Ask users to think aloud about information needs • Videotape screen and audiotape users • Transcribe tapes • Code transcripts, using tapes • Analysis of coding

34. Portable Usability Lab User’s Workstation 75 foot cable Video Monitor Converter Controller Microphone Video Converter Headphones VCR Cassette Recorder

36. Observational Results • Sites: • Medical /Surgical • CCU • Ambulatory Care • Number of Days: • 7 different days • Varying days of the week • Varying times of day • Total hours of tape: 15 hrs 29 minutes • Total Number of events: 154

37. Observational Results Total Number of events: 154

38. Observational Results Total Number by User Type

39. Event Coding • Types of Events: • Foreground/Background • Implicit/Explicit • Patient/Institution/Administrative/Domain • Types of Resources • Human/Computer/Paper • Types of Outcomes • Success/Deferred/Failed

40. Coding Event Type • Foreground • Specific information about patient management • Problem, Intervention, Comparison, Outcome • Background • Root question: who, what, where,when, why, how? • Related to health disorder or aspect of a disorder

41. Coding Event Type

42. Coding Event Type • Explicit: Subject expresses an information need • Implicit: Subject uses a resource other than WebCIS

43. Coding Event Type

44. Coding Event Type • Patient • A patient related need • Institution • An information need that is related to the infrastructure of the institution • Administrative • An institutional information need that is related to a specific patient • Domain • A domain related information need that is related to a specific patient

45. Coding Event Type

46. Coding Resources Used • Human • The subject sought information from a human interaction in person or via telephone • Computer • The subject sought information from a computer interaction either webCIS or other internet resource • Paper • The subject sought information from a paper resource – patient chart or text book

47. Coding Event Type

48. Coding Outcomes • Success • The subject sought and found the desired information • Deferred • The subject did not seek an answer • Failed • The subject sought, but did not find desired information

49. Coding Outcomes

50. Coding Events • Need to correlate across event types • Especially need to look at types of needs that were deferred or that failed • Good news: • Information needs exist and can be identified • Needed resources are computer-based • Bad news (good news for us): • Many needs failed or deferred • Solutions will require integration of patient data with information resources • Questions: • What are the triggers? • What resources can help?