Yuval Shahar, M.D., Ph.D.

1. Temporal Reasoning and Planning in MedicineVisualization and Exploration ofTime-Oriented Medical Data Yuval Shahar, M.D., Ph.D.

2. To be effective, care providers and other decision makers need to be able to visualize both clinical data and their multiple levels of abstraction Larkin and Simon [1987]: the benefit of visual representations is mainly due to reduction of logical computation through the use of direct perceptual inference reduction of necessary search for information through the use of efficient graphical representations. The Need for Visualization of Information

3. Not all events are measured the same way Example: Birthday is accurate to the day, but age is accurate to the year The granularity may even change over time Example: Age is measured first in months, then in years The Granularity Issue

4. Year Month Week Day Hour Minute Second Smaller... Note, not all quanta can easily be expressed in terms of other quanta. For example, how many weeks are there in a year? How many minutes are there in a month? How do you deal with mixed granularities? Standard Time Quanta

5. Pretend there is no granularity problem Arbitrarily choose some time quantum and use that for all measurements Works best when the most logical quantum is large, say 1 day (or larger) If all you have is a day, how do you choose the right second? The Granularity Denial Approach

6. Suppose we know some event occurred on March 15th, 1999 Assume that the chosen granularity is seconds Do we record 3/15/1999, 12:0:0? Or, 3/15/1999, 0:0:0? Or, something else? Complications of Ignoring Granularity

7. Each event class is assigned a relevant granularity Some classes have multiple valid granularities A given timeline can specify only a small number of granularity levels Granularity: an Object-Oriented Approach

8. Object-Oriented Example

9. Measure each event with the most relevant granularity When viewing data at a finer granularity, introduce uncertainty Uncertainty can include up to 6 degrees of freedom (start, duration, stop) The Interval-Uncertainty Approach

10. Nov. 3, 1999 can be represented as: 11/3/1999, 0:0:0 — 11/3/1999, 23:59:59 Feb. 6, 1986 at 13:37 can be truncated: 2/6/1986 Or rounded: 2/7/1986 Uncertainty Examples

11. More Uncertainty Examples Not all possible values need be stored: Start Maximum Duration Minimum Duration Start Stop

12. A timeline is a tuple <E, M>, where E is a finite set of events containing at least the special null event, and M is a measure function M:ER+. The measure function M assigns a temporal offset to each event in E. Timelines Cousins, S.B., and Kahn, M.G. The visual display of temporal information. Artificial Intelligence in Medicine3(6) (1991) 341–357

13. New: creates a new timeline containing only the null event. Add: adds an event e to an existing timeline, may increase length of timeline as a side-effect. Slice: remove events from one or both ends of a timeline, moves the null event as needed. Timeline Operators

14. Filter: remove all events not satisfying some predicate P; the null event cannot be removed Overlay: merges two timelines. If common events do not coincide, they are copied Timeline Operators (cont.)

15. Timeline Operator Examples Input Operation Output Slice(e1, e2) e1 e2 e1 e2 Filter(“b-ness”) b1 a1 b2 a2 b1 b2 Overlay (a, a) a b c a b c Overlay (b, b) a b c a b c c’ b c

16. A grounded timeline is one that can be directly mapped to a calendar (e.g., a Julian calendar) Slice and Filter are ‘safe’. Overlay and New may cause a timeline to become ungrounded Grounded Timelines

17. Prototype used to display diabetes patient data over time Created the formal definition of, and operators for, a timeline Provides GUI for manipulating timeline operators . Time Line Browser

18. For the previous week, display the patient’s logbook and personal calendar Summarize the patient’s blood sugar at breakfast, lunch, dinner and nighttime over the past month Sample Queries

19. Logbook and Calendar Mild Illness Hospitalized X-ray Monday Tuesday Wednesday Thursday Friday Saturday Sunday Monday Work Vacation in Florida Work 12 13 14 15 16 17

20. Blood Sugar Summary For Breakfast, Lunch, Dinner, Night: B L D N Note that multiple slices have been overlaid to produce this result

21. Interactive composition of (temporal-abstraction) queries Visualization of query results Exploration of multiple levels of temporal abstractions The semantics of the query, visualization and exploration operatorsshould be domain independent, but shoulduse the terms and relations specific to each (e.g., medical) domain Knowledge-Based Visualization andExploration of Time-Oriented Medical Data:Desiderata

22. KNAVE= Knowledge-Based Navigation of Abstractions for Visualization and Explanation Interactive queries regarding both raw data and multiple levels of time-oriented abstractions derivable from these data Visualization and manipulation of query results Dynamic exploration of the results using the domain’s temporal-abstraction ontology The semantics of all operators do not depend on any specific domain, but the interface uses each domain’s ontology to compute and display specific terms and explore their relations KNAVE accesses the data through a temporal-abstraction mediator, such as IDAN Knowledge-Based Visualization andExploration of Time-Oriented Data:The KNAVE-I and KNAVE-II Projects(Shahar and Cheng, 1999, 2000; Shahar et al., 2003, in press)

23. The KNAVE-II Browsing and Exploration Interface [Shahar et al., AIM 2006] Overall pattern Medical knowledge browser Intermediate abstractions Concept search Raw clinical data

24. Moving Data Panels Around

25. Global Temporal-Granule Zoom (I)

26. Global Temporal-Granule Zoom (II)

27. Global Calendar-Based Zoom

28. Global Content-Based Zoom (I)

29. Global Content-Based Zoom (II)

30. Local Time-Sensitive Zoom

31. Exploration Operators • Motion across semantic links in the domain’s knowledge base by using the semantic explorer; in particular, relations such as: - part-of - is-a - abstracted-from - subcontext • Motion across abstraction types: state, gradient, rate, pattern • Application of aggregation operators such as mean and distribution • Dynamic change of temporal-granularity (e.g., days, months) • Explanation by context-sensitive display of relevant knowledge • “What- if” queries allow hypothetical assertion or retraction of data and examination of resultant patterns

32. Semantic Exploration of Temporal Abstractions

33. Explanation: A Classification Function

34. Explanation: A Persistence Function

35. Site: Palo Alto Veterans Administration Health Care System Eight clinicianswith varying medical/computer use backgrounds A second study used 6 additional clinicians and more difficult queries Each user was given a brief demonstrationof the interface The evaluation used an online database of more than 1000 bone-marrow transplantation patients followed for 2 to 4 years Each user was asked to answer 10 queries common in oncology protocols, about individual patients, at increasing difficulty levels A cross-over study design compared the KNAVE-II module versus two existing methods (in the 2nd study, users chose which): Paper charts Anelectronic spreadsheet (ESS) Measures: Quantitative: time to answer and accuracy of responses Qualitative: the Standard Usability Score (SUS) and comparative ranking Evaluation of the KNAVE-II Intelligent Visualization and Exploration Module [Martins et al., AIM 2008]

36. Direct Ranking comparison: KNAVE-II ranked first in preference by all users Detailed Usability Scores: The Standard Usability Scale (SUS) mean scores: KNAVE-II 69, ESS 48, Paper 46 (P=0.006) (more than 50 is user friendly) Time to answer: Users were significantly fasterusingKNAVE-II as the level of difficulty increased, up to a mean of 93 seconds difference versus paper, and 27 seconds versus the ESS, for the hardest query (p = 0.0006) The second evaluation, using more difficult queries and more advanced features of KNAVE-II, emphasized the differences even further: The comparison with the ESS showed a similar trend for moderately difficult queries (P=0.007) and for hard queries (p=0.002); on the average, study participants answered each of the two hardest queries 277 seconds faster using KNAVE than the ESS Correctness: Using KNAVE-IIsignificantly enhanced correctness versus using paper, especially as level of difficulty increased, even in the initial study (P=0.01) (99% accuracy with K-II versus only 78% paper accuracy, 1st study; 92% with K-II vs. 57% for ESS, 2nd study) The correctness scores for KNAVE-II versus ESS in the second study, which used more difficult queries, are significantly higher for all queries (p<0.0001) The KNAVE-II Evaluation Results(Martins et al. AIM 2008)

37. VISualizatIon and exploration of Time-Oriented raw data and abstracted concepts for multiple subject RecordS Graphical queries enable end users to define the constraints for selecting the relevant population to further explore Knowledge-based interpretation of the data Visual display and interactive exploration of multiple records Aggregation of multiple records and creation of associations amongst subject-related [temporal] patterns The VISITORS System(Klimov and Shahar, AMIA 2005; Klimov et al., AIM 2010)

38. Three types of queries: Select subjects (Who had this pattern?) Select Time Intervals (When did this pattern occur?) Get Data (What were the data for these subjects?) Selection constraints include: Demographical constraints (non-temporal): ID, age, smoking, sex, political group, … Time and value knowledge-based constraints: measured parameters, interventions, temporal-abstraction concepts Pair-wise constraints between concepts both absolute and relative (following a reference event) time lines Statistical constraints: filter the subjects’ data on the basis of a specific statistical function Subject and Time Interval Queries[Klimov et al., JIIS 2010]

39. VISITORS: Multiple-Patient, Multiple-Concept Intelligent Browsing and Exploration Patient groups Knowledge browser Multiple-patients raw data Distribution of derived patterns over time Concept search

40. Temporal Association Charts [Klimov et al., MIM 2009] • support and confidence of association rules indicated by width and hue • Data of each patient are connected by line

41. Meaningful associations typically exist among clinically meaningful, interval-based, abstract concepts (e.g., 3-5M Moderate Anemia precedes 2M Deteriorating Renal Function), rather than among time-stamped raw data such as Hemoglobin and Creatinine values TemporalAbstraction can be used to create time-oriented, interval-based, abstract concepts and patterns By using domain knowledge; e.g., the knowledge-based temporal abstraction method [Shahar, AIJ, 1997] By using automated temporal discretization methods [Verduijn et al., AIM, 2007; Moskovitch et al., IDAMAP, 2009] Interval-based abstract concepts can be then be mined to discover time-intervals related patterns (TIRPs) Example: The KarmaLego algorithm [Moskovitch and Shahar, IDAMAP, 2009; AMIA. 2009]; used in several domains:, such as Analysis of diabetes-patients data Prediction of getting off an ICU ventilator Classification of Hepatitis type from the course of the disease Using Temporal Abstractions for Temporal Data Mining

42. A Temporal Interval Related Pattern (TIRP) is a conjunction of temporal relations among symbolic time intervals {A1 o B, A1 o D, A1 m C1, A1 b C2, A1 b A2, B o D, B c C1, B b C2, B b A, C1 b C2, C1 b A, C2 o A} Temporal Interval Related Patterns-An Example

43. Exploration of the Diabetes TIRPs Tree: An Example[Moskovitch and Shahar, AMIA 2009] 0.26 0.18 0.22 0.28 0.25 0.23 0.42 0.29 0.33

44. Visualization of TIRPs: The KarmaLegoV Tool • Enables browsing of a KarmaLego TIRP enumeration tree; includes several options: • Presenting the next level, i.e., the next time-interval related to the current TIRP, and its temporal relation • Sorting by vertical support (% of patients who have the pattern), mean horizontal support (number of instances of TIRP per patient), and interestingness measures • Visualizing the current [mean] TIRP and its instances • Visualizing the distributions of external static (non-temporal) properties, such as age and gender, or a classification outcome (e.g., recovery or not), for the patient class in which the TIRP was found

45. The KarmaLegoV Tool: An Example (I)

46. The KarmaLegoV Tool: An Example (II)

47. Interactive query, visualization, and exploration requires runtime access to the domain’s temporal-abstraction ontology The visualization and exploration semantics can be specific to the temporal-abstraction task, but need not be specific to the domain Typical examples: Computation, visualization and exploration of multiple time-oriented records, their aggregations, and their inter- and intra- temporal relations the VISITORS system [Klimov and Shahar 2005] The KarmaLego framework [Moskovitch and Shahar 2009] Knowledge-Based Visualization & Exploration of Time-Oriented Data: Conclusions