Rigour vs Relevance

1. Rigour vs Relevance Systems Simulation for Learning Clinical Expertise Geoff McDonnell CHI Seminar 3 April 2008

2. Jack Homer Friday 11th April • 12.30 -2.30 Lecture on Chronic Disease Modeling six years at CDC then Workshop till 4.30pm • Epidemiology meets Systems modeling • Forrester Award • Co-founder HPSIG • Math Stats Stanford • PhD MIT Sloan School • Leading SD consultant esp health • UNSW RANDWICK CAMPUS R31 NDARC

3. Working on the cure for tunnel vision & tunnel imagination • How do we improve health decisions in a connected world? • Patient, clinician, manager, system designer • Complex systems science toolkit, esp. • System Dynamics • Network Analysis • Agent Based Modeling Both Computable Narrative Synthesis and Computational Analysis

4. Context Mechanism Outcome Complexity & Systems Science Systems Modelling and Simulation Donald M. Berwick's National Forum keynote speech, Eating Soup with a Fork Nov2007 http://www.ihi.org/IHI/Programs/AudioAndWebPrograms/OnDemandPresentationBerwick.htm The Science of Improvement Don Berwick JAMA http://jama.ama-assn.org/cgi/content/short/299/10/1182?rss=1

6. H.L. Mencken quip, “For every complex problem, there is a solution that is simple, neat and wrong.”

7. Modelling & Simulation

8. Add an “in silico” simpler Health Sim World to Experiment with Complex Health Problems Sim World

9. Our Health Sim Projects Social Context InterGenerational Report & Politics National WHO & State Health Systems, eHealth & Policy Chronic Disease & Disability Dementia National Medicines Use Acute Aged Care Interface Pharmacy & Medical Workforce Hospital Flow Health IT Methadone Ward Medication Error ED Improvement Communication Acts

10. Multiscale Simulation & Modeling Physiome Project IEEE Computer 2005

11. Dynamic Social Network Analysis The modeling technique depends on the interaction between Problem and Solution Adapted from Andrei Borshchev, www.xjtek.com

12. Global Global Health Climate Change Geopolitics Continental Health Politics & Systems Culture & Society National Health Systems Social Determinants political economic cultural State Health Economics Scope (Spatial Scale) Health Services Region Environmental Health Community Public Health Epidemiology Household Clinical Medicine Individual Organs Cells Sub-cellular Genome Molecule Systems Biology/Omics Genetics Centuries Decades Years Weeks Days Hours Minutes Seconds Level of Detail (Time scale)

13. Systems modelling • Software now available to assist us with understanding and predicting change in a system – both qualitatively (behaviour) and quantitatively (status at a point in time) • We can safely ‘experiment’ by both ‘zooming in’ and ‘zooming out’.

14. Systems thinking & modelling for learning clinical decisions?

15. The System Dynamics Method • Systems Thinking is a “way of seeing” the world, the problems around us, • As “Systems”, indivisible collection of interconnected parts. • Dynamic Problems are where things change over time. • A (systemic) dynamic problem is a chronic problem that needs continuous monitoring and action (“management”).

16. Patterns of Behavior System Structure A Systems Perspective Reactive Events and Decisions Adaptive Increasing leverage Generative

17. B D A C E The systems thinking/ system dynamics method Issue Behaviour over time graphs What-if experiments now Time Structure v Behaviour Data Mental Models validation Objectives Visions Strategies Causal maps Stock-flow maps and simulation models

18. SCIENTIFIC AMERICAN February 1996 p43

19. Learning in and about Complex Systems Sterman (1994) Unknown structure Dynamic complexity Time Delays Impossible experiments Real World Virtual World Known structure Variable Complexity Controlled Experiments Selected Missing Delayed Biased Ambiguous Implementation Game playing Inconsistency Short term Information Feedback Decisions Misperceptions Unscientific Biases Defensiveness Strategy, Structure, Decision Rules Mental Models Inability to infer dynamics from mental models

20. The System Dynamics Iterative Modeling Process

21. Watching vs. shaping the future “We can forecast in the region where action is not effective. We can have influence in the region where forecasting is unreliable” -Jay Forrester

22. Clinical Reasoning • Classical Decision theory • Decision analysis • Bias adjustment • Naturalistic decision making • Frames and Data • Cognitive Schema • Image Theory • Cognitive/Neuroscience

23. Endsley MR. Toward a theory of situation awareness in dynamic systems. Hum Factors 1995;37:32–64.

24. Evidence based medicine integrates the best external evidence with individual clinical expertise and patients' choice. The best external evidence Scientific inquiry Operational Research Action Research Engaged scholarship Rigorous learning in Complex systems Clinical Expertise Patients’ choice BMJ 1996;312:71-72 (13 January) Editorial Evidence based medicine: what it is and what it isn't David L Sackett et al

25. The wisdom of practice is achieved only through the examined experience Adapted From Learning Clinical Decision Making Course Notes p166 Ken Cox 1994

27. The Experiment • One of the 3 great ideas • Peter Watson A History of Ideas from Fire to Freud • RCT • N of 1 • Virtual (SB and Physiome, Policy, Clinical?) • Rigour and relevance

28. Universal features of dynamics emerge across wide variation in context and scale: (multi-scale) John Sterman NIH Videocast, 2007

29. Realistic Evaluation Pawson and Tilley via Don Berwick

30. A program is theory incarnate

31. Realistic Evaluation: An Overview Nick Tilley, Nottingham Trent University Presented at the Founding Conference of the Danish Evaluation Society, September 2000

32. Fig4.2 p113 Andrew H Van de Ven Engaged Scholarship OUP 2007 ISBN 9780199226306 http://umn.edu/~avandeve

33. Engaged scholarship diamond model Study Context: Research purpose, perspective & complexity Develop variance/process model to study theory. Engage methods experts & people providing access & information. Criterion: Truth (Verisimilitude) Create, elaborate & justify a theory by abduction, deduction & induction. Engage knowledge experts in relevant disciplines & functions. Criterion: Validity Model Research Design Theory Building Iterate and Fit Solution Theory Problem Solving Problem Formulation Communicate, interpret and negotiate findings with intended audience. Engage intended audience to interpret meanings & uses. Criterion: Impact Situate, ground, diagnose & infer the problem up close and from afar. Engage those who experience & know the problem. Criterion: Relevance Reality Fig1.1 p10 Andrew H Van de Ven Engaged Scholarship OUP 2007 ISBN 9780199226306 http://umn.edu/~avandeve

34. Model Solution Theory Reality Engaged scholarship diamond model Study Context: Research purpose, perspective & complexity Develop variance/process model to study theory. Engage methods experts & people providing access & information. Criterion: Truth (Verisimilitude) Create, elaborate & justify a theory by abduction, deduction & induction. Engage knowledge experts in relevant disciplines & functions. Criterion: Validity Theory Building Iterate and Fit Research Design Problem Formulation Communicate, interpret and negotiate findings with intended audience. Engage intended audience to interpret meanings & uses. Criterion: Impact Situate, ground, diagnose & infer the problem up close and from afar. Engage those who experience & know the problem. Criterion: Relevance Problem Solving Fig1.1 p10 Andrew H Van de Ven Engaged Scholarship OUP 2007 ISBN 9780199226306 http://umn.edu/~avandeve

35. High Feasibility for Policy Formation Low Computer Modeling Data-based Expert Judgments Unrestricted Judgments Intervening Analysis (overt) True Experiments (physical sci) Modes of Inquiry Control-group Experiments & statistics Quasi- Experiments Relaxed ctrls Mode of Cognition Degree of Control Possible Intuition (covert) Representing High Interpersonal Conflict Potential Low Human Judgment and Social Policy Kenneth R Hammond OUP 1996 Fig 9.1 p235

36. Form of inference articulated by C.S.Peirce (1839-1914) “Abduction consists in studying the facts and devising a theory to explain them” What is abduction?

37. Peirce’s System of Inquiry: Integrates abduction, deduction, & induction Observe outcomes (Induction) Monitor implementation Evaluate Take action on the basis of the “hypothesis to the best explanation” Observe events ActionResearch Positivist Science Form hypotheses (Abduction) Develop resultant hypotheses (Deduction) & test Select & test hypotheses (Retroduction- experimentation/ triangulation)

38. The Dynamics of Peirce’s Inquiry System(C/w Dewey’s Instrumental Methodology) Cycle 3 Cycle 2 Cycle 1 In practice, this process appeals to both scientific research and “common sense” drawn from experience

39. Making Sense of Sensemaking 2: A Macrocognitive Model Gary Klein and Brian Moon, Robert R. Hoffman IEEE Intelligent Systems Sep-Oct 2006 p88-92 (www.computer.org/intelligent)

40. Replanning during Execution Gary Klein Flexecution, Part 2: Understanding and Supporting Flexible Execution IEEE Intelligent Systems (2007) 22;6 p108-12 (www.computer.org/intelligent)

41. Comparing AR with Positivist Science

42. Note: • Positivist science takes place within a “closed” system frame • Action research takes place within an “open” system frame • Action research & positivist science complement each other; positivist science is particularly associated with the retroductive process!

43. The scientific method as dialectic between analysis and syntheses [Barton and Haslett (2007: 148)] As a result of the dialectic process, new “categories” emerge with increased powers of explanation and understanding Time Extreme Reductionism New hypothesis/ category New data Emphasis on Parts New hypothesis/ category New data Action &/or Analysis using Deduction generating new data; confirmation with Induction Observable data (“Surprising facts”) Initial (systemic) hypothesis Synthesis using Abductive Reasoning Generating new hypotheses Extreme Holism The most general approach to abduction in systems thinking is what Churchman describes as Singer’s process of ‘sweeping in’ [Barton and Haslett (2007: 151)] Emphasis on Wholes

44. The Relationship Between Open and Closed Systems

45. Formation of abstract concepts and generalisations Select & Test Hypotheses RETRODUCTION Experimentation /triangulation Form Hypotheses ABDUCTION MACRO COGNITION Strategy,Structure Decision Rules Mental Models THEORY Sense-making MODEL Reflecting Develop Resultant Hypotheses DEDUCTION & test Working Knowledge EXPLANATION CYCLE Reflections Evaluate Decisions Information Feedback FLEXECUTION Surprise Doing/Thinking Mix Testing implications of concepts in new situations Observations Virtual World Briefing Debriefing Observe Events/Outcomes EXPERIENCE CYCLE SOLUTION Clinical Encounter REALITY Monitor Implementation Real World Take action on the basis of the “hypothesis to the best explanation” Concrete Experience

46. NIH Systems Approaches to Public Health Videocasts • Intro (esp John Sterman Track 3) • http://videocast.nih.gov/Summary.asp?file=13712 • Network Analysis • http://videocast.nih.gov/Summary.asp?file=13878 • Agent Based Modeling (bottom up) • Joshua Epstein and Michael Macy http://videocast.nih.gov/Summary.asp?file=13931 • System Dynamics (Jack Homer August 30) • NCI Tobacco Control Monograph 18: Greater Than the Sum: Systems Thinking in Tobacco Control http://cancercontrol.cancer.gov/tcrb/monographs/18/monograph18.html

47. Complex Systems Approaches to Population Health from Michigan videocasts Videocast Day 1 http://videocast.nih.gov/Summary.asp?file=13867 Videocast Day 2 http://videocast.nih.gov/Summary.asp?file=13869 http://www.hpsig.com/index.php?title=MEDINFO2007_eHealth_Simulation_Tutorial

48. Systems Biology & Systems Medicine • Genomic information & dynamic network interaction with its environment • Dynamic, Hierarchical, Multiscale, Integrate levels • Quantitative as possible Global measurements (genome OK protein not yet) • Need for Theory Reality balance • Data space is infinite, so we need to focus on relevant dataspace via a hypothesis (global middle or local) • Crossdisciplinary: learn language/teams physically located • Disease perturbed information networks • Leroy Hood Videocast: http://videocast.nih.gov/Summary.asp?file=14388

49. Network Medicine — From Obesity to the “Diseasome” Albert-László Barabási, Ph.D.

50. Social Structure [Institutions & Networks] Meaning-laden rules & resources Patterns of conduct Know/Learn Cognitive Schema Neuromatrix Sense of Control Produce & Reproduce Enable & Constrain Psyche Expression Values & Roles Embodiment Social actions Perceive Act Bio