Measuring eHealth Impact

1. Measuring eHealth Impact An eHealth evaluation framework that leverages process control theory and big data analytics. Health 2013 Ottawa, Canada

3. The next 20 minutes… What is the problem being addressed? Why is this problem…ummm… problematic? What is the new approach? How is this different? Who cares… and why?

4. What is the health impact of implementing eHealth?

5. Why can’t we answer this question? There is a disheartening dearth of good science regarding the monitoring and evaluation (M&E) of eHealth projects But to be fair… It is a hard question!

6. What makes it a hard question? Healthcare is typically characterised as a complex adaptive system It is difficult to measure “health” The connection between eHealth and the health production function is not well understood

7. Key takeaways… Today, we do not have good science that quantifies the health impacts of eHealth This is an important question We need to answer it in order to be able to justify eHealth expenditures in the face of other uses of scarce funds

8. So… what should we do?

9. Embrace a “WHOLE SYSTEM” approach.

10. How do we characterise the system? How do we measure health? How are eHealth and health related?

11. Seriously… how?!

12. Don’t worry ma’am… I’m an engineer.

13. Explaining system process control theory in 90 exhausting seconds

14. System

15. System

16. Measured inputs System Measured outputs

17. unMeasured inputs System unMeasured outputs

18. Controlled inputs System unControlled inputs

19. System FeedForward Process Control

20. System Inferential FeedForward Process Control

21. Feedback Process Control System

22. Inferential Feedback Process Control System

23. Hand in the tub Hand in the stream Watching the steam Feedback Feedforward Inferential

24. Examples… If a doctor orders meds, tracks the patient’s “readings” (blood pressure, fever, HbA1C), and adjusts the dosage accordingly – that is feedback control If Health Canada tracks the flu outbreaks in the southern hemisphere to inform next year’s flu shot cocktail, that is feedforward control

25. Key takeaways… We can shamelessly steal mature techniques from other disciplines to help us model the healthcare system Feedback and feedforward control may be exerted on any system where there are controlled inputs

26. How do we measure health?

27. More Aggregated HRQoL Indexes Generic Health Status Profiles Disease-specific Scales Multiple other indicators Less Aggregated

28. More Aggregated HRQoL Indexes Generic Health Status Profiles Disease-specific Scales Multiple other indicators Less Aggregated

29. 0.0 1.0 DALY 0.75 QALY Disease weight Health related quality of life 0.0 1.0 AGE Premature death 45 years LE 60 years Onset of disability 10 years After Robberstad, Norskepidemiologi, 2005

30. ∑QALY n = HALE n

31. What do we like about HALE? • A health adjusted life expectancy is usefully sensitive to morbidity as well as mortality • If we increase life expectancy, HALE goes up • If we reduce morbidity, HALE goes up • There are numerous HALE indicators • As it turns out, some may be calculated from standards-based eHealth transactional data

32. Key takeaways… Health adjusted life expectancy is a useful way to characterise population health We can calculate a HALE metric from eHealth transactional data

33. After Patton, Developmental Evaluation, 2011

34. How do we tie it all together? We develop a model that leverages the underlying tools and techniques We are working with complex adaptive systems… this affects our choice of an approach We must favour sense & adaptover plan & control

35. Inform Operationalizes Population-level health metrics Health Interventions eHealth Infrastructure Population Health Person-centric transactional data Generate Yield

36. Health Interventions Population Health Yield

37. Operationalizes Health Interventions eHealth Infrastructure Population Health Yield

38. Operationalizes Health Interventions eHealth Infrastructure Population Health Person-centric transactional data Generate Yield

39. Operationalizes Population-level health metrics Health Interventions eHealth Infrastructure Population Health Person-centric transactional data Generate Yield

40. Inform Operationalizes Population-level health metrics Health Interventions eHealth Infrastructure Population Health Person-centric transactional data Generate Yield

41. Key takeaways… We can employ process control loops and big data analytics to create a cyclic model This model may be employed to operationalize continuous quality improvement

42. How does this approach compare? Inputs Processes Outputs Outcomes Impacts Traditional logic framework approaches are “pipeline” based Common M&E approaches focus on outputs… or sometimes on outcomes… but not on impacts

43. Who cares… and why?

44. The Role of eHealth • eHealth infrastructure, at scale: • Supports care continuity over time and across different sites • Operationalizes guideline-based care • Health “transactions”: • Provide management metrics regarding care delivery • May be aggregated to generate population indicators

45. Strategic Elements

46. Standard Operating Procedures SOP-based Interventions Pervasive eHealth Infrastructure Modify SOP eHealth Transactional Data Financial, Management & Population Health Indicators

47. Operationalize guideline-based care Standard Operating Procedures SOP-based Interventions Pervasive eHealth Infrastructure Modify SOP eHealth Transactional Data Financial, Management & Population Health Indicators

48. Standard Operating Procedures SOP-based Interventions Pervasive eHealth Infrastructure Modify SOP eHealth Transactional Data Support care continuity Financial, Management & Population Health Indicators

49. Standard Operating Procedures SOP-based Interventions Pervasive eHealth Infrastructure Modify SOP eHealth Transactional Data Financial, Management & Population Health Indicators Provide management metrics

50. Standard Operating Procedures SOP-based Interventions Pervasive eHealth Infrastructure Modify SOP eHealth Transactional Data Financial, Management & Population Health Indicators Provide population indicators