Evaluating System Change Ventures

1. Evaluating System Change Ventures Innovations Toward a More Dynamic and Democratic Approach Bobby Milstein Measurement Knowledge Network WHO Commission on Social Determinants of Health March 22, 2005

2. Plan for Today • CDC Framework for Program Evaluation • Difference between research and evaluation • Link to institutional change • Steps and standards • Innovations for System Change Ventures • Syndemic orientation • Place of simulation modeling

3. SystematicMethods Evaluation Research Appreciating the Unique Character of Evaluative Inquiry “It is easier to find facts than it is to face them.” -- Anonymous Questions of Fact (descriptions, associations, effects) Questions of Values (merit, worth, significance) Centers for Disease Control and Prevention. What procedures are available for planning and evaluating initiatives to prevent syndemics? Syndemics Prevention Network, 2001. Available at <http://www.cdc.gov/syndemics/overview-planeval.htm>.

4. Institutional Climate for Evaluative Inquiry “The CDC’s senior leaders understood that strengthening evaluation capacity in public health would require a process of culture change, including significant reforms to their own organization.” Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002;2002(93):27-47.

5. Ways of Evaluating Informal Formal Low Stakes Involved High The Evaluation Continuum

6. Framework for Program Evaluation “Both a synthesis of existing evaluation practices and a standard for further improvement.” Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999;48(RR-11):1-40. Available at <http://www.cdc.gov/mmwr/PDF/RR/RR4811.pdf>.

7. Framework for Program Evaluation “A practical, nonprescriptive tool, designed to summarize and organize the essential elements of program evaluation.” Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999;48(RR-11):1-40. Available at <http://www.cdc.gov/mmwr/PDF/RR/RR4811.pdf>.

8. Standards for Effective Evaluation • Utility (7)Serve information needs of intended users • Feasibility (3)Be realistic, prudent, diplomatic, and frugal • Propriety (8)Behave legally, ethically, and with due regard for the welfare of those involved and those affected • Accuracy (12)Reveal and convey technically accurate information Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2nd edition ed Thousand Oaks, CA: Sage Publications, 1994.

9. Catalyst for Complementary Resources CDC Evaluation Working Group. Framework for program evaluation: adapted versions. Available at <http://www.cdc.gov/eval/framework.htm>.

10. Are We Posing Questions About Attribution or Contribution? “…if a program’s activities are aligned with those of other programs operating in the same setting, certain effects (e.g., the creation of new laws or policies) cannot be attributed solely to one program or another. In such situations, the goal for evaluation is to gather credible evidence that describes each program’s contribution in the combined change effort. Establishing accountability for program results is predicated on an ability to conduct evaluations that assess both of these kinds of effects.” p.11-12 Calls into question the conditions in which one focuses on a “program” as the unit of analysis

11. Acknowledging Plurality “You think you understand two because you understand one and one. But you must also understand ‘and’.” -- Sufi Saying • Efforts to Reduce Population Health ProblemsProblem, problem solver, response • Efforts to Organize a System that Assures Healthful Conditions for All Dynamic interaction among multiple problems, problem solvers, and responses Bammer G. Integration and implementation sciences: building a new specialisation. Cambridge, MA: The Hauser Center for Nonprofit Organizations, Harvard University 2003.

12. Living The Syndemics Prevention Network “You think you understand two because you understand one and one. But you must also understand ‘and’.” -- Sufi Saying • The word syndemic signals special concern for many kinds of relationships: • mutually reinforcing health problems • health status and living conditions • synergy/fragmentation in the health response system • Learning within innovative ventures • Comprehensive Community InitiativesPhilanthropy • Legacy InitiativesState Tobacco Settlements • Efforts to Eliminate Health Disparities Government and Philanthropy • Responses to Unjust Conditions Broad-based Citizen Organizations Health Power toAct Conditions A syndemic orientation clarifies the dynamic and democratic character of public health work Milstein B. Spotlight on syndemics. Centers for Disease Control and Prevention, 2001. <http://www.cdc.gov/syndemics>

13. PUBLIC HEALTH WORK SYSTEMS THINKING & MODELING (understanding change) SOCIAL NAVIGATION (governing movement) • What causes population health problems? • How are efforts to protect the public’s health organized? • How and when do health systems change (or resist change)? Directing Change • Who does the work? • By what means? • According to whose values? InnovativeHealth Ventures Charting Progress • How are conditions changing? • In which directions? PUBLIC HEALTH(setting direction) What are health leaderstrying to accomplish? Questioning the Character of Public Health Work

14. Syndemic Orientation Where are we going? What links to what? What influences what? Directed Public Work Connections Leverage Navigational View Network View Systems View Directional Data Proximity Data Causal Data Formalizing an OrientationJoining Concepts and Methods

15. 1840 1880 1950 1960 1980 2000 Changing (and Accumulating) Ideas in Causal TheoryWhat accounts for poor population health? • God’s will • Humors, miasma, atmosphere (“epidemic constitution”) • Poor living conditions, immorality (sanitation) • Single disease, single cause (germ theory) • Single disease, multiple causes (heart disease) • Single cause, multiple diseases (tobacco) • Multiple causes, multiple diseases (but no feedback dynamics) (social epidemiology) • Dynamic feedback among afflictions, living conditions, and public strength (syndemic)

16. The Feedback Thought “When X and Y affect each other, one cannot study the link between X and Y and, independently, the link between Y and X and predict how the system will behave. Only the study of the whole system as a feedback system will lead to correct results." -- System Dynamics Society System Dynamics Society. What is system dynamics? System Dynamics Society, 2002. Available at <http://www.systemdynamics.org/>. Richardson GP. Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press, 1991.

17. A Very Particular Distance

18. Looking Through the Macroscope “The macroscope filters details and amplifies that which links things together. It is not used to make things larger or smaller but to observe what is at once too great, too slow, and too complex for our eyes.” -- Joèel de Rosnay Rosnay J. The macroscope: a book on the systems approach. Principia Cybernetica, 1997. <http://pespmc1.vub.ac.be/MACRBOOK.html

19. Public Work Society's Health Response Tertiary General Targeted Primary Secondary Prevention Protection Protection Prevention Prevention Demand for response Becoming safer and healthier Safer Afflicted Afflicted with Vulnerable Healthier without Complications People People Developing Becoming Becoming Complications complications vulnerable afflicted Adverse Living Dying from Conditions complications Toward a Balanced System of Health Protection From: Milstein B, Homer J. The dynamics of upstream and downstream: why is so hard for the health system to work upstream, and what can be done about it?CDC Futures Health Systems Workgroup; Atlanta, GA; 2003.

20. Medical and Public Health Policy Healthy Public Policy & Public Work DEMOCRATIC SELF-GOVERNANCE MANAGEMENT OFRISKS & DISEASES • World of Transforming… • Deprivation • Dependency • Violence • Disconnection • Environmental decay • Stress • Insecurity • Etc… • By Strengthening… • Leaders and institutions • Foresight and precaution • The meaning of work • Mutual accountability • Plurality • Democracy • Freedom • Etc… • World of Providing… • Health education • Screening tests • Disease management • Pharmaceuticals • Clinical services • Physical and financial access • Etc… Balancing Two Major Areas of Emphasis Public Work Society's Health Response Tertiary General Targeted Primary Secondary Prevention Protection Protection Prevention Prevention Demand for response Becoming safer and healthier Safer Afflicted Afflicted with Vulnerable Healthier without Complications People People Developing Becoming Becoming Complications complications vulnerable afflicted Dying from complications Adverse Living Conditions

21. Citizen Involvement in Public Life Public Strength - Vulnerable and Afflicted People Fraction of Adversity, Social Division Vulnerability and Affliction Borne by Disadvantaged Sub-Groups (Inequity) Understanding Health as Public Work Public Work - Society's Health Response Tertiary General Targeted Primary Secondary Prevention Protection Protection Prevention Prevention Demand for response Becoming safer and healthier - Safer Afflicted Afflicted with Vulnerable Healthier without Complications People People Developing Becoming Becoming Complications complications vulnerable afflicted Dying from complications Adverse Living Conditions

22. Public Work Citizen Involvement - in Public Life Public Society's Health Strength Response - Tertiary General Targeted Primary Secondary Prevention Protection Protection Prevention Prevention Demand for response Becoming safer and healthier - Safer Afflicted Afflicted with Vulnerable Healthier without Complications People People Developing Becoming Becoming Complications complications vulnerable afflicted Dying from complications Adverse Living Conditions Vulnerable and Afflicted People Fraction of Adversity, Social Division Vulnerability and Affliction Borne by Disadvantaged Sub-Groups (Inequity) How Can Test a Dynamic Hypothesis? -- How can we learn about the consequences of actions in a system of this kind?-- Could the behavior of this system be analyzed using conventional epidemoiological methods (e.g., logistic or multi-level regression)?

23. Events Time Series Models Describe trends • Increasing: • Depth of causal theory • Degrees of uncertainty • Robustness for longer-term projection • Value for developing policy insights Multivariate Stat Models Identify historical trend drivers and correlates Patterns Dynamic Models Anticipate future trends, and find policies that maximize chances of a desirable path Structure Tools for Policy Analysis

24. Redirecting the Course of ChangeQuestions from System Dynamics and Social Navigation Where? 14% increase Why? How? Who? 2005 2025 2050 Zack MM, Moriarty DG, Stroup DF, Ford ES, Mokdad AH. Worsening trends in adult health-related quality of life and self-rated health–United States, 1993-2001. Public Health Reports 2004;119(September-October):493-505.

25. Outside assistance to alleviate and prevent affliction Outside assistance to build public strength B3a Effort to alleviate and R4a prevent affliction B1a Affliction Affliction Effort to build cross-impacts prevalence public strength & burden R1 B2 R2a Public work fraction Public R2c strength At-risk fraction Social disparity R3a Adverse Magnitude of R3b living ameliorative efforts conditions R2b United efforts Key Rectangle: Stock/state variable Blue arrow: same-direction link Green arrow: opposite-direction link Circled “B”: balancing causal loop Circled “R”: reinforcing causal loop Divided efforts B1b Effort to improve living conditions R4b B3b Outside assistance to improve living conditions Simulations for Learning in Dynamic SystemsThe Problem of Outside Assistance Dynamic Hypothesis (Structure) Behavior Over Time (Experiments) Homer J, Milstein B. Optimal decision making in a dynamic model of poor community health. Proceedings of the 37th Hawaii International Conference on System Science; Big Island, Hawaii; January 5-8, 2004. Available at <http://csdl.computer.org/comp/proceedings/hicss/2004/2056/03/205630085a.pdf>.

26. Access to Preventive Health Testing for Testing for Services PreDiabetes Diabetes PreDiabetes Diabetes Detection Detection Developing Diabetes from Undx Developing Behavior Over Time (Experiments) PreDiabetes PreD, Complications from People with People with Onset Dying from Undx Undx diab People with Undiagnosed, Undiagnosed, Complications Undiagnosed Uncomplicated Complicated PreDiabetes Diabetes Diabetes Recovering from People with PreDiabetes Normal Diagnosing Diagnosing Diagnosing Uncomplicated Glycemic Complicated PreDiabetes Diabetes Levels Diabetes Developing People with People with Dying from Complications People with Diagnosed, Complications Diagnosed, Recovering from Diagnosed Uncomplicated Complicated PreDiabetes Diabetes PreDiabetes Diabetes Onset Diabetes Risk for PreDiabetes & Diabetes Diabetes PreDiabetes Control Control Obese Fraction of Clinical Management Clinical the Population Management of of Diagnosed PreDiabetes Diabetes Medication Adoption of Ability to Self Physical Affordability Caloric Intake Healthy Lifestyle Monitor Activity Living Personal Conditions Capacity Simulations for Learning in Dynamic SystemsDiabetes Dynamics in an Era of Epidemic Obesity Dynamic Hypothesis (Structure) Deaths per Population 0.0035 0.003 Mixed Base 0.0025 Upstream 0.002 Downstream Striking an acceptable balance. 0.0015 1980 1990 2000 2010 2020 2030 2040 2050 Time (Year) Blue: Base run; Red: Clinical mgmt up from 66% to 90%; Green: Caloric intake down 4% (99 Kcal/day); Black: Clin mgmt up to 80% & Intake down 2.5% (62 Kcal/day) Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

27. Enact PolicyBuild power and organize actors to establish chosen policies Iterative Steps in System Dynamics Simulation Modeling Identify a Persistent ProblemGraph its behavior over time Convert the Map Into a Simulation ModelFormally quantify the hypothesis using allavailable evidence Learn About Policy Consequences Test proposed policies, searching for ones that best govern change Create a Dynamic Hypothesis Identify and map the main causal forces that create the problem Run Simulation ExperimentsCompare model’s behavior to expectations and/or data to build confidence in the model Milstein B, Homer J. Background on system dynamics simulation modeling, with a summary of major public health studies. Atlanta, GA: Syndemics Prevention Network, Centers for Disease Control and Prevention; February 1, 2005.

28. Learning In and About Dynamic Systems “The complexity of our models vastly exceeds our ability to understand their implications without simulation." -- John Sterman Benefits of Simulation/Game-based Learning • Formal means of evaluating options • Experimental control of conditions • Compressed time • Complete, undistorted results • Actions can be stopped or reversed • Visceral engagement and learning • Tests for extreme conditions • Early warning of unintended effects • Opportunity to assemble stronger support Complexity Hinders • Generation of evidence (by eroding the conditions for experimentation) • Learning from evidence (by demanding new heuristics for interpretation) • Acting upon evidence (by including the behaviors of other powerful actors) Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press). Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin McGraw-Hill, 2000.

29. “Simulation is a third way of doing science. Like deduction, it starts with a set of explicit assumptions. But unlike deduction, it does not prove theorems. Instead, a simulation generates data that can be analyzed inductively. Unlike typical induction, however, the simulated data comes from a rigorously specified set of rules rather than direct measurement of the real world. While induction can be used to find patterns in data, and deduction can be used to find consequences of assumptions, simulation modeling can be used as an aid to intuition.” A Third Branch of Science -- Robert Axelrod Axelrod R. Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P, editors. Simulating Social Phenomena. New York, NY: Springer; 1997. p. 21-40. <http://www.pscs.umich.edu/pub/papers/AdvancingArtofSim.pdf>.

30. Revisiting the Framework “Steps in the framework are starting points for tailoring an evaluation to a particular public health effort at a particular time.” Simulation Modeling Offers • Support for multi-stakeholder dialogue • A larger conception of the “program” context • Another avenue for experimentation and visceral learning • Ability to track interrelated indicators (both states and rates) • An emphasis on pragmatism (learning through action)

31. For Additional Information http://www.cdc.gov/syndemics

32. Enhancing Learning Through Simulation Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

33. Policy Resistance “The tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.” -- Meadows, Richardson, Bruckman “At least six times since the Depression, the United States has tried and failed to enact a national health insurance program.” -- Lee & Paxman Lee P, Paxman D. Reinventing public health. Annual Reviews of Public Health 1997;18:1-35. Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.

34. Flaws in Previous Attempts at Health System Reform in America (and Elsewhere) • Piecemeal approaches that do not address the full scope of the problem • Comprehensive strategies that are opposed by special interests • Assumption that healthcare dynamics are separate from other areas of public concern • Conventional analytic methods make it difficult to • Observe the health system as a large, dynamic enterprise • Craft high-leverage strategies that can overcome policy resistance • Been thinking of health and healthcare primarily as nouns (i.e., commodities distributed to consumers), not as verbs (i.e., public work to be done by citizens) Heirich M. Rethinking health care: innovation and change in America. Boulder CO: Westview Press, 1999. Kari NN, Boyte HC, Jennings B. Health as a civic question. American Civic Forum, 1994. Available at <http://www.cpn.org/topics/health/healthquestion.html>.

35. Event Oriented View Goals Problem Action Results Situation Feedback View Action Delay Delay “Side Effects” Goals Delay Delay Delay Delay Environment Delay Delay Goals of “Side Delay Delay Others Effects” Delay Action of Delay Others Basic Problem Solving Orientations Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

36. Building on Decades of SD Health Studies • Disease epidemiology • heart disease, diabetes, HIV/AIDS, cervical cancer, chlamydia, dengue fever, drug-resistant infections • Substance abuse epidemiology • heroin, cocaine, tobacco • Health care patient flows • emergency care, extended care • Health care capacity and delivery • HMO planning, dental care capacity, mental health care, disaster preparedness • Interactions between health capacity and disease epidemiology • chronic illness management, syndemics Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. American Journal of Public Health (in press).

37. Progress in Dynamic Modeling

38. …in an Era of Epidemic Obesity Transforming the Future of Diabetes… "Every new insight into Type 2 diabetes...makes clear that it can be avoided--and that the earlier you intervene the better. The real question is whether we as a society are up to the challenge... Comprehensive prevention programs aren't cheap, but the cost of doing nothing is far greater..." Gorman C. Why so many of us are getting diabetes: never have doctors known so much about how to prevent or control this disease, yet the epidemic keeps on raging. how you can protect yourself. Time 2003 December 8. Accessed at http://www.time.com/time/covers/1101031208/story.html.

39. Re-Directing the Course of ChangeQuestions from System Modeling and Social Navigation Where? How? Why? Who? 2010 2020

40. PreDiabetes Detection Diabetes Detection PreDiabetes Onset Recovering from PreDiabetes Diagnosing Diagnosing Diagnosing Diabetes Diabetes PreDiabetes Recovering from PreDiabetes Diabetes Dying from Developing Onset Complications Complications Obesity Prevention PreDiabetes Control Diabetes Control Diabetes System Modeling ProjectWhere is the Leverage for Health Protection? People with People with People with Undiagnosed, Undiagnosed, Undiagnosed Uncomplicated Complicated PreDiabetes Diabetes Diabetes People with Normal Glycemic Levels People with People with People with Diagnosed, Diagnosed, Diagnosed Uncomplicated Complicated PreDiabetes Diabetes Diabetes Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

41. Access to Preventive Health Testing for Testing for Services PreDiabetes Diabetes Clinical Clinical Management Management of of Diagnosed PreDiabetes Diabetes Medication Adoption of Ability to Self Physical Affordability Caloric Intake Healthy Lifestyle Monitor Activity Living Personal Conditions Capacity Diabetes System Modeling ProjectWhere is the Leverage for Health Protection? PreDiabetes Diabetes Detection Detection Developing Diabetes from Undx Developing PreDiabetes PreD, Complications from People with People with Onset Dying from Undx Undx diab People with Undiagnosed, Undiagnosed, Complications Undiagnosed Uncomplicated Complicated PreDiabetes Diabetes Diabetes Recovering from People with PreDiabetes Normal Diagnosing Diagnosing Diagnosing Uncomplicated Glycemic Complicated PreDiabetes Diabetes Levels Diabetes Developing People with People with Dying from Complications People with Diagnosed, Complications Diagnosed, Recovering from Diagnosed Uncomplicated Complicated PreDiabetes Diabetes PreDiabetes Diabetes Onset Diabetes Risk for PreDiabetes & Diabetes Diabetes PreDiabetes Control Control Obese Fraction of the Population

42. Deaths per Population 0.0035 0.003 Mixed Base 0.0025 Upstream 0.002 Downstream Striking an acceptable balance. 0.0015 1980 1990 2000 2010 2020 2030 2040 2050 Time (Year) Blue: Base run; Red: Clinical mgmt up from 66% to 90%; Green: Caloric intake down 4% (99 Kcal/day); Black: Clin mgmt up to 80% & Intake down 2.5% (62 Kcal/day)

43. Using Available Data to Calibrate the Model

44. Diabetes System Modeling ProjectConfirming the Model’s Fit to History Obese % of Adults Diagnosed Diabetes % of Adults Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).

45. Explaining the PastWhat Has Driven the Burden of Diabetes? Great Progress in Reducing the Burden for the Average Person with Diabetes Huge Growth in Number of People with Diabetes Overall, Total Diabetes Burden Held at Bay

46. Explaining the Past Deaths Due to Diabetes Have Fallen People with Diabetes per Thousand Adults Complications Deaths per Thous People w Diabetes 100 40 Model Output Model Output 90 30 80 20 Among people with diabetes, fewer dying every year 70 More people with diabetes 10 60 0 50 1980 1985 1990 1995 2000 2005 1980 1985 1990 1995 2000 2005 Time (Year) Time (Year) Deaths from Comps of Diabetes Per Thous Adults 2.5 Model Output 2 1.5 Combine to mean fewer U.S. adults dying 1980-2004 1 0.5 0 1980 1985 1990 1995 2000 2005 Time (Year)

47. After a delay Anticipating the FuturePrevalence Under ‘Status Quo’ Assumptions Obese Fraction of Adult Population People with Diabetes per Thousand Adults 0.6 130 Model Output Model Output 0.45 110 0.3 90 Prevalence continues to increase. Even if we assume the obesity epidemic has peaked… 0.15 70 0 50 1980 1990 2000 2010 2020 2030 2040 2050 1980 1990 2000 2010 2020 2030 2040 2050 Time (Year) Time (Year)

48. Anticipating the Future Deaths Under ‘Status Quo’ Assumptions Complications Deaths per Thous w Diabetes People with Diabetes per Thousand Adults 40 130 And if we can maintain current levels of care 30 110 20 90 If prevalence continues to increase, but no continued improvement… 10 70 0 50 1980 1990 2000 2010 2020 2030 2040 2050 1980 1990 2000 2010 2020 2030 2040 2050 Time (Year) Time (Year) Deaths from Complications of Diabetes Per Thousand Adults 2.5 Then prevalence overwhelms the improved care to boost the burden 1.25 1980 1990 2000 2010 2020 2030 2040 2050 Time (Year)

49. Policy Experiments • Continued downstream improvements • Upstream efforts • Downstream & Upstream

50. Downstream Downstream-Only Intervention Deaths per Population 0.0035 0.003 Base 0.0025 0.002 Disease control acts quickly but does not slow the growth in deaths. 0.0015 1980 1990 2000 2010 2020 2030 2040 2050 Time (Year) Blue: Base run; Red: Clinical mgmt of diagnosed up from 66% to 90%