Human Systems Dynamics Theory Applied to Evaluation Practice

1. Human Systems Dynamics Theory Applied to Evaluation Practice American Evaluation Association2008 Beverly Parsons, Ph.D.InSites bparsons@insites.org Meg Hargreaves, Ph.D.Mathematica Policy Research, Inc. mhargreaves@mathematica-mpr.com

5. Introduction to a Systems Perspective In Evaluation • This section presents: • System definitions • System features • System characteristics • Types of systems • Examples of types

6. Systems Definitions • Multiple definitions: • A group of interacting, interrelated, or interdependent parts forming a complex whole • A configuration of parts joined together by a web of relationships • The parts form a whole, which is greater than the sum of its parts

7. System Features Systems are as much an “idea” about the real world as a physical description of it: • Boundaries define who or what lies inside or outside the system • Differences among the parts influence the system’s dynamics • Relationships among parts, between parts and whole, and between whole and its environment, are key focus of systems

8. System Characteristics • Common patterns, behaviors, and properties: • Patterns – unorganized, organized, or organic (self-organized) • Behaviors – random, simple, complicated, or complex adaptive; linear or nonlinear • Properties – independent, interrelated, or interdependent relationships • Scale – small to large, self-similarity across levels (fractals)

9. System Types • Systems can be grouped by their level of complexity or organization: • Random (no system) - unorganized • Simple system - organized • Complicated system – organized • Complex adaptive system – organic

10. Random (Unorganized) • Random, chaotic activity – no pattern • Independent, unconnected parts • No cause-effect relationships – constant chaos and surprise • Turbulence - no equilibrium • Random parts without a system • No leadership - people react blindly • Unknowable

11. Random System Examples • War zone: Civilians caught in crossfire, random flight to escape conflict • Natural disaster: At landfall or in the eye of the storm, residents react instinctively to events • Leadership transitions: During changes in administration old patterns are suspended before new patterns are established

12. Simple System (Organized) • Stable, static pattern • Parts connected in linear ways • Predictable cause-effect relationships • Set equilibrium • System reducible to parts and replicated • Directive leadership - designed change • Known knowns – answers are evident

13. Simple System Examples • Baking a cake: Follow a recipe to assemble and combine ingredients into a batter that is baked at a pre-set temperature with predictable results • Flu shot clinics: Nurses use consistent procedures to administer the same shots to each person, following a set protocol in assembly-line fashion

14. Complicated (Organized) • Dynamic pattern of feedback loops • Many interrelated parts across subsystems, levels • Complex, nonlinear cause-effect relationships • Feedback can stabilize equilibrium – thermostat • System can be reduced to parts and replicated • Multiple answers – investigate options • Unknowns become known through expert analysis at multiple levels

15. Complicated System Examples • Space Shuttle Challenger disintegrated (1986) when O-ring failure caused a rocket booster breach, creating flare that damaged external fuel tank, spilling fuel that exploded • In large healthcare institutions, human behaviors are part of wider systems of causality, in which medical errors can occur in organizational and policy contexts that result in long (36-hour) shifts, large caseloads, and strained staff relations

16. Complex Adaptive System (Organic) • Dynamical patterns – parts adapting to each other and to environment as a whole • Parts are massively entangled, interdependent • Parts self-organize, learn, coevolve organically • Equilibrium in flux - sensitive to initial conditions • System not replicable, can’t repeat past • Emergent change – manage conditions of organic development and experimentation • Unknown unknowns – trial and error

17. Complex Adaptive System Examples • Economic system – interactions of homeowners, mortgage lenders, stock market traders, investors, federal banking institutions, and worried consumers are coevolving into global crisis and recession, despite governments’ interventions • User networks (Diabetes, AA) facilitate exchange of information and advice on care for chronic conditions among participants, learning from each other

18. Background about Systems Theories • This section presents: • General systems theory • Cybernetics – systems dynamics • Complex adaptive systems • Implications for evaluation

19. General Systems Theory • Holistic change ideas – ancient Greeks • General systems theory - von Bertalanffy (1930’s); earliest work by Bogdanov (1910) • Open systems – nonrandom elements organized into interacting, interrelated components that seek to survive through interactions with environment • Each system level nested in higher level (cells, organisms, families, organizations, communities, societies)

20. Implications for Evaluation • The whole can enable/constrain parts and the parts can contribute to and/or challenge stability of the whole • Because open systems are structured in hierarchies; useful to look one level above and one level below the ‘system in focus’ • Evaluate system viability – does system have both the parts and the information and decision flows among the parts that are needed to survive?

21. Cybernetics and System Dynamics • System dynamics founded by Forrester at MIT (1950’s) for electrical engineering • Method for calculating and modeling how many circular, interlocking, sometimes time-delayed relationships among parts are important in shaping system-wide behavior • Through negative feedback, adjustments made to keep system in balance; positive feedback used to move system in same direction, moving out of balance

22. Implications for Evaluation • Assess influence of feedback loops on behavior of system’s parts and on whole • Behavior of whole not only explained by behavior of parts (e.g. medical errors) • Feedback loops undermine sustainability of public interventions (policy resistance) • Evaluators cannot step outside social and ecological systems to observe (not value-neutral); self-reflection needed

23. Complex Adaptive Systems • Key CAS writers – Weaver (1948), Simon (1962), Prigogine (1987), Stacey (1997, 2007), Zimmerman et al (2001), Eoyang (2006) • CAS – many semi-independent and diverse agents, who are free to act in unpredictable ways, continually interact with each other, adapting to each other and to environment as a whole, creating system-wide patterns • Key concepts – emergence, organic self-organization, co-evolution, simple rules

24. Implications for Evaluation • Currently relevant evaluation criteria and measures may need to be updated as new conditions emerge • Measure frequently for emerging patterns • Avoid grand modeling projects for prediction; use smaller projects for ongoing experimentation and learning • Also visualize system interactions as networks; look outside nested levels for system patterns, drivers, and constraints • Ask what, so what, now what?

25. N T E O X T C Three Dynamics of a Social System and its Context Unorganized     dynamics (random unpatterned seemingly chaotic) far from agreement organic dynamics (emerging patterns     coherent but not predictable) Agreement • Organized  dynamics • (predictable •    orderly • controlled) close to agreement far from certainty close to certainty Certainty

26. Match of Evaluation Designs to Dynamics of Social Systems and Their Context N T E O X T C Exploratory Design far from agreement unorganized dynamic Organic Design Agreement Initiative Renewal Design organic dynamic Predictive Design close to agreement organized dynamic far from certainty close to certainty Certainty

27. Understanding Organic Dynamics (Activity) • Divide into triads • Selects one other triad member (doesn’t tell) and uninvolved person in refreshment area • Stay at least two feet apart and equidistant from the other two • Do this for about 1-2 minutes while trying to reach refreshments • Reflect on experience

28. Case Study Introduction Do the preconference professional development offerings contribute to effective evaluation-related work of association members? If so, how?

29. Unorganized System Dynamics What is happening? What boundaries, differences, similarities, and relationships might shape how the offerings contribute to participants’ evaluation-related work?

30. Organized System Dynamics Do participants receive high-quality content that is relevant to their evaluation-related work and is delivered through high-quality instructional methods?

31. Organized System Dynamics How do the format and content of the session support or hinder participants in understanding and using the session to apply the learning from the session to their evaluation work?

32. Organic System Dynamics What patterns among participants (including the session facilitators) before and during the session are likely to affect the participants’ understanding and application of the learning to their evaluation-related work?

33. Patterns

34. Patterns

35. Patterns

36. Patterns

37. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 10)

38. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 27)

39. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p.42)

40. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 78)

41. Patterns

42. Fractals:Patterns, Patterns Everywhere In nature . . . Mathematical construct of iterating nonlinear equation and plotting on complex number plane—Mandelbrot Set Similar shapes at all scales—Broccoli Biological scaling gives coherence in widely diverse entities—Oak tree Scale-free networks

43. Recognizing patterns is critical:similarities, differences, and relationships that have meaning across space and time Basic values or simple rules generate diverse, but self-similar behavior across scales Naming and telling stories about dynamics in a system help reinforce and shape fractal patterns Fractals:Patterns, Patterns Everywhere

44. Fractals

45. Looking at the Dynamics as a Whole • What is the overall picture of system dynamics affecting how the preconference professional development offerings contribute to effective evaluation-related activities of AEA members? • Given the findings from the three system dynamics within the preconference session, how might the preconference professional development process and offerings be modified to contribute more substantially to the quality of AEA members’ evaluation-related work?