1 / 45

Human Systems Dynamics Theory Applied to Evaluation Practice

Human Systems Dynamics Theory Applied to Evaluation Practice. American Evaluation Association 2008. Beverly Parsons, Ph.D. InSites bparsons@insites.org . Meg Hargreaves, Ph.D. Mathematica Policy Research, Inc. mhargreaves@mathematica-mpr.com.

deva
Download Presentation

Human Systems Dynamics Theory Applied to Evaluation Practice

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Human Systems Dynamics Theory Applied to Evaluation Practice American Evaluation Association2008 Beverly Parsons, Ph.D.InSites bparsons@insites.org Meg Hargreaves, Ph.D.Mathematica Policy Research, Inc. mhargreaves@mathematica-mpr.com

  2. Introduction to a Systems Perspective In Evaluation • This section presents: • System definitions • System features • System characteristics • Types of systems • Examples of types

  3. Systems Definitions • Multiple definitions: • A group of interacting, interrelated, or interdependent parts forming a complex whole • A configuration of parts joined together by a web of relationships • The parts form a whole, which is greater than the sum of its parts

  4. System Features Systems are as much an “idea” about the real world as a physical description of it: • Boundaries define who or what lies inside or outside the system • Differences among the parts influence the system’s dynamics • Relationships among parts, between parts and whole, and between whole and its environment, are key focus of systems

  5. System Characteristics • Common patterns, behaviors, and properties: • Patterns – unorganized, organized, or organic (self-organized) • Behaviors – random, simple, complicated, or complex adaptive; linear or nonlinear • Properties – independent, interrelated, or interdependent relationships • Scale – small to large, self-similarity across levels (fractals)

  6. System Types • Systems can be grouped by their level of complexity or organization: • Random (no system) - unorganized • Simple system - organized • Complicated system – organized • Complex adaptive system – organic

  7. Random (Unorganized) • Random, chaotic activity – no pattern • Independent, unconnected parts • No cause-effect relationships – constant chaos and surprise • Turbulence - no equilibrium • Random parts without a system • No leadership - people react blindly • Unknowable

  8. Random System Examples • War zone: Civilians caught in crossfire, random flight to escape conflict • Natural disaster: At landfall or in the eye of the storm, residents react instinctively to events • Leadership transitions: During changes in administration old patterns are suspended before new patterns are established

  9. Simple System (Organized) • Stable, static pattern • Parts connected in linear ways • Predictable cause-effect relationships • Set equilibrium • System reducible to parts and replicated • Directive leadership - designed change • Known knowns – answers are evident

  10. Simple System Examples • Baking a cake: Follow a recipe to assemble and combine ingredients into a batter that is baked at a pre-set temperature with predictable results • Flu shot clinics: Nurses use consistent procedures to administer the same shots to each person, following a set protocol in assembly-line fashion

  11. Complicated (Organized) • Dynamic pattern of feedback loops • Many interrelated parts across subsystems, levels • Complex, nonlinear cause-effect relationships • Feedback can stabilize equilibrium – thermostat • System can be reduced to parts and replicated • Multiple answers – investigate options • Unknowns become known through expert analysis at multiple levels

  12. Complicated System Examples • Space Shuttle Challenger disintegrated (1986) when O-ring failure caused a rocket booster breach, creating flare that damaged external fuel tank, spilling fuel that exploded • In large healthcare institutions, human behaviors are part of wider systems of causality, in which medical errors can occur in organizational and policy contexts that result in long (36-hour) shifts, large caseloads, and strained staff relations

  13. Complex Adaptive System (Organic) • Dynamical patterns – parts adapting to each other and to environment as a whole • Parts are massively entangled, interdependent • Parts self-organize, learn, coevolve organically • Equilibrium in flux - sensitive to initial conditions • System not replicable, can’t repeat past • Emergent change – manage conditions of organic development and experimentation • Unknown unknowns – trial and error

  14. Complex Adaptive System Examples • Economic system – interactions of homeowners, mortgage lenders, stock market traders, investors, federal banking institutions, and worried consumers are coevolving into global crisis and recession, despite governments’ interventions • User networks (Diabetes, AA) facilitate exchange of information and advice on care for chronic conditions among participants, learning from each other

  15. Background about Systems Theories • This section presents: • General systems theory • Cybernetics – systems dynamics • Complex adaptive systems • Implications for evaluation

  16. General Systems Theory • Holistic change ideas – ancient Greeks • General systems theory - von Bertalanffy (1930’s); earliest work by Bogdanov (1910) • Open systems – nonrandom elements organized into interacting, interrelated components that seek to survive through interactions with environment • Each system level nested in higher level (cells, organisms, families, organizations, communities, societies)

  17. Implications for Evaluation • The whole can enable/constrain parts and the parts can contribute to and/or challenge stability of the whole • Because open systems are structured in hierarchies; useful to look one level above and one level below the ‘system in focus’ • Evaluate system viability – does system have both the parts and the information and decision flows among the parts that are needed to survive?

  18. Cybernetics and System Dynamics • System dynamics founded by Forrester at MIT (1950’s) for electrical engineering • Method for calculating and modeling how many circular, interlocking, sometimes time-delayed relationships among parts are important in shaping system-wide behavior • Through negative feedback, adjustments made to keep system in balance; positive feedback used to move system in same direction, moving out of balance

  19. Implications for Evaluation • Assess influence of feedback loops on behavior of system’s parts and on whole • Behavior of whole not only explained by behavior of parts (e.g. medical errors) • Feedback loops undermine sustainability of public interventions (policy resistance) • Evaluators cannot step outside social and ecological systems to observe (not value-neutral); self-reflection needed

  20. Complex Adaptive Systems • Key CAS writers – Weaver (1948), Simon (1962), Prigogine (1987), Stacey (1997, 2007), Zimmerman et al (2001), Eoyang (2006) • CAS – many semi-independent and diverse agents, who are free to act in unpredictable ways, continually interact with each other, adapting to each other and to environment as a whole, creating system-wide patterns • Key concepts – emergence, organic self-organization, co-evolution, simple rules

  21. Implications for Evaluation • Currently relevant evaluation criteria and measures may need to be updated as new conditions emerge • Measure frequently for emerging patterns • Avoid grand modeling projects for prediction; use smaller projects for ongoing experimentation and learning • Also visualize system interactions as networks; look outside nested levels for system patterns, drivers, and constraints • Ask what, so what, now what?

  22. N T E O X T C Three Dynamics of a Social System and its Context Unorganized     dynamics (random unpatterned seemingly chaotic) far from agreement organic dynamics (emerging patterns     coherent but not predictable) Agreement • Organized  dynamics • (predictable •    orderly • controlled) close to agreement far from certainty close to certainty Certainty

  23. Match of Evaluation Designs to Dynamics of Social Systems and Their Context N T E O X T C Exploratory Design far from agreement unorganized dynamic Organic Design Agreement Initiative Renewal Design organic dynamic Predictive Design close to agreement organized dynamic far from certainty close to certainty Certainty

  24. Understanding Organic Dynamics (Activity) • Divide into triads • Selects one other triad member (doesn’t tell) and uninvolved person in refreshment area • Stay at least two feet apart and equidistant from the other two • Do this for about 1-2 minutes while trying to reach refreshments • Reflect on experience

  25. Case Study Introduction Do the preconference professional development offerings contribute to effective evaluation-related work of association members? If so, how?

  26. Unorganized System Dynamics What is happening? What boundaries, differences, similarities, and relationships might shape how the offerings contribute to participants’ evaluation-related work?

  27. Organized System Dynamics Do participants receive high-quality content that is relevant to their evaluation-related work and is delivered through high-quality instructional methods?

  28. Organized System Dynamics How do the format and content of the session support or hinder participants in understanding and using the session to apply the learning from the session to their evaluation work?

  29. Organic System Dynamics What patterns among participants (including the session facilitators) before and during the session are likely to affect the participants’ understanding and application of the learning to their evaluation-related work?

  30. Patterns

  31. Patterns

  32. Patterns

  33. Patterns

  34. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 10)

  35. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 27)

  36. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p.42)

  37. Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 78)

  38. Patterns

  39. Fractals:Patterns, Patterns Everywhere In nature . . . Mathematical construct of iterating nonlinear equation and plotting on complex number plane—Mandelbrot Set Similar shapes at all scales—Broccoli Biological scaling gives coherence in widely diverse entities—Oak tree Scale-free networks

  40. Recognizing patterns is critical:similarities, differences, and relationships that have meaning across space and time Basic values or simple rules generate diverse, but self-similar behavior across scales Naming and telling stories about dynamics in a system help reinforce and shape fractal patterns Fractals:Patterns, Patterns Everywhere

  41. Fractals

  42. Looking at the Dynamics as a Whole • What is the overall picture of system dynamics affecting how the preconference professional development offerings contribute to effective evaluation-related activities of AEA members? • Given the findings from the three system dynamics within the preconference session, how might the preconference professional development process and offerings be modified to contribute more substantially to the quality of AEA members’ evaluation-related work?

More Related