Evaluating the Complex: Navigating Interpretive Frameworks in Evaluation Practice

1. Evaluating the Complex:Getting to MaybeMichael Quinn Patton Oslo, Norway 29 May, 2008 Michael Quinn Patton May, 2008

2. Evolving Understandings I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they haven’t been thinking. Noam Chomsky “The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13. Michael Quinn Patton May, 2008

3. Interpretive Frameworks • May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber. • They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment." Michael Quinn Patton May, 2008

4. They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have. Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data. In essence, they found that interpretive capacity, or "mind-sets," distinguish high-performance more than data quality and accuracy. Michael Quinn Patton May, 2008

5. Evaluation’s TraditionalInterpretive Framework In the beginning… Michael Quinn Patton May, 2008

6. Original Primary Options Formative and Summative Evaluation (Mid-term and End-of-Project Reviews) Michael Quinn Patton May, 2008

7. Evidence-based Practice Evaluation grew up in the “projects” testing models under a theory of change that pilot testing would lead to proven models that could be disseminated and taken to scale: The search for best practices andevidenced-based practices Michael Quinn Patton May, 2008

8. Fundamental Issue:How the World Is Changed Top-down dissemination of “proven models” versus Bottoms-up adaptive management Michael Quinn Patton May, 2008

9. Models vs. Principles Identifying proven principles for adaptive management (bottoms-up approach) versus Identifying and disseminating proven models (top down approach) Michael Quinn Patton May, 2008

10. Conditions that challenge traditional model-testing evaluation • High innovation • Development • High uncertainty • Dynamic • Emergent • Systems Change Adaptive Management Michael Quinn Patton May, 2008

11. Mintzberg on Strategy Two types of strategy: Intended & Emergent Unrealized Strategy Intended Strategy Deliberate Strategy Realized EmergentStrategy Strategy Michael Quinn Patton May, 2008

12. Re-conceptualizing Use • Use is a process not a event • Use involves an interaction not just a report • Use involves training for use not just delivery of results Michael Quinn Patton May, 2008

13. Some premises: • Evaluation is part of initial program design, including conceptualizing the theory of change • Evaluator’s role is to help users clarify their purpose, hoped-for results, and change model. • Evaluators can/should offer conceptual and methodological options. • Evaluators can help by questioning assumptions. • Evaluators can play a key role in facilitating evaluative thinking all along the way. • Interpretative dialogue is critical. • Designs can be emergent and flexible. Michael Quinn Patton May, 2008

14. Three ways of conceptualizingand mapping theories of change • Linear Newtonian causality • Interdependent systems relationships • Complex nonlinear dynamics Michael Quinn Patton May, 2008

15. Linear Logic Model INPUTS (people, materials) ACTIVITIES (processes)  OUTPUTS  OUTCOMES  CHANGES IN PEOPLES LIVES  IMPACTS  CHANGES IN COMMUNITIES Michael Quinn Patton May, 2008

16. Michael Quinn Patton May, 2008

17. Causal Diagram of Beckhard’sChange Formula Believability First Steps Michael Quinn Patton May, 2008

18. Feedback Michael Quinn Patton May, 2008

19. Michael Quinn Patton May, 2008

20. Michael Quinn Patton May, 2008

21. Systems • Parts are interdependent such a change in one part changes all parts • The whole is greater than the sum of the parts • Focus on interconnected relationships • Systems are made up of sub-systems and function within larger systems Michael Quinn Patton May, 2008

22. Systems Concepts in Evaluation – An Expert Anthology. 2006. Bob Williams and Iraj Imam AEA Monograph, EdgePress/AEA Point Reyes CA. Michael Quinn Patton May, 2008

23. Michael Quinn Patton May, 2008

24. Understanding the Elephant from a Systems Perspective Michael Quinn Patton May, 2008

25. The relationship between what goes in and what comes out What conceptual framework informs front-end evaluation work? Michael Quinn Patton May, 2008

26. Teen Pregnancy Program Example Michael Quinn Patton May, 2008

27. Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens 2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome) Michael Quinn Patton May, 2008

28. Prenatal program staff Her parents & other family members Teachers/ other adults Young pregnant woman's attitudes & behaviors Child's father & peers Her peer group Systems web showing possible influence linkages to a pregnant teenager Michael Quinn Patton May, 2008

29. Prenatal program Prenatal Clinic and Hospital Outreach Other Systems -- welfare -- legal -- nutrition programs -- transportation -- child protection -- media messages Context factors -- politics -- economic incentives -- social norms -- culture -- music Youth Culture SCHOOL SYSTEM Young pregnant women's attitudes & behaviors Other community-based youth programs Church Program systems web showing possible institutional influences affecting pregnant teenagers: Michael Quinn Patton May, 2008

30. Using Different System Lenses to Understand a “particular” System • Economic System • Inputs/Outputs • Cost/Waste/Value/Benefits • Customers/Suppliers • Political System • Power • Governance • Citizenship • Equity • Biologic System • Emergence • Coordination/synergy • Structure, Process, Pattern • Vitality • Sociologic System • Relationships • Conversations • Interdependence • Loose-tight coupling • Meaning/sense • Anthropologic System • Values • Culture/Milieu SYSTEM DIMENSIONS • Information System • Access • Speed • Fidelity/utility • Privacy/security • Storage • Mechanical / Physical System • Flow • Temporal Sequencing • Spatial Proximities • Logistics • Information • Psychological System • Organizing • Forces Field • Ecological/Behaviour Settings Michael Quinn Patton May, 2008

31. Michael Quinn Patton May, 2008

32. Complex Dynamic Systems Configuration Michael Quinn Patton May, 2008

33. HIV/AIDS Example • Hits every system: health, family, social, religious, economic, political, community, international • Requires multiple interventions on multiple fronts in all subsystems simultaneously • Resulting reactions, interactions, consequences dynamic, unpredictable, emergent, and ever changing Michael Quinn Patton May, 2008

34. Challenges: Situation Recognition and Appropriate Evaluation Designs Michael Quinn Patton May, 2008

35. The nature of EXPERTISE Michael Quinn Patton May, 2008

36. Contingency-basedEvaluation • Situational analysis & responsiveness • Context sensitivity • Clarify and focus on intended users: stakeholder analysis • Clarify and focus on intended uses • Methodological appropriateness • Criteria for evaluating the evaluation: credibility, meaningfulness Michael Quinn Patton May, 2008

37. Seeing Through A Complexity Lens “You don't see something until you have the right metaphor to let you perceive it”.Thomas Kuhn Michael Quinn Patton May, 2008

38. Complex Nonlinear Dynamics • Nonlinear: Small actions can have large reactions. “The Butterfly Wings Metaphor” • Emergent: Self-organizing, Attractors • Dynamic: Interactions within, between, and among subsystems and parts within systems can volatile, changing • Getting to Maybe: Uncertainty, unpredictable, uncontrollable Michael Quinn Patton May, 2008

39. SNOWDEN’s CYNAFIN FRAMEWORK Simple, Complicated, Complex, Chaotic and Disordered Behaviours Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

40. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008

41. Conceptual Options • Simple • Complicated • Complex Michael Quinn Patton May, 2008

42. Types of Community IssuesThe Stacey Matrix Far from Agreement Close to Far from Certainty Close to Michael Quinn Patton May, 2008

43. Simple Far from Agreement Simple Plan, control Close to Far from Certainty Close to Michael Quinn Patton May, 2008

44. Simple (Known) Inter-relationships between elements : Tight, centralised connections. Anyone can see the things the way they are. Very simple linear cause and effect. Everyone knows the right answer within the current context (which of course may not be self-evident or known to others – hence importance of context). Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

45. Simple Complicated Complex Following a Recipe A Rocket to the Moon Raising a Child • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipes produce standard products • Certainty of same results every time Michael Quinn Patton May, 2008

46. Complicated (Knowable) Inter-relationships between elements: Relationships are looser but still clustered around a central core. Cause and effect is non-linear. Relationships able to be modelled and able to predicted. An expert would know the right answer(s) Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

47. Technically Complicated Far from Agreement Technically Complicated Experiment, coordinate expertise Simple Plan, control Close to Far from Certainty Close to Michael Quinn Patton May, 2008

48. Simple Complicated Complex Following a RecipeA Rocket to the MoonRaising a Child • Formulae are critical and necessary • Sending one rocket increases assurance that next will be ok • High level of expertise in many specialized fields + coordination • Rockets similar in critical ways • High degree of certainty of outcome • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipes produce standard products • Certainty of same results every time Michael Quinn Patton May, 2008

49. Socially Complicated Far from Socially Complicated Build relationships, create common ground Agreement Simple Plan, control Technically Complicated Experiment, coordinate expertise Close to Far from Certainty Close to Michael Quinn Patton May, 2008

50. Socially complicated Implementing human rights agreements, like gender equity or outlawing child labor Environmental Initiatives • Many different and competing stakeholders • Diverse vested interests • High stakes Michael Quinn Patton May, 2008