Matching Evaluation to the Nature & Conceptualization of the Problem

1. Matching Evaluation to the Nature & Conceptualization of the Problem Simple, Complicated and Complex Problem Framings and the Implications for Evaluation Brenda Zimmerman, Schulich School of Business York University, Toronto, Canada Presentation to the Joint Canadian Evaluation Society/American Evaluation Association Conference Toronto Oct 29, 2005

2. Social Transformation – our definition • An alteration of what is established by the introduction of new elements or forms (including new ideas, practices, or resource flows). In particular the alteration of social relationships to allow for an improvement in or transformation of intransigent and broadly based social problems

3. Our approach to social innovation • Connecting understanding of large scale patterns or dynamics to action at the level of individuals or groups • Concerned with how to understand these interactions in such a way that we could help those trying to make a difference

4. The Decision to Use a Complexity Lens • Complexity theory leads us to look at the relationship between the micro and the macro • It suggests how individuals act effectively in arenas which they cannot control (complex is more than complicated) • Complexity theory points to what is happening “in the between”; in the relationships between things. • Implications for evaluation??

5. Simple Complicated Complex Following a Recipe A Rocket to the Moon Raising a Child • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipe notes the quantity and nature of “parts” needed • Recipes produce standard products • Certainty of same results every time

6. Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Simple Complicated Complex Following a RecipeA Rocket to the MoonRaising a Child • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipes produce standard products • Certainty of same results every time

7. Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Simple Complicated Complex Following a Recipe A Rocket to the Moon Raising a Child • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipes produce standard products • Certainty of same results every time

8. Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Simple Complicated Complex Following a Recipe A Rocket to the Moon Raising a Child • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipe notes the quantity and nature of “parts” needed • Recipes produce standard products Certainty of same results every time

9. Sources for charts of next three slides Begun, Zimmerman and Dooley, “Health Care Organizations as Complex Adaptive Systems”, in S. M. Mick and M. Wyttenbach (eds.), 2003 Advances in Health Care Organization Theory San Francisco: Jossey-Bass, pp 253-288 Also can be found at www.change-ability.ca

10. Adapted from Dent 1999

15. The Brazil Case Study • annual per capita income is less than $5000 • In the 1980s, Brazil’s AIDS problem was worse than South Africa’s • By 2000, South Africa’s HIV infection rate was 25% whereas Brazil’s was 0.6% • In 1992, the World Bank predicted that Brazil would have 1.2 million AIDS cases by 2000 • …but the actual count was 0.5 million. Brazil Analysis is from Begun, Zimmerman and Dooley, 2002

16. The Brazil Complex Problem • How do you respond to an AIDS epidemic in a developing country? • No money for expensive drugs • problems of affordability of drugs • A very iffy health care system • problems of treatment • High levels of illiteracy • problems of compliance even if there were drugs • High levels of poverty and hunger • problems of nutritional needs for drugs

17. Making the Brazil Problem Complicated • “What will drug costs be for our infected population?” • “What resources are needed to manage drug therapies for illiterate patients?” • “What resources are needed to assure compliance with drug associated nutrition in this population?” • “What are the resources needed for an effective prevention program?”

18. World Bank Responds to AIDS as Complicated • Meaningful solutions require sophisticated, integrated national health care systems • We cannot provide treatment to all when the drug costs are so high • We cannot afford resources to manage treatment compliance • With our limited resources, we should focus more on prevention than treatment • It will therefore take a long time for the problem to work itself through

19. Brazil Implicitly Recognized the Complexity • Began to address the AIDS issue head-on in 1994 • Were unwilling to accept the answers of the World Bank • Hence – had to change the questions… • Changing the questions changes the focus, changes what is “analyzed”, changes what is seen as possible/impossible ?

20. “What will drug costs be for our infected population?” “What resources are needed to manage drug therapies for illiterate patients?” “How can we reduce costs so that we can provide treatment to all who need it?” “What methods of communication will work to convey the drug therapy routine to a patient – even a homeless, illiterate patient?” The Brazil Questions Assume Complexity World Bank Questions Brazil Questions

21. “What resources are needed to assure compliance with drug associated nutrition in this population?” “What are the resources needed for an effective prevention program?” “If food is an issue, how can we ensure greater compliance with the routine by linking up with charities that can provide food at the right times of day?” “How can we achieve our prevention goals while treating all of those currently infected?” The Brazil Questions World BankQuestions Brazil Questions

22. Meaningful solutions require sophisticated, integrated national health care systems We cannot provide treatment to all when the drug costs are so high We cannot afford resources to manage treatment compliance With limited resources, focus more on prevention than treatment It will therefore take a long time for the problem to work itself through Find ways to use the resources we have to respond to the problem Provide drugs to all by finding ways to reduce drug costs Use our informal system to train people to care for themselves Prevention will be part of the treatment Seek short and long term results The Brazil Conclusions Brazil Conclusions World Bank Conclusions

23. The Brazil Responses • A stable container: Brazil built on existing infrastructure (natural network existed) • A somewhat shaky health system of hospitals and clinics • Added to by 600 NGOs, churches (hubs) • Free drugs to all AIDS patients: faced down drug companies in pursuit of national interest

24. The Brazil Responses - cont’d • Patients managed their own drugs • Illiterate people were taught by local “trustworthy” folks (hubs were sought) • Free treatment spread prevention ideas • No labeling of those affected • Prevention information readily available

25. Brazil saw the value in the relationships as the key to change

26. The Brazil AIDS story - optimism • COURAGE– to challenge WTO, USA, large pharmaceuticals, World Bank • COMPLEXITY – lived with the complex nature of the society, and the AIDS problem • RELATIONSHIPS – used the power of existing relationships to learn and enhance their connectedness

27. Matching Evaluation to Cycles of Intervention & Program Development Ecocycle & Panarchy as a Framework

28. The ecocycle as a metaphor for successful social innovation • Based on the work of ecologists studying natural systems (C.S. Hollings in particular) • Suggests four distinct stages in a “life cycle”, linked to two key dynamics: connectedness or sameness and stored capital or potential

29. MATURITY GROWTH BIRTH

32. The birth, growth, destruction and renewal of a forest Released POTENTIAL Stored 1 Exploitation Variety CONNECTEDNESS Sameness

34. exploitation

35. The Psychosocial place called “exploitation” • Triggers:Choice! Groups of people have converged on some promising products or processes and are devoting time and energy to these. Decision to invest in one or more and build capacity • People: the implementers come into their own. Invest in people who love to take ideas and develop them into reality, the team-builders and the engineers come into their own. • Experience: Excitement, flow, high energy, doors opening, long hours, time flying by….often correlated with peak experiences. A time of intense activity, flow and productivity with a steep organizational “learning curve” • Evaluation: “Formative” – look for a continuous feedback systems as the start-up phase moves towards greater goal clarity and efficiency in delivery • Management: Supporting capacity building. Careful attention to the need for training, hiring and developing systems to support bringing a program “to market” and moving it up the scale. Evaluation geared to providing feedback for increasing effectiveness and efficiency. • Traps: Failure to build sufficient capacity or sufficient “lock-in” to achieve economies of scale and delivery efficiency

36. The birth, growth, destruction and renewal of a forest Released POTENTIAL Stored 1 Exploitation Variety CONNECTEDNESS Sameness

38. conservation

39. The Psycho-social place called “conservation” • Triggers: Reaching a level of efficiency and learning where improvements become incremental. The “mature” phase, productivity high and measurable, peak “earnings” and efficiency. • People: Invest in those excellent at management, system design excel, others may be getting restless, or even a little bored. • Experience: satisfaction at success, the pride in the well oiled machine, anxiety about mushrooming demand, loss of momentum • Evaluation:Summative evaluation – success based on goals should be measurable by quantitative or qualitative means. • Management: awareness of vulnerability, standing still (again), openess to creative destruction. Encourage “letting go” • Traps: rigidity, loss of peripheral vision, over-commitment. Fear of change.

40. The birth, growth, destruction and renewal of a forest Conservation Released POTENTIAL Stored 1 Creative Destruction Exploitation Variety CONNECTEDNESS Sameness

42. Creative destruction • Not devastation of “soil” • Crisis…. from root word “to sift” • Creation requires destruction • To honor creation… open to destruction • Externally imposed • Internally driven

43. Creative destruction

44. Creative destruction

45. The psychosocial space called “creative destruction” • Triggers:most clearly signaled by the discontinuation, destruction or collapse of a given program or process and the simultaneous release of resources and energies for other things. • People: those who thrive on crisis, on new beginnings are happy here – others may be depressed or in mourning • Experience: of confusion, identity crisis,, change in relationships, anxiety, elation. • Management: providing reassurance, “standing still”, ‘active” reflection, contemplation, listening. • Evaluation: organizations should be “harvesting”: identifying lessons learned and thinking about implications for the future. • Traps: Fear and guilt can act to isolate – communication is needed. • Outputs: Focus on new ideas, investment of capital in exploration.

46. Renewal Also known as Exploration or Reorganization

48. Renewal/exploration

49. The psychosocial space called “exploration” • Triggers:The need for innovation. Development of multiple “random walks”, experiments, initiatives which lead to little in the way of measurable outcomes for considerable time • People: people who learn by doing are happy here; invest in individuals who are entrepreneurs/ innovators with a track record. • Experience: Reflection moving to experimentation, lots of false starts and sometimes frustration and mounting anxiety about inputs/output ratios • Management: Generate resources and connections and exchange of information, encourage experimentations and learning, have faith in the self-organizing properties of information and groups. Introduce new ingredients as required. Exert increasing pressure for output. • Evaluation: geared to capturing learning, building transparent accounts of innovation. • Traps: “spinning wheels”, the incapacity to select a course and commit resources to it to the required degree. Ungovernable competition.

50. The birth, growth, destruction and renewal of a forest 4 2 Renewal Conservation Little CAPITAL STORED Much 3 1 Creative Destruction Exploitation Weak CONNECTEDNESS Strong