Evaluation and the Science of Complexity

Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29th -30th May 2008

Agenda • Evaluations – some common issues • Complexity science – origins and ideas • Implications for evaluations • Summary

The history of M&E in the international development sector is in four distinct phases • 1960s to 1979: Early developments • 1979-1984: Rapidly growing interest • 1984 to 1988: M&E matures • 1988 to the present: the crossroads

Wealth of tools, techniques and approaches are now available • Logical framework analysis • Results-based management • Needs Assessments, Impact Assessments • Ex ante and ex post assessments • ZOPP • GANTT • Social Network Analysis • Appreciative Enquiry • Most Significant Change • Outcome Mapping • Many many more!!

For many organisations, evaluations are at the centre of a vicious circle... Increased competition ...causing problems for M&E Increased pressure to show results and impact Growing need for high profile fundraising and advocacy work Poor learning and accountability Lack of professional norms and standards

Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes

Reflection, learning and analysis are threatened by existing agency cultures and processes Existing culture & process Accountability Org. Learning Evaluation

Evaluations, like other similar initiatives, often sit on top of existing organisational silos, inefficiencies and power imbalances, rather than resolving them Evaluation KM Silos

Agencies plough the same evaluation “field”, but stick to their own furrows

Understanding of the effectiveness and use of evaluations is weak at best… Where’s the data??!

Evaluations tend to be based on wish lists, not strategies, and therefore are often overloaded

Leadership and political buy-in to evaluation is rare and unreliable, with two common reactions

...all of which means that (1) evaluation efforts resemble this iceberg... What is planned What actually happens

And (2) evaluators spend most of their time feeling like this... working against this...

“Exploring the science of complexity” • Primary aim was to explore the potential value of complexity science for those who work on change and reform initiatives within the aid sector • Drew on scientific and experimental literature – physiology, physics, mathematics, public sector reform, sociology, economics, organisational theory, plus case studies, reports and evaluations from the aid sector • Reviewed over 250 articles, books, reports and evaluations • 10 peer reviewers, including 5 directors of leading research institutes • Published February 2008 • Available to download from www.odi.org.uk

A (well-known) story… A man was walking home one dark and foggy night. As he groped his way through the murk he nearly tripped over someone crawling around by a lamp post. “What are you doing?” asked the traveler. “I’m looking for my keys” replied the other. “Are you sure you lost them here?” asked the traveler. “I’m not sure at all,” came the reply, “but if I haven’t lost them near this lamp I don’t stand a chance of finding them.”

A closer inspection of the light under the lamp revealed… INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

Logical frameworks / results chains articulate a clear world view and theory of change: “The light under the lamp” • The machine metaphor - universe as clockwork • The future is knowable given enough data points • Phenomena can be reduced to simple cause & effect relationships • Dissecting discrete parts would reveal how the whole system worked; science was the search for the search for the basic building blocks • The role of scientists, technologists & leaders was to predict and control - increasing levels of control (over nature, over people, over things) would improve processes, organisations, quality of life, entire human societies

Key Assumptions • Assumptions about systems • Ordered • Reductionist - parts would reveal the whole • Assumptions about how systems change • Linear relationships • influence as direct result of force from one object to another - predictable • Simple cause & effect • Assumptions about human actions • Rational choice • Behavior specified from top down • Certainty and “knowability”

...Reality of aid is a little different...

But a new light is being turned on (slowly, unevenly, using a dimmer switch)…

Complexity science is a science of understanding change • A loosely bound collection of ideas, principles and influences from a number of other bodies of knowledge, including • chaos theory • fractal geometry • cybernetics • complex adaptive systems • postmodernism • systems thinking • Discovery of similar patterns, processes and relationships in a wide variety of phenomena • related to the nature and dynamics of change

From microscopic chemical reactions…

...to the evolution of galaxies...

Complexity scientists use a range of ideas and concepts (familiar, challenging and baffling) to make distinctions between simple, complicated and complex systems and phenomena

Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Complicated Complex Simple Following a Recipe A Rocket to the Moon Raising a Child • The recipe is essential • Recipes are tested to assure replicability of later efforts • No particular expertise; knowing how to cook increases success • Recipe notes the quantity and nature of “parts” needed • Recipes produce standard products • Certainty of same results every time

The claims of complexity scientists • The complexity of real world systems is (usually) not recognised or acknowledged by scientists and policy makers • Or, that if it is not acknowledged, they don’t deal with them • Or, that if they do deal with them, they don’t do so effectively • Or, that if they do deal with them effectively, it’s because they used an specific approach / framework • ...that is also available to you, dear client, at a reasonable daily rate plus a per diem • [JOKE]

There have been diverse efforts to apply ideas to social, economic and political analysis and practice • Arthur, Ormerod - Economics • Stacey, Snowden - Organisations • Jervis, Urry, Cutler - Intl relations • De Mancha - History • Gilchrist - Community development • Education policy - Sanders and McCabe • Health policy - Zimmerman • Government reform - Chapman • Strategic thinking – Saunders • Evaluation - Williams

Complexity and systems approaches have already proved useful in re-thinking aid and development issues • Uphoff, 1990s • Chambers, 1997 • Sellamna, 1999 • IDRC, Outcome Mapping, 2001 • Warner, 2001 • Rihani, 2002 • Lansing and Miller, 2003 • Inclusive Aid, 2004 • ECDPM, 2004-06 • Eyben, 2006 • Guijt, various • Davies, Network Analysis, various

10 key concepts and implications Features of systems 3 Emergence from Simple Rules 1 Interconnected and interdependent elements and dimensions 2 Feedback processes Dynamics of change 7 Strange attractors and the edge of chaos 5 Sensitivity to initial conditions 6 Phase space and attractors 4 Non-Linearity Behaviours and relationships ` 8 Adaptive Agents 9 Self organisation 10 Co-Evolution

Complex systems • Collection of parts, which collectively have a range of dimensions • Parts share an physical or symbolic environment / space • Action by any part can affect the whole • E.g. individuals, families, communities, cities, markets, societies, populations, economies, nations, planets

Complex systems are interconnected and interdependent to different degrees • Interconnectedness may occur between any elements, dimensions, systems and environments • This interconnectedness leads to interdependence between the elements and the dimensions of a system, and gives rise to complex behaviour • Complex systems can be tightly or loosely coupled, internally and with their environment, giving rise to different kinds of complex behaviours • Tightly coupled: Global FOREX markets • Loosely coupled: US University system, global construction industry • Systems can be understood via mapping techniques, followed by analysis to understand the dynamics and interactions of the system

Foot and mouth disease: an example of failure caused by focusing on one part of the system and ignoring the links between sub-systems (biology, geography, economics) Economic rationalisation of abattoirs and EU subsidies increased the interconnectedness of herds to a critical point. Changes to foot and mouth reporting rules delayed the isolation of infectious animal The relationship between these actions and the epidemiology of F&M was not appreciated in advance where it mattered because the livestock industry was not viewed as a interconnected, interdependent system

Complexity also means that systems need to be understood at different scales Communities Atom Organisms Molecule Tissue Cell Organs

E.g. evaluating the effectiveness of child health programmes Private Sector Other NGOs A.N. NGO Community and Family Local partners Church Developing Country Govmts Civil Society

Example: evaluating resource flows in the humanitarian system

Issues of interconnectedness, interdependence and scale usually do not become apparent until a crisis... • Foot and mouth disease, UK • Economics, cattle management, disease • September 11th • Globalisation and terrorism • Climate change • Western consumerism and Southern disasters • Credit crunch • US mortgage market mis-selling and the world economy • Food prices • Biofuels and food consumption • Vulnerability to natural disasters • Sichuan earthquakes and dams • Human trafficking • Desire, economics and rights abuses

...indicating that we have biases in the way we view the world... • Three different kinds of problems have been identified, along with some common biases in dealing with them • They are: • “Messes” • “Problems” • “Puzzles”

“Messes” are issues that do not have a well defined form or structure. NB not a value statement, but a description • There is often not a clear understanding of the problem faced • Messes often involve economic, technological, ethical and political issues • It has been suggested that all of the really important issues in the world start out as messes. • For example, how was rising HIV/AIDS incidence in Brazil dealt with in the 1990s? • concerned money, technology, ethics, social relations, politics, gender relations, poverty • all of these dimensions of the problem had to be dealt with simultaneously, and as a whole

Many of the major problems we face are “messes”! • Credit crunch • US mortgage market mis-selling and the world economy • Food prices • Biofuels and • Vulnerability to natural disasters • Sichuan earthquakes and dams • Climate change • Western consumerism and Southern disasters • September 11th • Globalisation and terrorism • Human trafficking • Sexual preferences and human rights abuses • Arms trade • Economics and war • Etc, etc, etc

“Problems” are issues that have a known or knowable form or structure • The key dimensions and variables of a problem are known and the interaction of dimensions may also be understood, even if only partially. • With problems, there is no single clear cut way of doing things • there are many alternative solutions, depending on the constraints faced • Expertise matters • For example, dealing with the sewage system in a particular city may rely on amount of money available, technology, political stance of leaders, climatic conditions, urban development, the road system, and so on

Puzzles are well defined and well structured known problems with a specific “best” solution • Solutions can be worked out and improved • Solutions are replicable - “best practices” are possible

Policy and traditional science is biased towards puzzle-solving • Real-world, complex, messy nature of systems is frequently not recognised • Simple puzzle-based solutions are applied to complex messes • E.g. Global War on Terror has been applied as the single best solution to the mess of terrorism

“Some of the greatest mistakes have been made when dealing with a mess, by not seeing its dimensions in their entirety, carving off a part, and dealing with this part as if it were a problem, and then solving it as if it were a puzzle, all the while ignoring the linkages and connections to other dimensions of the mess”

Or to put it another way: dividing a cow in half does not give you two smaller cows

Implications: analyse and deal with the reality of the system • Multidimensionality, interdependence and interconnectedness of poverty and humanitarian crises (and responses to them) should be recognised by those designing, managing and evaluating aid interventions • Analysis may need to be in line with historical research - not ‘did x cause y?’ but ‘what happened and why?’, building narratives about events and processes. • The task of selection and synthesis of data becomes as important as analysis • Different perspectives on what the system is need to be taken into account, especially when these perspectives differ as to the nature, interconnectedness and scale of the system • Messes, problems and puzzles need to be identified and dealt with using appropriate approaches

Interconnectedness and interdependence gives rise to a range of phenomena and behaviours

To find out more, read the paper! Features of systems 3 Emergence from Simple Rules 1 Interconnected and interdependent elements and dimensions 2 Feedback processes Dynamics of change 7 Strange attractors and the edge of chaos 5 Sensitivity to initial conditions 6 Phase space and attractors 4 Non-Linearity Behaviour of agents ` 8 Adaptive Agents 9 Self organisation 10 Co-Evolution

Evaluation and the Science of Complexity