Assessing Contribution and Enhancing Social Learning: An Introduction to Outcome Mapping (OM)

1. Assessing Contribution and Enhancing Social Learning:An Introduction to Outcome Mapping(OM) 16 December 2004 Kaia Ambrose

2. Presentation Outline • OM in Carchi, Ecuador. • Underlying logic of OM. • Brief outline of stages of OM. • Some key concepts - boundary partners, their changes and how we measure that change. • OM and learning organizations. • The OM toolbook. • Conclusions and ideas for the afternoon session.

3. First, a little context… • Ecopar • Ecuadorian NGO focused on research, training and capacity building in tropical ecosystems/life zones. • The Ceja Andina Project • IDRC-funded. • Sustainable use of agriculture and forest biodiversity in the ceja andina (cloud forest) region of the northern Ecuadorian Andes. • Research-oriented (traditional and participatory), social learning processes, policy development and strengthening of local government. • Developing OM since March 2002 – “walking it through and making it make sense in our context”.

4. M&E concerns in the Ceja Andina Project • As a team and as an NGO, how can we learn from the project in order to improve the way we “do” development (in a research context)? How can we become a learning organization while still being accountable? • How can we learn (iterative processes) together with our local partners? • How can we bring creativity and real interest into the M&E process? (create culture of M&E) • What is the balance between rigor and utility in our M&E objectives? • Assess need for OM (complementary, bits of).

6. What is Outcome Mapping? • an approach for designing M&E in relation to the broader development context but assessment is within your sphere of influence. • an integrated and participatory M&E approach; also for planning. • an approach that views outcomes as changes in the behaviour, relationships, or actions of partners. Development is accomplished through changes in the behaviour of people. • a methodology that characterizes and assesses the program or project’s contributions and influence to the achievement of outcomes.

7. Program / project’s Sphere of Influence The Real World PROGRAM (performance and strategies) = Program’s partners (behavioural change)

8. Outcome Mapping does not: • Focus on impact (we’re not in the impact business, we’re in the business of change). • Look for attribution (we look for contribution). • Isolate contributions of program / project in achievement of results. • Promote linear, cause-effect thinking in a “sterile” context. • Focus solely on actions promoted by program / project.

9. Influencing Outcomes Partners Program Assessing Program Influence Assessing Internal Performance Assessing Changes in Behaviour

10. Outcome Mapping

11. OM can help us: • Besides designing or clarifying programme logic, helps record and assess monitoring data: • How far have our boundary partners progressed towards achieving outcomes? • What are we doing to support the achievement of outcomes? • Indicate cases of positive performance and areas of improvement; front stage and back stage. • Evaluate intended and unexpected results; positive deviance. • Gather data on the contribution that a program made to bringing about changes in its partners (strategies - new tools, aptitudes, knowledge). • Establish evaluation priorities and evaluation plan.

12. Boundary Partners – Core ideas • Boundary partners – individuals, groups, organizations with whom the program interacts directly and with whom the program anticipates opportunities for influence. • Outcome challenges – description of the ideal changes in the behaviour, relationships, activities and/or actions of a BP that contribute to vision. • Progress markers – a set of graduated indicators of changed behaviours for a BP that focus on the depth and quality of change.

13. Boundary Partners • Social change is about relationships. • We’re all interconnected. • Social learning – create consensus, among multiple cognitive beings, regarding direction of program in terms of influencing outcomes, in order to arrive at concerted action. • Boundary partners participate in process of OM (in M&E).

14. Boundary Partners (have their own BPs) Ceja Andina Project Municipality Rural Agriculture Association Town population Pesticide salesmen Programme’s Boundary Partners Boundary Partners’ Boundary Partners Programme

15. Ask yourselves … « In which individuals, groups, or organisations is our programme trying to encourage change so that they can contribute to the vision?  With whom will we work directly? » « Are we choosing X boundary partner because we want to influence their behaviour and actions, or because they will influence others? Or both?» « What behavioural changes do we (collectively, between project and boundary partners) want to see in BP that will contribute to the vision (Outcome Challenges)?.»

16. Progress Markers • Describes progression of changed behaviours in the boundary partner (is a set, as opposed to a single indicator in order to demonstrate the depth and complexity of change process). • Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement. • Expect to see, like to see, love to see.

17. Progress Markers = Change Ladder Love to see PMs Truly transformative. Set quite high. Like to see PMs More active learning, engagement. Expect to see PMs Early response to program’s basic activities.

18. Example Progress Markers The project Expects to See local communities: 1.Participating in field day demonstrations. 2. Establishing a structure for cooperation in the partnership. The project would Like to seelocal communities: 3. Articulating a vision for the ceja andina forest that is locally relevant. 4. Requesting new opportunities for training. The project would Love to seelocal communities: 5. Influencing national policy and debates on CBNRM.

19. Progress Marker Checklist Each Progress Marker: • Describes a changed behaviour by the boundarypartner • Can be monitored & observed • Limit the number of PMs. As a set, Progress Markers: • Are graduated from easier to more difficult to achieve changes in behaviour • Describe the change process of a single boundary partner Remember: progress markers are changeable!!

20. 6 Types of Strategies The program facilitates new tools, techniques, resources in order to contribute to outcomes and development process. Strategy Causal Persuasive Supportive • I-1 • Direct Output • I-2 • Arouse New Skills/ Thinking • I-3 • Supporter who guides change over time Aimed at the Boundary Partner • E-1 • Alter physical or regulatory environment • E-2 • Modify the information system • E-3 • Create / Strengthen a Peer Network Aimed at the Boundary Partner`s Environment

21. Creating your strategy map

22. Organizational Practices • Help us to define our role as an NGO according to changing contexts and shifting paradigms. • Important to how the program is going to function to effectively fulfill its mission. • Supporting change in its boundary partners requires that the program be able to change and adapt as well. • The things that we do as an organization to: • foster creativity & innovation • seek the best ways to assist partners • maintain our niche

23. Eight Organizational Practices • Prospecting for new ideas, opportunities, & resources. • Seeking feedback from key informants. • Obtaining the support of your next highest power • Assessing & (re)designing products, services, systems, and procedures. • Checking up on those already served to add value. • Sharing your best wisdom with the world. • Experimenting to remain innovative. • Engaging in organizational reflection.

24. Journals Set overall intentions and strategies; design and articulate program’s logic (how it will contribute to change within a complex system). Record internal and external data: How we are progressing towards outcomes. How we are contributing to change. Indicate cases of positive performance /improvement. Assess intended and unexpected results.

25. Outcome and Performance Monitoring Boundary Partners Program Strategies and Activities Organizational Practices Outcome Challenges and Progress Markers Strategy Journal Outcome Journal Performance Journal

26. Conclusions • OM can help a program to be more strategic about the actors it targets, the changes it expects to see, and the means it employs to support and facilitate those changes. • OM is not based on a cause-effect framework; it recognizes that multiple, nonlinear events lead to change. It sees interrelationships, not a snapshot. • Attribution becomes contribution.

27. Conclusions • Focus is on behavioural change – monitors and evaluates whether a program has contributed to changes in behaviours in a way that would be logically consistent with supporting development changes in the future. • Program must also learn and change, reconsider and adjust its goals, methods, interventions. • Self-assessment, systematically collected.

28. Conclusions • Improving rather than proving. • Understanding rather than simply reporting. • Creativity and knowledge, rather than just taking credit.

29. For more information, case studies, materials, articles and presentations on Outcome Mapping: www.idrc.ca/evaluation

30. Afternoon session (what I propose) • Fun dynamic! Identify your boundary partner! • Continued, in-depth presentation on: • Boundary partners. • Outcome challenges. • Progress markers. • Practical exercise on progress markers. • Challenges / successes of the methodology. • Open discussion. • Postcard exercise (key messages).

31. Not everything that counts can be counted. And not everything that can be counted, counts. Albert Einstein

32. OM challenges • Workload / paperwork– we made too much work for ourselves and didn’t prioritize (too many progress markers, tried to monitor everything). • Trial-and-error reporting system. • Scaling-up within NGO. • New boundary partners. • Different boundary partners have different experiences with the projects (example: municipalities). • Project team had no formal training on the methodology.

33. OM challenges • Weak monitoring plan (no one to lead OM from the beginning). • How to combine with more quantitative data collection. • Encouraging on-going boundary partner participation in OM M&E activities.

34. OM successes • Build cohesion with team and with partners. • Identify and develop interconnectedness among boundary partners through dialogue, consensus and collaborative action; social learning space. • How to connect with people becomes part of the methodology. • Highly participatory. • Donor is flexible – encourages learning, creativity, adjustments. Makes space for dialogue and interaction (and really listens!) • Self-assessment “all around” (of team, of boundary partners).

35. OM successes • Innovation of tools and application of tools. • Build culture of M&E within NGO / team and boundary partners. • Concrete data for developing evaluation plan. • Meetings for discussion / self-assessment = learning organization (analyze our learning). Permanent system of reflection. • Interest to learn more about OM by other NGOs in Ecuador (request for information, presentations, etc.) = good promotion of methodology (and project).