1 / 45

IDRC`s Evaluation Principles

Documenting Results as Behavioural Change: The Use of Outcome Mapping by International Research Organizations Friday, November 9, 2001 AEA Conference St. Louis, Missouri. IDRC`s Evaluation Principles. Learning & action oriented Process and product Each study is unique Participation

seanna
Download Presentation

IDRC`s Evaluation Principles

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Documenting Results as Behavioural Change:The Use of Outcome Mapping by International Research OrganizationsFriday, November 9, 2001AEA ConferenceSt. Louis, Missouri

  2. IDRC`s Evaluation Principles • Learning & action oriented • Process and product • Each study is unique • Participation • Capacity building for participants

  3. Results Along the Influence Chain Inputs Activities Outputs Outcomes Impacts

  4. Shifting Influence Over Time Influence Time

  5. Impact Implies: Cause & effect Positive, intended results Focus on ultimate effects Credits a single contributor Story ends when program obtains success Dev`t Research Implies: Open system Unexpected positive & negative results occur Upstream effects are important Multiple actors create results & need credit Change process never ends The Problem with « Impact »

  6. Our Methodological Approach

  7. What is Outcome Mapping? • an integrated PM&E tool • an approach that focuses on changes in the behaviour, relationships,or actions of partners (as outcomes) • a methodology that characterizes and assesses the program’s contributions to the achievement of outcomes • an approach for designing in relation to the broader development context but assessing within your sphere of influence

  8. Assessing External Results Boundary Partner Program

  9. Assessing Internal Performance Boundary Partner Program

  10. Assessing Influence Boundary Partner Program

  11. Program`s Sphere of Influence The Real World Program = Program`s Boundary Partners

  12. Assessing Change: Progress Markers • A graduated set of statements describing a progression of changed behaviours in the boundary partner • Changes in actions, activities, & relationships leading up to the ideal outcome statement

  13. Levels of Progress Markers The program sets out what it would: • Expect to seeits boundary partners doing? • Like to seeits boundary partners doing? • Love to seeits boundary partners doing?

  14. Why Graduated Progress Markers? • Articulate the complexity of the change process • Permit on-going assessment of partners’ progress (including unintended results) • Encourages the program to think about how it can intentionally contribute to the most profound transformation possible • Make mid-course corrections & improvement easier

  15. Savita KulkarniSwayamsiddha Project,BAIF Development Research Foundation (BAIF)Pune, India

  16. Swayamsiddha…. • Women’s Health and Empowerment • At 9 locations in 6 states of India • Multi-Stakeholder Project • Gender Integration, Health, Linkage, Research, Capacity Building, Generating Learning are key factors • Aims at behavioral changes at various levels

  17. Process…. • Introduced in June 2000 to all implementing teams – partial introduction • Introduced the idea to the core groups of two organizations • Core groups introduced the idea to the field teams • OM is seen as a tool for self-assessment - for implementing teams

  18. Why OM in Swayamsiddha? • Means to capture process (women’s empowerment), richness, diversity • Means to capture behavioural changes • Need for a better monitoring tool • Emphasis on learning • Sustainability

  19. Learnings….. • OM motivates – it helps to plan & monitor better • Is a useful tool for working with even illiterate people • OM useful in visioning the process – brings out different perceptions • Concept of boundary partners • More useful when not related to LFA • M&E clarity is the pre-requisite to implement OM

  20. Challenges…... • Number of Boundary Partners involved • Boundary Partners at different stages (e.g. Women’s Self-Help Groups) - heterogeneous • Who should define the Progress Markers and for whom? • Monitoring of Progress Markers - by whom? How?

  21. Challenges…. • Compilation of Progress Markers at different levels • Quantitative and Qualitative aspects even when used for self-assessment • Boundary Partners go on changing as the project intervention progresses - frequent visit to monitor Progress Markers becomes necessary

  22. OM in Swayamsiddha Today • Using it for self-assessment - for all implementing teams in 4 areas - Gender, Research, Health and M&E • Progress Markers for Gender for central team are being defined. • Chaitanya using it for SHG and Education • Using OM - beyond project - for organizational capacity building

  23. Raj VermaNagaland Empowerment of People Through Economic Development Program(NEPED)Nagaland, India

  24. OUTCOME MAPPING: FOR SELF ASSESSMENT, DESIGN, MONITORING AND EVALUATION. NAGALAND EMPOWERMENT OF PEOPLE THROUGH ECONOMIC DEVELOPMENT (NEPED) PROJECT

  25. WHERE IS NAGALAND AND WHAT DOES NEPED DO? NAGALAND • WORKS WITH SHIFTING CULTIVATION FARMERS TO: • PROMOTE AGRO FORESTRY • CONDUCT ON FARM FARMER LED TRIALS • ENCOURAGE CASH CROP PLANTATION FOR INCOME • PROVIDE CAPITAL TO SET UP MICRO FINANCE • EMPOWER LOCAL GRASSROOT INSTITUTIONS • ADDRESS GENDER AND LAND TENURE ISSUES • DISSEMINATE BEST PRACTICES • IMPROVE VILLAGE GOVERNANCE • FACILITATE FARMER ACCESS TO MARKETS • ALLEVIATE POVERTY BY PROVIDING ALTERNATIVES

  26. WHY OUTCOME MAPPING AT END OF FIRST PHASE? • LACK OF BASELINE SURVEY AT INCEPTION • SCANTY AND DIS-ORGANIZED DOCUMENTATION • INTERNAL M&E, SELF ASSESSMENT • EXTERNAL MID-TERM EVALUATION RECOMMENDATION • TO PREPARE FOR END-OF-PROJECT EXTERNAL EVALUATION • TO REFLECT AS A TEAM FOR SELF EVALUATION • TO SHARE AND RECORD EXPERIENCES

  27. OUTCOME MAPPING IN PHASE 2 • REALIZATION OF ITS VALUE IN PHASE 1 • RECOGNITION OF MULTI-PURPOSE USE • WILL ENRICH AND FEED INTO DONOR SPECIFIED M&E FORMAT • MORE CONSCIOUS UTILIZATION • EASE IN USING, COMFORTABLE WITH IT

  28. FOR WHAT OUTCOME MAPPING WILL BE USED • FOR PROJECT DESIGN, PLANNING STRATEGIES • FOR FINE TUNING, INTRODUCING ‘TWEAKS’ • FOR INTERNAL MONITORING, IDENTIFYING GAPS • FOR EVALUATING PROGRESS PERIODICALLY • FOR ASSIMILATING AND COLLATING DATA • FOR MEETING REPORTING NEEDS • FOR PRIORITIZING ACTIVITIES • FOR BUILDING PARTNERSHIPS

  29. PREPARATION FOR VALIDATION WITH BOUNDARY PARTNERS • SIMULATED ROLE PLAY • METHODOLOGY & TOOLS TO BE USED • LOGISTICS REQUIRED • FLAG “DON’T DO” AND “MUST DO” • TEAMWORK – ASSIGN ROLES

  30. “VISION” STATEMENT OF FARMERS AS VISUALIZED BY THEM: WHAT WOULD CHANGE FOR THEM IF NEPED WERE TO BE “WILDLY SUCCESSFUL”

  31. WHAT WE GAINED FROM OUTCOME MAPPING • CONCEPTUAL CLARITY OF THE PROJECT • DEVELOPMENT IS ABOUT PEOPLE AND CHANGE • EMPOWERING OTHERS, BROADENING THE CANVAS • CAPACITY BUILDING AND NEW SKILLS • COHESION WITHIN THE TEAM AND PARTNERS

  32. Thank you!We welcome your feedback, suggestions, & questions.evaluation@idrc.cabaif@vsnl.comrajverma@vsnl.comhttp://www.idrc.ca/evaluation

  33. Additional Slides

  34. Step 3: Boundary Partners Those individuals, groups, & organizations with whom the program interacts directly to effect change & with whom the program can anticipate some opportunities for influence.

  35. Boundary Partners (have boundary partners) Program’s Boundary Partners Boundary Partners’ Boundary Partners Program

  36. Example Progress Markers The program Expects to See local communities: • Participating in regular model forest (MF) partnership meetings 2. Establishing a structure for cooperation in the partnership 3. Acquiring new skills for involvement in the MF 4. Contributing resources necessary to get the MF operational (minimum)

  37. Example Progress Markers The program would Like to See local communities: • Articulating a vision for the MF that is locally relevant • Promoting their involvement with the MF nationally • Expanding the partnership • Calling upon external experts when necessary • Requesting new opportunities for training • Disseminating concrete examples of benefits arising from MF activities • Seeking out new partners for the MF • Obtaining funding from different national sources

  38. Example Progress Markers The program would Love to See local communities: 13. Helping other MFs establish themselves 14. Sharing lessons-learned internationally 15. Influencing national policy debates &formulation on resource use and management

  39. Facilitation Questions ? « How can the program know the boundary partner is moving toward the outcome? What would they be doing? » « What milestones would be reached as the boundary partner moves towards their intended role in contributing to the vision? »

  40. Step 6: Strategy Map • Outlines the program`s approach in working with the boundary partner • Indicates the relative influence the program is likely to have on boundary partner • Helps pinpoint strategic gaps in the approach or if the program is overextended • Suggests the type of evaluation method appropriate to track and assess the performance of the program

  41. 6 Types of Strategies Strategy Causal Persuasive Supportive • I-1 • Direct Output • I-2 • Arouse New Skills/ Thinking • I-3 • Supporter who guides change over time Aimed at the Boundary Parnter • E-1 • Alter physical or regulatory environment • E-2 • Modify the information system • E-3 • Create / Strengthen a Peer Network Aimed at the Boundary Partner`s Environment

  42. 6 Types of Strategies

  43. Facilitation Questions ?

  44. Comparing Boundary Partners

More Related