Practically promoting learning and accountability Analysis of three approaches

1. Practically promoting learning and accountabilityAnalysis of three approaches

2. Choosing and combining an approach and method that are based on the situation in which an evaluation takes place ensures that the process not only promote learning and accountability at the point of evaluation… Learning and Accountability

3. … it will also contribute to strengthening learning and accountability for the life of the program as well as dramatically improving relationships partnerships program quality organisational practice Learning and Accountability

4. Peer to peer learning evaluation in India & Bangladesh Three evaluation approaches Participatory impact evaluation in Laos Real time evaluation in Solomon Islands

6. Encouraged wonderful participation and ownership Offered each other great insights, very quickly Fostered incredible trust and honesty around difficult issues Built relationships for learning Its resource intensive! What we learned…about learning

7. Encouraged accountability to peers and colleagues and co-workers Highlighted accountability to other stakeholders Connections and relationships built on regional Indigenous people’s issues Challenges to relationship with Oxfam – our accountability to our partners Met AusAID requirements What we learned…about accountability

8. Reviewing the Poverty Reduction program in Lao 2007 Mrs Manivanh interviewing Miss Soy, village midwife Lapung village March 2007

9. Review process 10 month process Extensive training on M&E with OAus staff Review of exisiting PRA data 3 day field visits to 9 villages in Met, Feuang and Ta Oy districts – with follow up visits to verify data Village and focus group discussions: women meetings, men, poorest villagers & young people Individual semi-structured interviews Village mapping Verification workshop with OAus staff – draft outcomes of review presented for feedback Final report and production of DVD

10. Village mapping March 07

11. Some of the review outcomes Clean water systems: reduction in water borne diseases and women and children’s work load Health education: changes in behaviour Latrines: decrease in diseases Buffalo and cow banks: Reduced workload, reduced debt and increased status Animal vaccination: reduction in illness and death of animals. Village medicine funds: Reduced health problems: diarrhoea, fever, gastrointestinal problems.

12. Some outcomes Village health volunteer: change of behaviour Rice bank and other revolving funds assisted to reduce periods of food shortage Youth education program: young people earning income working in their own village and in neighbouring villages – mechanics, carpentry and tailoring School teacher’s houses: increased retention of teachers, improved quality of teaching and space for meetings.

13. OAus staff with 2001 PRA data, Saravan office March 2007

14. Key recommendations Prioritise vulnerable communities (ie: climate change, government resettlement program) Monitor nutrition levels of children Provide literacy classes for adults Explore opportunities for research, other collaborations and partnerships Be realistic re: program planning and resources Strengthen distance management systems

15. Key learning's from the review process Importance of engaging National field staff – including leadership – from the beginning of the review process Ensure there is open, regular and honest communication between Head Office / National office Clarify the purpose of the review clearly at the outset – and for whom. But be open to the exploration of unintended outcomes of the review process itself. Build a shared (Head office and National field staff) understanding of ‘impact’ and ‘outcomes’.

16. Key learning's Challenge pre-conceived Head office notions and expectations of impact and success Value existing indigenous knowledge, skills and experience Build M & E capacity & knowledge of local staff Create safe & open space for the articulation & discussion of negative or unintended outcomes Provide clear information re: the future use of the material collected through the review process – stories, photographs & videos.

17. SRI rice paddy field, Feuang district Photo: Mr Keovsai Bonsory, August 2007 “They visit together…they stayed overnight….they eat with us….they taught us how to do it ourselves and they keep coming back” (villager, Lapung village, Ta Oy district, March 2007).

18. Feedback from staff about the review process itself Staff acknowledged feelings of anxiety and nervousness – about an evaluation of their program, about doing M&E itself and the scrutiny of a HO and AusAID. At the end of the review process staff commented on: how much they had learnt; how the review had validated the work they were already doing re: PRAs and monitoring and evaluating existing programs and how much they learnt about the impact of their work with communities. In addition, workers commented that they had learnt to ask different questions of villagers and listen differently to their comments and experiences.

19. Some unanswered questions How do we develop local definitions / understandings of ‘success’? How do we engage with a closed, conservative and patriarchal political, social and cultural context to receive feedback re: our work with indigenous communities and women? How do we create a safe space for villagers to share information re: what wasn’t working well and why? How to ensure staff have the time and resources to build their skills and knowledge; conduct review; while also implementing programs?

20. Local Nek women, Vac Tai village, Kalum district (March 2008).

21. Real time evaluationSolomon Islands • Humanitarian response Tsunami/Earthquake in Western Province of Solomon Islands • Shelter materials • WASH – water, sanitation and health

22. Approach …a rapid evaluation carried out during the early stages of a humanitarian response …findings immediately disseminated to key relevant stakeholders …directly feed into the development and adaptation of the program and associated processes and practices …written reports can be produced for communication and institutional learning purposes

23. Approach Stage one Apr-June management, decision-making and resourcing & preparedness Stage two Sept quality, participation, accountability, impact and collaboration, from a partner and beneficiary perspective How? Focus group discussions, workshops timeline mapping exercises, visits and discussions in camps and communities Who? OAus staff PDU, EMU, SI DM Co-ord, SI consultant

24. What we learned … about learning • Encouraged conversations • Timeliness was key – e.g. recruitment, planning for rehabilitation • Led to more learning – assessments, evaluations • Ongoing institutional changes in humanitarian response • Staff time constraints an issue

25. What we learned? …about accountability • Focussing on accountability and quality was important • Significant improvements in accountability to and participation of communities • Assisted a lot with gender sensitive programming • Enhanced accountability to NZAid • Clear actions and accessible reporting led to action!

26. Summary – three approaches