Methods Enabling Constituency Voice in Impact Evaluation for Improving Development

1. MethodsEnabling Constituency Voicein Impact Evaluation for Improving Development Andre Proctor, Keystone March 2009

2. Rethinking IMPACT • Impact is not something that happens at the end of a linear chain of activities and outcomes. • It occurs wherever we (independently or with others) cause a significant change in a system that shifts that system towards sustaining the outcomes that we strive for. Hence IPAL - Impact planning, assessment and learning

3. IPAL innovations Simple practical Reliable Useful transformative

4. If we can’t imagine sustainable system change then how will we ever be able to achieve it?

6. A new theory of relativity… …..e=mc2

7. What is Constituency Voice • Bringing the voices of constituents, especially those meant to benefit, meaningfully into decision-making. • All constituents play a role in • How we define success • How we work towards it (strategy) • How we learn (measure impacts and reflect on their meaning) • How we communicate it (reporting and sharing)

9. CV addresses the 6 purposes of Impact Planning, Assessment and Learning (1) • Improve projects • ensures usefulness for constituents • Demonstrate impact • from different perspectives on key enablers of change, insight in causal mechanism, triangulation • Inform strategy • real time feedback

10. CV addresses the 6 purposes of Impact Planning, Assessment and Learning (2) • Sustain credibility and legitimacy • Confidence and trust among constituents • Build capacity • empowers constituents • Educate society • reporting re-imagined as a learning activity both for constituents and wider society

11. Comparative Constituency Feedback • An alternative approach to measuring performance and driving improvements: • “Ask them!” • Ask recipients how useful they find an organisation’s work. • Related to ‘customer satisfaction’. • Empowers primary constituents.

12. An emerging field Macro level: • Humanitarian Response Index www.daraint.orgAll OECD government funding for humanitarian response • Center for Effective Philanthropy www.effectivephilanthropy.orgGrantee perceptions of US foundations Micro level: • Measuring Empowerment quality.bond.org.ukSocial movement in Bangladesh • Listen First www.listenfirst.orgConcern Worldwide pilot

13. Comparative feedback

14. Comparative constituency feedback • Structured feedback from primary constituents on their experience of receiving funding / services / assistance. • Generates quantitative data, for a group of similar actors. Performance can be aggregated and benchmarked. • Provides data from the bottom upwards for learning and improvement.

15. Case study Thanks to Centre for Effective Philanthropy

16. East-African grantmakers • Cohort of 10 grantmakers. • Keystone surveyed all of their grantees. • 50% response rate, 336 completed surveys • Six key areas:

17. Ethical Framework

18. Summary findings (extract) “XXX’s grantees are very satisfied with XXX’s application processes, rating it 15.3 out of 20. … XXX keeps grantees better informed of progress and provides them with clearer information than most other grantmakers.” “XXX’s grantees rated it 6.4 out of 20 for non-financial support (e.g. capacity building or advice). All grantmakers struggle in this area. Around 40% of XXX’s grantees do not receive assistance in each area; and in some areas over 20% of grantees receive assistance that they do not find helpful.”

22. Follow up questions… • “Could XXX consider: • more site visits, and think about what would be most valuable to do on a site visit? • reviewing its communications – especially web site and its annual report as communication tools? • improving the way that it listens and responds to feedback from Grantees (responding to the feedback from this survey could be a good start). • developing its staff understanding of developmental issues and create opportunities for learning from constituents? • others …”

23. Conclusion • Robust data on grantmaker’s performance – not just focusing on grantees. • Comparison drives insight and learning; approach fits with other monitoring activities. • Drives real change (e.g. CEP). • Adaptable to many different situations; the field is emerging. • Puts the rhetoric of empowerment and accountability into practice. • The future … ??

24. Discussion questions CV • How does Constituency Voice contribute to Impact Evaluation for Improving Development? • How useful could Comparative Constituency Feedback be in what you do?