EVALUATING, REVIEWING REFLECTING & REVISING

1. EVALUATING, REVIEWING REFLECTING & REVISING Ros Tennyson

2. Some definitions: Essentially we seek to understand / describe the process of acquiring, analyzing, interpreting, adapting, disseminating and (ultimately) transforming experience and knowledge

3. Some further definitions:

5. In Phase 1: • Set up parameters, baselines and performance indicators • Creating review / evaluation systems • Ensuring commitment (‘buy-in’) to evaluation and review from partners

6. In Phase 2: • Collecting project data (outputs and outcomes) • Tracking performance / Keeping records (decisions, events, changes) • Maximising partner resource contributions / relationships and engagement

7. In Phase 3: • Evaluating project outcomes and impacts • Assessing the the partnership as an effective mechanism for sustainable development activity • Reviewing partner relationships to assess and maximise value

8. In Phase 4: • Undertaking and disseminating a final evaluation Or: • Agreeing new parameters and performance indicators • Revising systems • Creating new review and evaluation procedures

9. Evaluating Partnerships: What do we most want to know? 1. That the partnership has been effective in achieving its aims 2. That the partners have all benefited from their involvement 3. That the partnership approach was / is the best way to do it We need to understand the bits that are not obvious and that we can’t see Painting by Magritte

10. 1 IMPACT ASSESSMENT 3 EVALUATING PARTNERING AS A MECHANISM 2. PARTNER RELATIONSHIP REVIEW AN EMERGING EVALUATION APPROACH FOR PARTNERSHIPS:

11. 1. Impact assessment To evaluate: • Tangible impacts: • – Technical / Physical • – Human • – Financial • Intended / unintended outcomes • Value (of investment)

12. 3. Evaluating partnering as a mechanism To understand: • Transaction costs / value added •  Sustainability of outcomes • Strategic influence • Systemic impact / reform Whether a partnership approach was ultimately better than the next best alternative

13. Formula for assessing ‘added value’ of a partnership • AV = (OP + SC) – (RC + NA + EC + OC + FC) • Key: • AV = Added Value of a Partnership • OP = Outcomes of Partnership • SC = Social Capital • RC = ResourcesContributed • NA = Net Benefit of the Next Most Likely Alternative • EC = Environmental Contributions • OC = Opportunity Costs (eg time spent) • FC = Facilitation Costs

14. 2. Partner relationship review To reveal:  Value for partner organisations (& other stakeholders) – expected – unexpected – potential  Degree of effectiveness / efficiency / impact  Level of influence (sectoral / strategic)

15. This form of ‘evaluation’ links to: • A review of partnering principles in practice (ie is the partnership equitable, transparent and mutually beneficial?) • Whether the partnership is achieving individual goals / underlying interests of partner organisations or not • Exploring whether the partners have made maximum use of the range of resources available • Whether the partnership could work better – if so, how? • Whether the partnership could do something quite different – if so, what?

16. Who are the audiences for a partner review? Brainstorm…

17. This is a new ‘science’ and it raises questions about: • Validity? • Reliability? • Legitimate measurement? • Ethical considerations? • Added value?

18. A working hypothesis… Any truly valid and effective review of a partnership always need to: • Involve all partners & key stakeholders in design and data collection • Include a genuine feedback loop so that the process truly informs the development of the partnership • Find a good balance between external ‘objectivity’ and internal knowledge / experience of the partnership’s history

19. Sources of knowledge: • Your own and others experiences and observations • Formal records (eg minutes, proposals, action plans, agreements) • Physical evidence • Other relevant materials giving contextual information (eg newspapers, legislation, local events) • Theories and hypotheses (to challenge your thinking and understanding) • Guidelines / manuals • Case studies

20. Ways of Knowing… • Sensory (sight, smell, touch, hearing, taste) • Mental (memory, patterns, shapes) • Intellectual (concepts, logic, assumptions) • Intuitive (instinct, feelings) • Imaginative (vision, empathy)

21. Capturing knowledge involves… • Looking for tangible evidence • Empowering and inviting people to reveal what they know • Active listening / Astute observation • Asking ‘open’ questions • Record keeping (endless note-taking; keeping a ‘log book’) • Good filing systems (or a good memory)

22. Being aware of the ‘filter’ of the reviewer… • Preconceptions, assumptions and prejudices • Cultural / sectoral / political ‘lens’ • Personal values / belief system • Professional discipline / training • Experiences (good or bad) of partnering to date • Capacity to make sense of complex and multi-source data

23. Making sense of data involves… • Active engagement and interest • Attention to detail • Willingness to explore contradictions • Sifting and selecting material • Interpretation and clarification • ‘Triangulation’ (confirmation) of findings • and above all…

24. Time for quiet reflection! Painting by Salvador Dali

25. Tools: Questionnaires SWOT analysis Open questioning Observed activity Tangible evidence Document analysis Other? Processes: Written responses 1:1 interviews Partner groups Partnership groups Third-party inputs Contextual evidence Other? Tools and technologies

26. “As a reviewer, make no assumptions and always remember to expect the unexpected” THE UNEXPECTED ANSWER by Rene Magritte