1 / 43

Choosing and Using Evaluation Options: Sharing Information and Resources

This resource provides information and guidance on selecting and implementing evaluation methods. It includes a wide range of evaluation options and encourages the sharing of experiences and expertise. The content covers various topics related to evaluation processes, tools, and techniques.

geraldineboone
Download Presentation

Choosing and Using Evaluation Options: Sharing Information and Resources

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sharing information about choosing and using evaluation options Patricia Rogers – RMIT University Canberra Evaluation Forum20 June 2013 Canberra

  2. Challenges It is hard to choose the right combination of evaluation methods and proceses It is hard to know how to implement them well

  3. Opportunities Lots of material on evaluation methods (some not maintained) Many people willing to share their experience and expertise

  4. Documenting Sharing COMMUNITY R & D Events Descriptions WEBSITE Comments Examples Guides Tools

  5. Financial Supporters Founding Partners

  6. Contributors and collaborators include …

  7. Co-creation by practitioners, researchers, sector experts, method experts 

  8. Reflection, processes for quality and authenticity

  9. Rooted in practical experience

  10. Advice from experienced practitioners

  11. Photo: David Masters Task oriented taxonomy: Support to navigate options

  12. FRAMEWORK Over 200 methods/options related to 35 tasksin 7 clusters

  13. Methodological pluralism Simon Kneebone‘Show Me The Change”http://www.flickr.com/photos/smtc/sets/72157624112171834/

  14. Spotlight on invisible evaluation tasks and less well known methods

  15. Respectful, inclusive and constructive discussion

  16. What methods might you use – or suggest in a ToR? Under what circumstances might you use them? How could you oversee the quality of their use? Scenario 1- describing a situation • Community renewal project • Have existing social indicator profile • Want to learn how the residents see their community – what are its strengths and problems? • Intended users of the evaluation – local authorities for planning and advocacy

  17. Information from individuals • Deliberative Opinion Polls • Email Questionnaires: • Face to Face Questionnaires • Global Assessment Scales • Goal Attainment Scales (GAS) • Internet Questionnaires • Interviews • Key Informant • Logs and Diaries • Mobile Phone Logging • Peer/Expert Reviews • Photolanguage • Postcards • Projective Techniques • Questionnaires • Seasonal Calendars • Sketch Mapping • Stories (Anecdote) • Telephone Questionnaires

  18. Information from groups • After Action Review • Brainstorming • Card Visualization: • Concept Mapping • Convergent Interviewing • Delphi Study • Dotmocracy • Fishbowl Technique • Focus Groups • Future Search Conference • Hierarchical Card Sorting • Keypad technology • Mural • ORID • SWOT Analysis • World Cafe

  19. Scenario 1- PhotoVoice Participatory photography method Particularly useful for giving a voice to marginalized people Community members : • Take photos • Select the most significant photos • Tell stories about what they mean • Identify key themes What questions would you ask?

  20. What methods might you use or suggest in a ToR? Under what circumstances might you use them? How could you ensure quality? Scenario 2(a)- causal inference • Pilot project of service delivery to individual clients • Intended users – department managers to inform decision about future funding

  21. What methods might you use – or suggest in a ToR? Under what circumstances might you use them? How would you ensure quality? Scenario 2 (b) - causal inference • Completed health intervention to reduce early childhood obesity • Series of evening information sessions • Ten participating families • Data (from service records) show children’s weight was high at the beginning and lower afterwards • Intended users – management of health service to inform decision about future funding

  22. Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers. Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne College of Advanced Education. Presented as a case study in Owen, J. (1999) Program Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin

  23. Weights of subsequent children Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers. Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne College of Advanced Education. Presented as a case study in Owen, J. (1999) Program Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin

  24. Scenario 2 - Check timing is consistent with program theory • Use time series data to go beyond just ‘Before’ & ‘After’ • See if the change trajectory is consistent with the theory that the program has caused the observed results

  25. Scenario 2 - General Elimination Method • Identify possible alternative explanations • Gather and analyze data to see if these can be ruled out

  26. Scenario 2 - methods for causal inference • Which methods were suggested by the group? • Which methods were not suggested? • What new methods sound potentially useful? • What information would you need to be able to use (or oversee someone using) a new method?

  27. Content • Over 200 options pages • Over 500 resource pages • 16 approaches (package of methods) • 8 thematic pages – sectors or cross-cutting issues

  28. Choose the right optionUse it well Information on methods : • Description • Examples • Advice for choosing the method • Advice for using the method • Short list of recommended resources • A full list of all resources, ordered by user rating • This information helps users determine if the method suits their needs and requirements

  29. www.betterevaluation.org

  30. How could you benefit from BetterEvaluation?

  31. How could you contribute to BetterEvaluation?

  32. Partnering with BetterEvaluation An AusAID perspective

  33. Context • Pressures to demonstrate effectiveness of growing aid program • Independent Review of Aid Effectiveness recommendations • Strengthen our evaluation approach (evidence) • More rigorous reporting on aid program effectiveness • Standards for whole of government aid initiatives • All projects (>$3m) to undertake an evaluation • Challenges • lack of baselines/data, • range of programs and partners, • constraints of working in developing countries • diverse staff training needs

  34. Why partner with BetterEvaluation? • Because it’s GREAT! • Generous – free access • Rigorous –promotes rigor and good quality resource materials • Easy – to navigate and interactive options • Accessible & Active – housed in Australia but often travelling! • Timely – coincides with increasing demand for monitoring and evaluation tools by aid program and other managers

  35. What do we get out of it? • Helpful – practical advice & insights • Encouraging – opportunity for feedback / discuss ideas • Leveraging Opportunities – access to new networks & opportunity for us to influence direction • Promotion – opportunities to share our evaluation products and learning

  36. What’s next? • To GROW the Partnership • Go places – BE is expanding relationships with other organisations and agencies/greater reach • Reach out Overseas – to our partners in Asia-Pacific, development of local case study materials in local languages & translations • Work more collaboratively – stronger focus on developing Canberra and Post skills & increasing participation on the site

  37. Questions/Comments?

  38. Further Information • Comprehensive Aid Policy Framework • http://www.ausaid.gov.au/about/Documents/capf.pdf • 2012 Annual Review of Aid Effectiveness • http://www.ausaid.gov.au/makediff/Pages/annual-review-aid-effectiveness.aspx • 2013 OECD DAC Peer Review of Australia • http://www.oecd.org/dac/peer-reviews/OECD%20Australia%20FinalONLINE.pdf • ODE Website: www.ode.ausaid.gov.au

More Related