1 / 31

Creating and Using Evaluation Tools for Program Success

Creating and Using Evaluation Tools for Program Success. Greater Richmond Association for Volunteer Administration July 10, 2014. What is Evaluation?.

blaine
Download Presentation

Creating and Using Evaluation Tools for Program Success

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Creating and Using Evaluation Tools for Program Success Greater Richmond Association for Volunteer Administration July 10, 2014

  2. What is Evaluation? Program evaluation is carefully collectingand analyzing information about a program or some aspect of a program in order to make necessary decisions. Continual improvement is an unending journey.

  3. Types of Simple and Meaningful Tools • Surveys • Personal interviews • Focus groups • Other data collections tools • Existing databases

  4. What Can Evaluation Tools Capture? Designing a good tool requires more time and thought than you may think. Questions can tap into: • Behaviors • Opinions • Feelings • Knowledge • Experiences • Characteristics

  5. Who Can Be Helpful? • Depends on the purpose – who can answer your questions? • If not apparent, ask for guidance from key informants. • If potential population is large, generate a list of candidates and key characteristics then randomly select.

  6. What is a Survey? A quantitative research project in which a relatively large number of people are interviewed, each being asked a standard set of questions, posed in the same way each time.

  7. Why Use Surveys? • To collect specific, standardized data across respondents. • To collect data across many people as efficiently as possible. • To increase understanding. • To collect data quickly, even across multiple groups. • Open-ended quotes may add impact and credibility.

  8. Types of Surveys Method • Phone • Snail Mail • Web-Based • Blended Approach Question Types • Open-ended • Closed-ended • Combination

  9. The Survey Tool DO: • Keep the survey as short as possible. • Balance white space and length. • Keep questions neutral. • Be specific in your wording. • Pretest your questions. • Address only one construct per question. • Use a blend of closed and open-ended questions, if reasonable. • Mix types of questions throughout. • Provide an opportunity for additional comments.

  10. How to Enhance Participation DO: • Provide a reason for the survey. • Use clear instructions. • Explain confidentiality protections. • Give a realistic sense of the time involved. • Track respondents. • Use reminders. • Provide your contact information. • Request contact information from the respondent, if appropriate. • Establish an end date.

  11. What is an Interview? An interview is the collection of data by asking people questions and following up or probing their answers.

  12. Why Use Interviews? • To increase understanding. • As an exploratory first step to creating quantitative tools. • Enhance understanding of interesting findings which emerged from other processes. • Real-world quotes may add impact and credibility. • To collect specific, standardized data across respondents.

  13. Types of Interviews Method • Face-to-face • Telephone Style • Conversational: go with the flow • Interview guide

  14. Conversational Approach • Questions emerge through conversation • Highly individualized • May provide greater insights • Requires interviewer with strong content and interpersonal skills • Not systematic • Can be more difficult to analyze

  15. Interview Guide Approach • Outline of topics exists, but wording and order can vary • Topics covered is more systematic • Interview setting remains fairly informal • Also required considerably skilled interviewer • Inhibits spontaneous discussion of new topics

  16. Open-Ended Structured Approach • Strict script for question, but free-response format for answers • Most efficient of qualitative techniques • Reduces bias and guides less experienced interviewers

  17. The Interview Tool DO: • Keep the interview as short as possible. • Keep questions neutral. • Be specific in your wording. • Pretest your questions. • Address only one construct per question. • Use a blend of closed and open-ended questions, if reasonable. • Provide an opportunity for additional comments.

  18. Recruitment Techniques DO: • Provide a reason for the interview. • Give a realistic sense of the time involved. • Explain how your interview candidate fits into the bigger goal.

  19. Preparing for the Interview DO: • Plan an appropriate interview location. • Exchange contact information with the interviewee. • Prepare to capture information. • Create response cards, if needed.

  20. Building Rapport DO: • Be on time. • Review the purpose of the interview. • Discuss how the results will be used. • Reiterate confidentiality protections. • Acknowledge the time frame you have set aside. • Provide your contact information on paper. • Ask if they have questions before you begin.

  21. General Tips • Don’t be judgmental! • Use appropriate non-verbal cues. • Attend to non-verbal cues. • Repeat and clarify when needed. • Practice active listening. • Let the interviewee know when you are switching gears. • Keep the interviewee focused. • Don’t make assumptions.

  22. What is a Focus Group? A focus group is a group discussion. Participants are brought together in a neutral location for the specific purpose of discussing and issue or responding to ideas or materials of interest.

  23. Why Use Focus Groups? • To increase understanding. • As an exploratory first step to creating quantitative tools. • Enhance understanding of interesting findings which emerged from other processes. • Real-world quotes may add impact and credibility. • To collect specific, standardized data across respondents.

  24. Types of Focus Groups Method • Exploratory • Issue-Focused

  25. Focus Group Model DO: • Plan on a 2 hour session • Secure 8-12 participants and a few alternates • Identify 5-7 key questions beforehand • Provide background information to group. • Keep questions neutral. • Record all input. • Have subject matter experts available to answer questions. • Use an objective facilitator.

  26. Recruitment Techniques DO: • Provide a reason for the interview. • Give a realistic sense of the time involved. • Explain how your interview candidate fits into the bigger goal. • Establish potential participant list • Invite participants by letter followed by phone call • Consider offering incentives

  27. Preparing for the Session DO: • Secure a roomy, centrally-located venue • Select a facilitator • Exchange contact information with the participants • Prepare to capture information.

  28. Building Rapport DO: • Be on time. • Review the purpose of the interview. • Discuss how the results will be used. • Reiterate confidentiality protections. • Acknowledge the time frame you have set aside. • Provide your contact information on paper. • Ask if they have questions before you begin.

  29. Other Tools • Observations • Checklists (process documentation) • Attendance logs • Case records • Existing databases

  30. Parting Tips • Keep it simple • Balance simplicity with precision • Focus on interpretation • Blend methods if appropriate

  31. Questions? Please contact: Trina Willard Principal Knowledge Advisory Group 804-564-6969 Trina@KnowledgeAdvisoryGroup.com

More Related