Download
1 / 51
510 likes | 629 Views
Session 3. Implementing Outcome Evaluation (Part Two): Practical Implementation of Your Evaluation Framework.
E N D
Implementing Outcome Evaluation (Part Two): Practical Implementation of Your Evaluation Framework • At this workshop, we will help you design useable measurement tools and address the practical issue of developing an organizational structure that allows you to implement and sustain your evaluation.
What to Expect from this Series • Learn about steps required to implement outcome evaluation • Design a program logic model • Identify question and indicators • Identify appropriate evaluation/measurement tools • Develop an implementation plan
From outcomes, to questions, to indicators, to methods What were we trying to change? (Long-Term Outcome Objectives) What particular contribution were we going to make to that change? (Short-Term Outcome Objectives) Did our program have an impact??? How were we planning to make that contribution? (Activities) What sorts of things would we see if the expected change was happening? (Indicators) How can we document those observations in a systematic way? (Methods)
Today: • FIRST, figure out what the implications of your indicator exercise are for you • SECOND, prioritize the possible actions, and • THIRD, come up with a viable action plan
A Framework forOutcome Measurement (We will go through the elements of this table one by one throughout the day)
Analyzing your Indicator exercise data Figuring Out What Your First Steps Ought to Be
Typical Situations • We’re having a hard time coming up with evaluation questions • We don’t have a lot of existing indicators: we don’t gather much data now • We gather lots of data, but we don’t really analyze it
Typical Situations • We gather data and we use it, but it isn’t telling us much that’s new • We don’t make good use of our evaluation findings when promoting our programs • We’ve got so many programs, ideas and needs, we don’t know where to start. Different programs are at very different stages.
Pulling out the specific implications (example) The measurement possibilities that emerged out of the exercise: • Extracting some more information out of existing data, particularly around trends over time in attendance and access, and around participant satisfaction. • Making contact with partner agencies that run similar programs and/or program which refer to or from yours, in order to get their impressions of how your clients are doing and also to find out what their experiences have been with attendance rates. • Creating some type of new tool to gather feedback from clients. Some combination of a pre/post knowledge test, a more detailed feedback form, and/or more exploratory focus group discussions. • More ambitious and sophisticated ideas like videotaping parent/child interactions and tracking your clients after they leave your program to see what happens with future pregnancies or housing choices.
Pulling out the specific implications (example) The next step for you seems to be an intensive and focused analysis of your rich existing data You may also want to think about new ways of communicating these findings in new ways for outside audiences. In addition, you are thinking of adding a very in-depth and unstructured qualitative component to the mix through some case studies.
Pulling out the specific implications (example) Possible Next Steps: • Doing some intensive analysis of the data you have already gathered. • Revamping your existing client feedback survey to ask more behavioural questions and perhaps switching to a pre-test post-test format • Developing a tool that would allow you to seek feedback from caregivers or family members
Discussion What are the emerging first steps for you?
Keep in Mind • We will talk this afternoon about estimating resource implications of these ideas.
Pros and cons of popular measurement techniques Less Rigour • Retrospective reflections or stories • Self report (interview or survey) • Peer/family/worker report • Direct observation • File review • Clinical assessments More Rigour
The “Big Three” • Surveys • Focus groups • Individual Interviews
When to Use Focus Groups: The Advantages • Quick, cheap, easy to assemble • direct interaction between researcher and respondents • flexible - can be used for many purposes in many settings • good for obtaining data from children or those who are not literate • provides an opportunity to involve people in data analysis through open recording • produces results which are easy to understand • is often useful and enjoyable for participants
Limitations of Focus Groups • convenience sampling and small sample size limit ability to generalize • responses of participants are not independent • a few members can dominate the results • focus groups require a fair bit of skill to lead well • the data which result, though rich, can sometimes be difficult to analyze because they are so unstructured
Tips for Writing Good Survey Questions • Stay Focused: Know why a survey is the right tool for a job, know the questions you want the survey to answer, and know how you hope to act on the findings. • Provide Feedback: If you let people know about generated from a survey, they are more likely to participate next time • Pilot Test!
Tips for Writing Good Survey Questions • Include a good mix of question types. Open-ended questions which provide some direction for people (such as, what are some of the things you like best about the program?) tend to elicit more responses than completely open questions like “please include any additional comments here”
Tips for Writing Good Survey Questions • Keep the text of the survey short, but lay it out in an easy-to-read way. • Avoid Leading Questions: • How much do you think we should increase our evaluation budget? • Avoid Double Negatives: • Would it be better not to avoid increasing our evaluation budget, yes or no? • Avoid “War and Peace” Questions: • Please describe your idea for a new management and accountability structure for our organization.
Organizing a good satisfaction survey • Provide an introduction that explains the purpose • Start by asking for basic descriptive information about people’s experience with the program (e.g., how long have you been coming? How did you find out about us?) • Move to questions about changes in knowledge (what did you learn? How much do you feel you know about x?)
Organizing a good satisfaction survey • Proceed to questions about behaviour & about whether they have used what they have learned • Satisfaction and critical reflection questions work well next: (What was most useful? Would you do it again?) • Personal information at the end
Suggested Steps in Data Analysis 1. Organize Data 2. Review Original Questions 3. Summarize and Code Centre for Research & Education in Human Services
Steps in Data Analysis 4. Generate Themes 5. Begin Writing 6. Provide and Receive Feedback Centre for Research & Education in Human Services
Creative ideas for extracting the most out of data: • Natural time series or comparison group designs • Data from similar organizations • Don’t forget the qualitative! • Be selective and speak to the questions you can speak to • The power of triangulation
Rapport Youth Services A Case Study in Analysis of Existing Data
The Context • Large youth counselling agency • Long history (11 years) of using a client tracking database to keep track of who was served, what kind and how much service, etc. • Also included data from a client satisfaction form
Program Activities Administration and Management Agency Management/Administration ▪ Recruiting staff ▪ Training staff ▪ Supervising staff ▪ Gauging comm. needs ▪ Developing programs ▪ Regulating case flow ▪ Securing resources ▪ Developing policies Clinical Administration ▪ Case preparation ▪ Research ▪ Session planning ▪ Case documentation Program Supports Community Relations Programming • Counseling • Therapy • Develop therapeutic • relationships • Help youth to build • their strengths • Promote protective • factors • Address risk factors • Teach skills e.g. • Stress reduction • Anger man’gmnt • Communication • Support parents • Develop healthy • relations within • social networks • (rapport) • Assist youth with • school difficulties • Enable youth to • make good choices • Groups • Provide safe setting • for youth to meet • together • Promote protective • factors • Address risk factors • Teach skills e.g. • Stress reduction • Anger managem’t • Communication • Provide information • Support parents • Develop healthy • relations within • social networks • (rapport) • Enable youth to • make good choices • ECLYPSE • Provide multi- • service under one • roof (Drop-in) • Reach out to hard • to reach students • Introduce youth • to services • Identify crises • Provide technical • support • Promote • Mental health • Supportive relations • Harm reduction • (drugs/alcohol use) • Safe sexuality • Provide employment • support • Intake/ • Assessment • Assess clients’ • needs • Determine • appropriateness • of cases to • Rapport’s mandate • (goodness of fit) • Referral • Refer clients to • appropriate • services • Provide information • about other services • Community • Connections • Network with • community • Market and • promote services • Promote • partnerships Rapport Youth & Family Services:Details of Program Activities Slide 1
Program Activities Program Supports Community Relations Programming Intake/ Assessment Referral Community Connections ECLYPSE Counseling Therapy Groups Rapport Youth &Family Services:Linking Activities to Short-term and Long-term Objectives Outreach to community Short-term Objectives/ Outcomes Connect/reconnect youth to community services Promote protective factors Address risk factors Connect clients to appropriate services Reduce service duplication Understand clients/family situations Build credibility Behavioural Outcomes Social Outcomes Increased use of healthy coping mechanisms Increased communication skills Increased healthy relations within families Increased individual self- worth Increased anger management skills Increased stress reduction skills Raise awareness of services Slide 1 Youth making better choices Increased healthy behaviors Empowered Youth and Families Increased pro- social relations Increased social/emotional well being Increased school success among youth Decreased CAS/YJS involvement Long-term Outcomes Engaged happy youth Youth who are successful in life Healthier Communities
Our approach • Developed a logic model • Spent time cleaning and organizing data and moving it from various formats into one spreadsheet. • Conducting basic descriptive analyses • Presented these to staff, and went back to do more in-depth analyses
Some of the Challenges • Who counts as a “case”? What if someone comes back to the organization for more support 4 years later? Or participates in 4 different programs at once? • How do you deal with changes in how the staff made use of the database over time (e.g., geography fields)? • What to do when the satisfaction form (the only source of “outcome” data) isn’t filled out by very many people?
Findings About Presenting Issues • About 41% of all cases presented were conduct issues, • 26% were family, peer or relational issues • 16% were anxieties, depression or emotional issues. • Male clients were more likely to present conduct issues • Female clients were more likely to present family, peer and relational issues, and anxieties, depression and emotional issues. • An analysis of presenting issues by geographical area revealed a similar pattern for Rapport’s three main geographical areas.
Findings About Outcomes • Clients who completed the follow-up survey experienced improvements in family dynamics, fighting in the home, and interactions at school. • 99.5% of clients who completed the client satisfaction questionnaire said they were satisfied with Rapport’s services, 93% said they received the services they needed, and 95% said the services they received helped them to better deal with their problems. • 90% of clients rated Rapport’s services as good or excellent, 93% of them indicated that they would return to Rapport if they needed help, and 95% said they would recommend Rapport if a friend needed help.
Criteria for Judging the Trustworthiness of an Evaluation • accurate recording in the field • prolonged engagement - knowing the context takes time • peer checks - members of a research team • debrief and challenge one another • saturation - qualitative information is rich and detailed enough to ensure that key themes have not been missed
Criteria for Judging the Trustworthiness of an Evaluation (continued) • well established "audit trail" to track where you drew information from to reach conclusions • triangulation - findings are supported by multiple methods and multiple stakeholders • endorsement of participants - findings are fed back and validated by stakeholders • clear program description and statement of objectives • links between exploration and confirmation
Communication: Questions to Consider • How much interpretation do you want to do (how much do others do)? • Who should act after learning about the findings? • What do you want them to do with the findings? • How do you package information in order to facilitate this?
Discussion Continued What are the emerging first steps for you?
