1 / 28

CECC Webinar Series: Measuring Program Outcomes

Join Terry Tolan and John Roden as they introduce a system for measuring your CECC's outcomes and how it can increase program effectiveness and communicate value.

javen
Download Presentation

CECC Webinar Series: Measuring Program Outcomes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CECC Webinar Series:Measuring Program Outcomes Presented by Terry Tolan and John Roden

  2. Purpose To introduce you to the concept and a system for measuring your CECC’s outcomes.

  3. What is Outcome Measurement? Outcome Measurement is the regular and systematic measuring of progress toward intended outcomes in order to: • Increase the effectiveness of programs and services • Communicate the value of those programs and services

  4. Our Business Model Children Enter Kindergarten Ready High Quality Early Learning Environments Supportive Families Access to Data Common Kindergarten Entry Screener Families understand child health and developmental needs Participation in STARS A great early childhood workforce Families are engaged Children have access to appropriate services Data is shared by early childhood programs Scholarships & PD Plans

  5. What CECCs Measure

  6. Program Outcome Model What we use What we do What we count

  7. Program Outcome Model How THEY change! What we use What we do What we count

  8. Outcomes Logic Model – A System of Measurement SITUATION INPUTS ACTIVITIES OUTPUTS OUTCOMES INTIAL INTERMEDIATE LONG-TERM EXTERNAL INFLUENCES, ENVIRONMENTAL, RELATED PROGRAMS

  9. Parenting Education Program • Parents from 10 families attend the workshops • Six group workshops are conducted • Parents’ understanding of children’s developmental issues increases • Parent provide more age appropriate guidance to children

  10. Inputs Through Outcomes: The Conceptual Chain Input Activity Output Outcomes Inputs Through Outcomes: The Conceptual Chain Initial Intermediate Long-Term

  11. Logic Model Framework Children Enter School Ready

  12. Long-term OUTCOMES Intermediate Intermediate Initial ACTIVITIES INPUTS

  13. Figuring Out Our Outcomes • What do we want to be true of participants because of their involvement in the program? What do we want to be able to say about them? • If we succeed with a participant (or don’t), what has changed (or hasn’t)? • If we conduct the activity, then what do participants believe, know, have, or do as a result? And what benefit or change follows that?

  14. Program Outcome Criteria for EachOutcome • Is it reasonable to think the program can influence the outcome in a non-trivial way even though it can’t control it? • Would measurement of the outcome help identify program successes and pinpoint problems? • Will the programs various “publics” accept this as a valid outcome of the program?

  15. Outcomes Logic Model SITUATION INPUTS ACTIVITIES OUTPUTS OUTCOMES INTIAL INTERMEDIATE LONG-TERM EXTERNAL INFLUENCES, ENVIRONMENTAL, RELATED PROGRAMS http://www.uiweb.uidaho.edu/extension/LogicModel.pdf

  16. Outcomes vs. Indicators Outcome: Benefits for participants • Teens follow proper nutrition and health guidelines Indicator: The specific information that is tracked to indicate success in achieving the outcome • Proper weight • Does not smoke • Takes a prenatal vitamin

  17. Decision Tree

  18. Sample Indicators Activity Outcome Indicator

  19. Data Gathering

  20. Outcomes Logic Model: Parents Reading Daily to Children Only 25% of Children Enter School Ready Planning Facilities Books Printing Costs Literacy Training Book Distribution Demonstration Number of Books Distributed Number of Parents Attending Training Number of Brochures Mailed OUTCOMES Gained Knowledge of Importance of Reading Awareness of Role/Impact Intentionality of Modeling Increase number of times child read to daily/weekly Increase number of parents reading ChildrenEnter School Ready EXTERNAL INFLUENCES, ENVIRONMENTAL, RELATED PROGRAMS *Linking Parents to GED, Adult Education, and ELL Programming* http://www.uiweb.uidaho.edu/extension/LogicModel.pdf

  21. Gathering Data

  22. This checklist can help you decide which data collection methods are most appropriate for your outcome measurement. SURVEYS 1. Do I need data from the perspective of the participant, client, beneficiary, or customer? 2. Do I have a systematic way to get it from these individuals? 3. Do I need data that are standardized so that statistical comparisons can be made? (For example, will I need to report percents or other statistics?) 4. Will participants be able to understand the survey questions? (Consider age, cultural backgrounds, etc.) 5. Do participants have the necessary knowledge or awareness to accurately answer questions about the outcomes? Check List for Selecting Data Collection Methods http://www.strengtheningnonprofits.org/resources/guidebooks/MeasuringOutcomes.pdf

  23. Don't ask a question if the answer is obvious. • Avoid abbreviations and jargon. If they must be used, clearly define them. • Ask yourself whether several questions are actually necessary or if you can get the information in one question. Don't try to cram too much into one question. • Make your questions easy to understand. Make sure your sample population understands them. • If a list of answers is provided, make sure all possible answers are present. Even with "yes" and "no" questions, it may be necessary to include a neutral "undecided" or "don't know." • Don't mix "I feel" or "I think" questions with questions regarding facts. Keep factual and perception questions in separate groupings. • Place sensitive demographic questions (such as age or income) at the end of the survey. General Guidelines for Survey Questions http://www.mad.state.mn.us/survey-guide

  24. Sometimes it takes just one word to bias a question. Avoid using inflammatory words in surveys, such as:allege, allude, arbitrary, blame, claim, demand, error, failure, fault, ignore, ill-advised, ill-informed, incompetence, ineptness, insist, just, maintain, misinformed, must, neglected, one-sided, only, overreact, peremptory, purport, questionable, rejection, rigid, so-called, unfortunately, unilateral, unreasonableValue-laden questions, especially those that attempt to be global in scope, tend to overwhelm respondents. For example, making respondents choose between a healthy environment and a vital economy will probably bias results. Don't distill complex issues into "black" or "white" scenarios. Rather, explore the "gray" areas. Avoid “Red Flag” Words http://www.mad.state.mn.us/survey-guide

  25. Questions to Ask About Questions • Is the question relevant? Is it consistent with survey goals? • Does the question ask for "need to know" or "nice to know" information? • What will be the value of a response? If 95 percent say, "Yes," would this affect decision making? • Will respondents be able to answer the question? Will they have the information? • Does the question lead to a particular response? (Is it a leading question?) • If a set of answers is provided, are all possible answers listed? Is one side of the issue represented more than another? • Does the question use negative phrases or words? Are positive adjectives or phrases used? • If a scale is used for responses, is it balanced (for example, 1 to 5, with 3 being neutral)? • Are "dead giveaway" words used, such as "all," "every," or "always"? • Is the question wordy? • Were ambiguous words used - words with more than one meaning? • Is the question worded simply? http://www.mad.state.mn.us/survey-guide

  26. Outcomes Logic Model: Parents Reading Daily to Children Only 25% of Children Enter School Ready Planning Facilities Books Printing Costs Literacy Training Book Distribution Demonstration Number of Books Distributed Number of Parents Attending Training Number of Brochures Mailed OUTCOMES Gained Knowledge of Importance of Reading Awareness of Role/Impact Intentionality of Modeling Children being read to daily/weekly Parents reading to children routinely ChildrenEnter School Ready Measurement of Process Indicators: Number of Trainings Held/Number of Attendees Number of Books Distributed Number of Demonstrations held/Number of Parents attending Events (Mostly Counting: Does not truly measure Impact) Measurement of Outcomes Indicators: Pre-Post Survey of gained knowledge/awareness of parent impact Increase in the number of children being read to daily Increase in the number of parents reading to their children (Measuring Change: Change in Behavior/Practices) http://www.uiweb.uidaho.edu/extension/LogicModel.pdf

  27. http://www.mad.state.mn.us/survey-guide • http://www.strengtheningnonprofits.org/resources/guidebooks/MeasuringOutcomes.pdf • http://www.uiweb.uidaho.edu/extension/LogicModel.pdf Resources

  28. Happy Measuring! Terry Tolan 502-782-0200 terry.tolan@ky.gov John Roden 859-623-7233 john.roden@headstart1.org

More Related