Program Quality & Youth Outcomes

1. Program Quality & Youth Outcomes A Ready by 21® Quality Counts Webinar March 13, 2008 The Forum for Youth Investment

2. Submit a Question at Any Time! • To ask a question, go to the “Question and Answer” box in the upper right-hand corner of your screen. Click on the small arrow next to “Question and Answer,” type in your question, and click send.

3. Agenda • Welcome, Review Goals, Speakers, Logistics - Nicole Yohalem • Making the Case for Quality - Karen Pittman • Population Level Outcomes & Indicators - Karen Finn • Program/System Level Quality & Outcomes - Charles Smith • Summary Reflections - Karen Pittman • Question and Answer Period • Closing, Review Key Resources – Nicole Yohalem

4. Why is Program Quality the Focus of this Initiative? Making the Case for Quality Karen Pittman The Forum for Youth Investment

5. Improving Community Supports is the intermediate step toward improving Youth Outcomes. While improving youthoutcomes is the goal we are all working toward, we believe that to get there at scale, our focus needs to be on helping leaders improve the quality and reach of programs and services available in the community.

6. Increasing program availability isn’t enough. • There is basic agreement that young people need programs and services – in particular that they need more structured, voluntary activities to fill their out-of-school hours. • There is a basic belief, and evidence, that participation in these programs is associated with improved youth outcomes – academic, social, emotional, civic, health, management of risky behaviors. • There is, consequently, growing public and political will, to increase youth participation in programs and to ensure that the programs are of good quality. • There is a growing base of research that suggests that improvement in youth outcomes is strongly related to level of participation and level of program quality – more refined measures of both are needed. • Measuring quality at the system level has always been challenging.

7. What the research says. • “Dosage and duration” matter. Young people who participate regularly in programs over a sustained period of time show gains in targeted outcomes. Those who attend sporadically don’t. • Program quality also matters. Controlling for participation, young people who participate in high quality programs achieve greater gains than those who do not. • Recent meta-analysis by Durlak and Weissberg suggests that low-quality programs may have no effect on adolescent outcomes.

8. Durlak and Weissberg study • Recent meta-analysis by Durlak and Weissberg suggests that low-quality programs may have no effect on adolescent outcomes. • 73 after-school programs reviewed grouped into 2 clusters based on SAFE criteria: • Sequenced set of activities to achieve goals • Active learning techniques for skill acquisition • Focus, to some extent, on personal/social development • Explicit objectives for personal and/or social skill development • Programs that had the SAFE features showed positive effects on almost every outcome – school performance, social behavior, attitudes and beliefs. • Programs that did not have the SAFE features showed no effect on any outcome.

9. Implications for Programs, Public and Policy Makers • Quality counts as much as participation. • Two youth, spending equal time in two programs of different quality will reap significantly different benefits. • Overtime, the cumulative effect of participation in low quality programs, especially when coupled with participation in low quality schools and harried families, can be huge. • Improving program quality at the point of service – where youth and adults interact – has the biggest impact on youth outcomes. • But program quality is either not measured, not measured well or not measured consistently across programs. • This means that increasing funds for program expansion and participation, without increasing funding and monitoring for quality, may net mediocre results.

10. Program/SystemLevel Performance QUALITY How well do we do it? PROGRAM OUTCOMES Is anyone better off? PARTICIPATION How much do we do? Frequency, duration, intensity of participation Point of Service Quality Improved skills, knowledge & behaviors As measured by indicators e.g., % with improved decision-making skills POS Quality Meets System Performance Measurement • Results-based accountability helps program/system directors answer 3 questions: How much do we do? How well do we do it? Is anyone better off? • These 3 questions can be easily linked to the concepts of participation, POS quality, and youth outcomes.

11. Defining Terms POPULATION LEVEL Outcomes or Results: A condition of well-being for children, adults, families or communities (e.g. youth ready for college, work, life). Indicators: Measures that help quantify achievement of a result or outcome (e.g. rate of high school graduation, teen birth rates, employment rates). PROGRAM/SYSTEM LEVEL Performance Measurement: Measuring how well a program, organization or system is working: • Participation. Measuring frequency, duration and intensity of program attendance. • Program Outcomes: Impacts or changes for participants during or after a program, often expressed in terms of knowledge, skills or behaviors. • Program Quality: Youth access to developmentally significant experiences in a setting. Although program quality can be defined more broadly, point of service quality refers primarily to staff practices during specific activities with children and youth. Based on Mark Friedman’s Results-Based Accountability

12. Defining Terms Point of Service Quality: Youth access to developmentally significant experiences in a setting. Although program quality can be defined more broadly, point of service quality refers primarily to staff practices during specific activities with children and youth.

13. Program/SystemLevel Performance QUALITY How well do we do it? PROGRAM OUTCOMES Is anyone better off? PARTICIPATION How much do we do? Frequency, duration, intensity of participation Point of Service Quality Improved skills, knowledge & behaviors As measured by indicators e.g.,% with improved decision-making skills FAMILY & COMMUNITY CHILD & YOUTH LEVEL GOALS POPULATION OUTCOMES or RESULTS SUPPORTS or INPUTS How is our community doing? How are our young people doing? e.g. Caring Adults Safe Places Effective Ed Healthy Starts Opportunities to Contribute As measured by indicators e.g., % with caring adult(common resources: Search external assets) e.g. Connecting Thriving Leading Working Learning As measured by indicators e.g., Teen birth rate (common resources: Kids Count; Search internal assets)

14. The Maryland Ready by 21 Action AgendaKaren Finn The Forum for Youth Investment

15. Babies born healthy Healthy Children Children enter school ready to learn Children are successful in school Children complete school Children are safe in their families and communities Stable and economically self-sufficient families Communities that support family life Maryland’s Child Well-Being Results

16. “They need to craft programs around older youths’ interests….bring in adults in the same field of interest and partner them with youth…” “….foster parents and youth workers…..show them how to connect with kids….” Youth Told Us…..

17. Maryland’s Ready by 21 Action Agenda POPULATION LEVEL GOALS ACTION AREAS DESIRED RESULTS How is our community doing? How are our young people doing? Competent, Caring Adults Accessible, Affordable Housing Access to Health Care Pathways to Education and Employment Equal Treatment Under the Law Ready for College, Work & Life As measured by indicators e.g., educational attainment, employment, health care coverage, HIV/AIDS rate… Program/System Level Performance QUALITY How well do we do it? PROGRAM OUTCOMES Is anyone better off? PARTICIPATION How much do we do? Frequency, duration, intensity of participation Point of Service Quality Improved skills, knowledge & behaviors

18. Quality & Outcomes at the Program/ System Level Charles Smith The Forum for Youth Investment

19. POS Quality and Program Outcomes: OST Settings Durlak & Weissberg, 2007 SAFE NRC Blue Book, 2002 “research specifically linking…[setting] features to outcomes is rare…”, that “we are only beginning to understand how the different combinations of features in organized activities interact to promote positive development…”, and finally that “precisely which features are involved and how they co-act to produce specific developmental change has not yet been evaluated” (Mahoney et al., 2005, p. 12-13). Robert Halpern

20. POS Quality and Program Outcomes: Classroom Settings Research on Motivation: Relatedness, autonomy, and competence Research on Learning: Positive affect, active learning and meta-cognition

21. What is actually happening in 599 OST settings?Does prior research make sense to you? Reflect Plan Choose Engagement Encouragement Skill building Active engagement Be in small groups Interaction Supportive Environment Reframing conflict Experience belonging Welcoming atmosphere 5 = occurred for everyone 3 = occurred for some 1 = did not occur Safe Environment

22. Findings from Several Samples POS quality findings: Supportive environment related to: Attendance Interaction related to: Interest in program Engagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspension Note: No offerings get to high engagement without high support and high interaction Quality Improvement (YPQI) Findings

23. Are the Youth PQA and YPQI Improvement Model RESEARCH BASED? YES – The Youth PQA has been the subject of lots of research can be considered a reliable and valid measurement instrument by widely shared standards of evaluation YES, conditional – The YPQI sequence has been tested in at least one high fidelity implementation and use of the YPQI sequence is related to POS quality change IN PROCESS – The YPQI sequence is currently the subject of a randomized field trial, results available in fall 2008 Fidelity to broad developmentally focused intervention models

24. How to Talk about Outcomes OST programs can produced positive change in social-emotional skills and academic achievement for both at-risk and lower-risk populations of children and youth. Not all OST programs produce positive change so we should be careful to protect investment by doing quality assurance. It’s cost effective to ensure that staff are actually doing good stuff. Programs can raise child outcomes by striving to improve the quality of their offerings. Broad developmentally-focused programming is a research-based strategy.

25. Question & Answer Period • To ask a question, go to the “Question and Answer” box in the upper right-hand corner of your screen. Click on the small arrow next to “Question and Answer,” type in your question, and click send.

26. Program Quality & Youth Outcomes A Ready by 21 Quality Counts Webinar March 13, 2008 The Forum for Youth Investment

27. Key Readings Population Level • Trying Hard is Not Good Enough (Mark Freidman) www.resultsaccountability.com • Finding Out What Matters for Youth (Gambone, Klem & Connell, 2002) www.ydsi.org/ydsi/pdf/WhatMatters.pdf • Maryland’s Ready by 21 Five-Year Action Agenda www.ocyf.state.md.us/ • Forum Focus: What Gets Measured Gets Done www.forumfyi.org/node/149 • Steering a Course Toward Effective Youth Policies: Dashboards for Youth www.forumfyi.org/node/68 Program/System Level • YPQA validation study (Smith & Hohmann, 2005) www.highscope.org/Content.asp?ContentId=308 • Afterschool Quality and School-Day Outcomes (Blazevski & Smith, 2007) www.highscope.org/Content.asp?ContentId=308 • Improving Afterschool Program Quality (Granger, Durlak, Yohalem, Reisner, 2007) www.wtgrantfoundation.org/usr_doc/Improving_After-School_Program_Quality.pdf • The Impact of After-school Programs…(Durlak & Weissberg, 2007) www.casel.org/downloads/ASP-Full.pdf • Participation During Out-of-School Time (Forum for Youth Investment, 2004) www.forumfyi.org/node/80

28. Program Quality & Youth Outcomes A Ready by 21 Quality Counts Webinar March 13, 2008 The Forum for Youth Investment