Accountability for Quality: Policy Innovation in Out-of-school Time Programs

1. Accountability for Quality: Policy Innovation in Out-of-school Time Programs November 2, 2007

2. Introduction • Why focus on quality? • Quality matters • Quality is measurable • Quality is malleable

3. Assessing Afterschool Program Practices Tool (APT) National Institute on Out-of-School Time and the MA Department of Education Out-of-School Time Observation Instrument (OST) Policy Studies Associates Program Quality Observation (PQO) Deborah Vandell and Kim Pierce Promising Practices Rating Scale (PPRS) WI Center for Education Research and Policy Studies Associates, Inc. Program Observation Tool (POT) National Afterschool Association Quality Assurance System (QAS) Foundations Inc. Program Quality Self-Assessment Tool (QSA) New York State Afterschool Network School-Age Care Environment Rating Scale (SACERS) Frank Porter Graham Child Development Center, UNC Youth Program Quality Assessment (YPQA) High/Scope Educational Research Foundation IntroductionA New Guide to Quality Metrics

4. IntroductionExemplar: The Youth Program Quality Assessment (YPQA) • YPQA is a good example of new class of tools that: • Produce data that leads to real change in staff/teacher performance • Provide continuity that is place-based, not silo-based • Link accountability policy with workforce development policy • Offer a more efficient and effective use of resources than a sole focus on measuring child outcomes

5. Part I.About the YPQA • Characteristics • Observation at point of service • Takes 1-2hours, training available but not required • Applies to across content areas • Purposes: • Metric for most important part of education and human service programs – point of service • Increase access to experiences that motivate clients to attend and engage • Staff learning and performance change • Foundation for more effective accountability

6. Make choices • Reflect • Plan Youth Engagement • Experience belonging • Partner with adults • Be in small groups • Lead and mentor Peer Interaction • Reframing conflict • Welcoming atmosphere • Skill building • Encouragement • Session flow Supportive Environment • Active engagement • Psychological & emotional safety • Program space and furniture • Emergency procedures • Physically safe environment Safe Environment • Healthy food and drinks Part II: How YPQA Works in ProgramsHigh/Scope YPQA Pilot Site Winter 2007 Youth Program Quality Assessment 5 North Branch Norm Scores* 1.33 2.53 2.17 2.96 4.01 3.78 4.92 4.41 * The norm column represents mean scores for approximately 635 offerings within 170 programs collected during the years 2001-2006.

7. Part II. YPQA in Programs • North Branch Area Community Education • Provide a comprehensive learning program for all ages; • Seek out human resources in our community; • Develop citizen advisory councils; • Strive toward minimum use of facilities

8. Part II: YPQA in ProgramsYouth Classes 2006-2007 We offered 178 youth classes in our Fall, Winter, and Summer sessions, having a total of 3482 participants.

9. Part II: YPQA in ProgramsNorth Branch Area SAC Program • Goals School Age Care (SAC) program will provide appropriate activities which will positively contribute to the total development of each youth in areas of: • personal discipline • social development • physical skills • health and safety skills

10. Part II: YPQA in ProgramsProgram, Plan and Staff Procedures Handbook • Program Curriculum Examples provided by State Department • Behavior Guidance and Procedure • Procedures for Tornado/Fire/Lock Down Drills • Accidents/First Aid • Staff Guidelines for Working with Children

11. Part II: YPQA in ProgramsSo what? Now what? “If you always do what you always did, then you will always get what you have always got.” Our youth need us to do better than we currently do. Program Quality Defined in North Branch: Atmosphere + Activities = Quality - But how do we get more specific?

12. Make choices • Reflect • Plan Youth Engagement • Experience belonging • Partner with adults • Be in small groups • Lead and mentor Peer Interaction • Reframing conflict • Welcoming atmosphere • Skill building • Encouragement • Session flow Supportive Environment • Active engagement • Psychological & emotional safety • Program space and furniture • Emergency procedures • Physically safe environment Safe Environment • Healthy food and drinks Part II: YPQA in ProgramsQuality Scores in North Branch Winter 2007 YPQA Scores 1.33 2.17 4.01 4.92

13. Part II: YPQA in ProgramsData-Driven Improvement Goals Activities support active engagement. (YPQA quality scale II-H) - Creative new science lessons to supplement current themes.

14. Part II: YPQA in ProgramsData-Driven Improvement Goals Youth have an opportunity to develop a sense of belonging. (YPQA quality scale III-L) Youth have an opportunity to participate in small groups. (YPQA quality scale III-M)

15. Part II: YPQA in ProgramsData-Driven Improvement Goals Youth have opportunities to act as group facilitators and mentors. (YPQA quality scale III-N)

16. Part II: YPQA in ProgramsChallenges and Opportunities • Staff and student habits are hard to change. • Time and funds for staff training on new strategies. • New lens through which to see our program. • Creative, purposeful planning. • Engaged kids!

17. Part III.YPQA in State Accountability Policy Context • Child Care Services School Age Child Care • County Youth Bureaus NYC Out of School Time • State Education Department 21st Century Community Learning Centers • TANF - Advantage After School Program

18. Part III.YPQA in State Accountability Policy • Three similar quality “trains” are moving in NY state, all focused on raising quality and building more effective forms of accountability: • NYSAN Quality Self Assessment • Accreditation for School Age Child Care • YPQA

19. Part III.YPQA in State Accountability Policy • Key roles for state policy • Set clear priorities about quality • Fund provider networks to support implementation of quality policy • Meet provider-level demand for more effective forms of accountability

20. Part III.YPQA in State Accountability Policy • How YPQA is used in Advantage After School Programs

21. Part IV. Scope of Use • YPQA is part of state and county accountability policies: • Cross sector (DHS& DOE) snapshots: IA, WA • Statewide 21st Century: MI, ME, MN, RI • Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach

22. Part IV. Scope of UseGeneric Acct Policy Elements Using Quality Metric Accountability Policy Elements: External Data Collection, Reporting & Data Analysis, Training, Coaching IMPROVEMENT Sequence STEP 2a Program Self-assessment STEP 3 Plan for improve-ment STEP 4 Targeted T&TA STEP 6 Re-assess and move forward! STEP 1 Decide to build system STEP 2b External assessment STEP 5 External assessment with criterion MONITORING & ACCOUNTABILITY Sequence

23. Part IV. Scope of UseLeveraging Impact from Existing “Change” Resources Professional Development Local Evaluators Accountability Measurement POS Quality Access to key developmental and learning experiences

24. Summary • The field is looking for more effective ways to spend resources that are already dedicated to accountability, evaluation, and professional development. Observational quality assessment metrics are increasingly the answer • Youth PQA (and tools like it) allow us to measure the most important elements of programs – and raises the possibility of increased returns on existing investments • With the right kind of supports, this form of accountability can be effective and sustainable part of programs that serve children and youth • Constituencies of state government want to move in this direction – want to be accountable in ways that actually and demonstrably change the quality of services available to youth • State and local governments are already experimenting with new accountability policies based on the youth PQA – and they are affordable