Overview of Outcome Measures Online Toolbox

1. afterschooloutcomes.org Overview of Outcome Measures Online Toolbox Deborah Lowe Vandell, PhD. Principal Investigator Professor and Founding Dean School of Education University of California, Irvine Project Researchers: Valerie Hall, Ph.D. PilarO’Cadiz, Ph.D. Andrea Karsh, M.A. afterschool@uci.edu

2. Background and Rationale for Online Toolbox • Research indicates high quality afterschool programs can have beneficial effects on students both socially and academically • The Afterschool Outcome Measures Project at UC Irvine developed an Online Toolbox in response to the State’s legislative mandate for alternative accountability measures to standardized test scores for publically funded afterschool programs in California* • Online Toolbox includes reliable and valid measures of Positive Behavior Change and Skill Development for use by diverse afterschool programs • Field Test of online toolbox at over 200 afterschool and summer learning program sites • Online Toolbox Statewide Implementation in Fall 2012/ Spring 2013

3. Development and Testing of Outcome MeasuresTHREE PHASES PHASE 1—INSTRUMENT DEVELOPMENT (2008-2009) • Literature review • Survey of the field (expected outcomes and current practices) • Selection of measures with input from the CDE and Accountability and Outcomes Subcommittee (representation from diverse programs) PHASE 2—PILOT STUDY (2009-2010) • Pre-piloting at 9 sites to test procedures and instructions & to determine feasibility of online administration of surveys • Piloting at 28 sites to test protocols under varying conditions: • Range of program size, geographic location, and level of operation • Sites with/without internet access PHASE 3—FIELD STUDY (2010-2011) • Test of measures & online administration across the State of California • 196 field test sites representing diverse program sizes and populations served by 29 programs in 9 regions statewide

4. NEXT PHASE: Statewide Implementation of Online Toolbox for California ASES & 21stCCLC Programs • Opportunity to participate open to all publically funded afterschool programs in California: • Afterschool Education and Safety Program (ASES) • 21st Century Community Learning Center (21stCCLC) • Up to 1000 sites statewide in 2012-2013 • Up to 2000 sites to participate by 2014 • Representation of diverse programs across 11 regions in the state

5. Outcome Measures Online Toolbox • WHAT MEASURES ARE IN TOOLBOX ? • WHO TAKES THE SURVEYS? • HOW DO PROGRAMS ADMINISTER SURVEYS? • WHEN DO ASSESSMENTS TAKE PLACE? • WHY DO IT?

6. Overview of Outcome Measures • Web-based surveys of student performance collected from three respondents: • Students (grades 3-12) • Program staff • Classroom teachers • PRE and POST Surveys collected at TWO time points, for example: • Time 1 (early Fall/Winter) & Time 2 (Spring) • Beginning of Summer & End of Summer • Technical Assistance provided through: • Online Instructions and Resources • Email and Telephone Communication • Individualized reports are provided to all participating programs. • These reports enable programs to compare their students’ skills and behaviors over time, and to compare their program and site-level outcome scores to aggregate scores across the State.

7. Description of Measures Programs have option to administer one, two, or three surveys: (1) Student (grades 3-12); (2) Program Staff; (3) Classroom Teachers PROGRAM STAFF & CLASSROOM TEACHER REPORTS —of Student Performance Skill Development • Work Habits • Task Persistence • Academic Performance (classroom teachers only) Positive Behavior Change • Reductions in Aggressive Behaviors • Prosocial Behavior with Peers • Social Skills Student Self-Reports Skill Development • Work Habits • Reading Efficacy • Math Efficacy • Science Efficacy Positive Behavior Change • Social Competencies • Reductions in Misconduct Program Quality (post only) • Affiliation with Peers • Relations with Program Staff • Interest in Program Activities

8. Field Test Key Findings • The Online Toolbox is an easy-to-use, inexpensive method of assessing skill development and positive behavior. • Elementary and middle school students reported having high quality experiences at their ASES programs. • Quality of experiences at ASES programs is linked to students’ skill development and positive behaviors. • Significant improvements in student performance were found between the fall and spring assessments.

9. SIGNIFICANT FINDINGS 2010-2011 Field Test PRE-POST OUTCOMES in MATCHED SAMPLE Staff Reports of Elementary Students • Increase in Prosocial Behavior ** Staff Reports of Middle School Students • Increase in Prosocial Behavior ** Teacher Reports of Elementary Students • Increase in Academic Performance ** • Decrease in Aggressive Behavior** Student Reports of Quality Program Experience • Positive student reports of program experience linked to positive results in ALL student outcomes ** p<.01

10. SIGNIFICANTFINDINGS Staff Reports on Student Prosocial Behavior Increase in pre-post scores—matched sample MiddleSchool Students Elementary School Students

11. SIGNIFICANT FINDINGS Teacher Reports on Elementary Students Improvements in Academic Performance Reductions in Aggressive Behavior

12. Program Quality Distribution Staff & Activities Peer Affiliation

13. Quality to Outcomes Student reports of higher quality experiences with staff & activities are linked to greater Math Efficacy Student reports of higher quality experiences with peers are linked to less Aggressive Behavior

14. How does it work? EASY TO ADMINISTER ONLINE SURVEYS Step-by-Step Instructions available to download from website http://afterschooloutcomes.org/

15. Sign-up & Instructions on Website To apply to participate in Statewide Implementation Click Here To download Instructions for Administering Surveys Click Here

16. Program Staff & Teacher SurveysReports on Students in Afterschool Program • Program Staff and Classroom teachers are assigned individual access codes • Use both individual access code and the ID code for each student for whom they complete a report • Each student report takes 3-5 minutes to complete. • Student reports may be completed in multiple sessions. For example: A staff completing reports for 20 students can do four reports a day (a 20-minute session) on five different days.

17. Why the Online Toolbox? • Provides a battery of valid, reliable, and easy to administer youth outcome measures • Expands accountability tools beyond standardized test scores and other academic measures • Value in tracking individual student performance over time with matched pre-post surveys, providing more powerful & interpretable data • MATCHED = Respondent data are matched for both pre and post survey administrations (i.e. same students, staff and/or teachers in fall and spring) • Information can be useful to guide program improvement based on student reports of program experience AND on student performance

18. For more information on measures: • Forum for Youth Investment publication(September 2011) • Review of measures in Toolbox • with a technical appendix • Down Load at: • forumfyi.org • Or, link from our website: • http://afterschooloutcomes.org/

19. Contact Info: EMAIL: afterschool@uci.edu