California Afterschool Outcome Measures Project

1. California Afterschool Outcome Measures Project afterschooloutcomes.org Overview of Online Toolbox UNIVERSITY OF CALIFORNIA, IRVINE PRINCIPAL INVESTIGATOR: Deborah Lowe Vandell RESEARCHERS: Pilar O’Cadiz, Valerie Hall and Andrea Karsh CAOMP Email: afterschool@uci.edu

2. Background and Rationale for Online Toolbox • Research indicates high quality afterschool programs can have beneficial effects on students both socially and academically • The California Outcome Measures Project (CAOMP) developed an Online Toolbox in response to the State of California’s legislative mandate for alternative accountability measures to standardized test scores for it’s publically funded afterschool programs* • Toolbox includes reliable and valid measures of Positive Behavior Change and Skill Development to be used as an accountability measure by California’ State funded programs • Field Test of measures using CAOMP Online Toolbox carried out at over 200 afterschool and summer learning program sites concluded in Fall 2011 • Online Toolbox available for general use in Spring 2012 *In partnership with UC Davis and with support David and Lucile Packard Foundation & California Department of Education

3. Development and Testing of Outcome MeasuresTHREE PHASES • PHASE 1—INSTRUMENT DEVELOPMENT (2008-2009) • Literature review • Survey of the field (expected outcomes and current practices) • Selection of measures with input from the CDE and Accountability and Outcomes Subcommittee (representation from diverse programs) • PHASE 2—PILOT STUDY (2009-2010) • Pre-piloting at 9 sites to test procedures and instructions & to determine feasibility of online administration of surveys • Piloting at 27 sites to test protocols under varying conditions: • with/without researchers present • with/without internet access • range of program size, geographic location, and level of operation • PHASE 3—FIELD TEST (2010-2011) • Test measures & online administration across the State of California • 28 programs recruited statewide from 9 regions • 193 field test sites representing diverse program sizes and populations served

4. FIELD TEST (2010-2011) Research Aims & Design AIMS • Assess the feasibility of a low cost and easy to administer 3rd party assessment of student skill development and positive behavior change to be used by ASES programs in reports to the CDE • Develop an understandable and helpful reporting system of student performance that can be used by programs and sites to inform their planning and program improvement DESIGN • Web-based surveys of student performance collected from three respondents: (1) students, (2) program staff, & (3) classroom teachers • Surveys collected at two time points: Fall 2010 & Spring 2011 • Participation open to all ASES programs in the State • Technical Assistance provided to programs by email and telephone • Programs receive reports of their results to compare to statewide data

5. CAOMP Outcome Measures Programs have option to administer one, two or all three surveys: (1) Student, (2) Program Staff, and/or (3) Classroom Teachers Student Self-Reports—Elementary & Middle (5-6 subscales) Skill Development • Work Habits • Reading Efficacy • Math Efficacy Positive Behavior Change • Social Competencies • Misconduct Program Quality (spring only) • Staff & Activities • Peer Affiliation PROGRAM STAFF & CLASSROOM TEACHER REPORTS —of Elementary & Middle School Student Performance (5-6 subscales) Skill Development • Work Habits • Task Persistence • Academic Performance (classroom teachers only) Positive Behavior Change • Aggressive Behavior with Peers • Prosocial Behavior with Peers • Social Skills

6. FIELD TEST (2010-2011) SURVEYS COMPLETED • NUMBERS COMPLETED • 10,973 students completed 13,623 surveys (7,744 elementary; 3,229 middle school surveys) • 2,650 students completed surveys in both the fall and the spring Total Number of Pre & Post Teachers, Staff and Students Surveys Completed • REPORTS ON SURVEY TAKING EXPERIENCE • The overwhelming majority of Students, Staff, and Teachers reported surveys were easy to read, easy to understand and easy to answer and tookless than 10 minutes to complete.

7. Program Quality Distribution Staff & Activities Peer Affiliation

8. SIGNIFICANT FINDINGS From 2010-11 Field Test PRE-POST OUTCOMES in MATCHED SAMPLE Staff Reports of Elementary Students • Increase in Prosocial Behavior ** Staff Reports of Middle School Students • Increase in Prosocial Behavior ** Teacher Reports of Elementary Students • Increase in Academic Performance ** • Decrease in Aggressive Behavior** Student Reports of Quality Program Experience • Positive student reports of program experience linked to positive results in ALL student outcomes ** p<.01 * p<.05

9. SIGNIFICANT FINDINGS Staff Reports on Student Prosocial Behavior Increase in pre-post scores—matched sample Elementary School Students Middle School Students

10. SIGNIFICANT FINDINGS Teacher Reports on Elementary Students Academic PerformanceIncrease in pre-post scores—matched sample Aggressive Behavior Decrease in pre-post scores—matched sample

11. Quality to Outcomes Student reports of higher quality experiences with staff & activities are linked to greater Math Efficacy Student reports of higher quality experiences with peers are linked to less Aggressive Behavior

12. Why Use the Online Toolbox? • Expands accountability tools beyond standardized test scores and other academic measures • Online toolbox is a feasible means of assessing student outcomes across diverse out-of-school time program sites • Measures tested at over 200 afterschool and summer learning program sites • Found to be reliable, valid and easy to administer • Currently includes: Skill Development, Positive Behavior Change & Program Experience • Process of testing measures of STEM (Science, Technology, Engineering and Math) learning in afterschool • Menu of measures that match a range of student outcome goals for diverse out-of-school time programs will be made available • Considerable value exists in tracking individual student performance over time—with matched pre-post surveys—providing more powerful & interpretable data • Respondent data is matched for both pre and post survey administrations (i.e. same students, staff and/or teachers in fall and spring; or beginning and end of summer) • Information can be useful to individual programs and sites to guide program improvement based on student reports of program experience AND on student performance

13. How does it work? EASY TO ADMINISTER Step-by-Step Instructions Available to download from CAOMP Website afterschooloutcomes.org Updated Website to Launch January 2012 For information about using the Online Toolbox email us at: afterschool@uci.edu Online Toolkit Available to public SPRING 2012