110 likes | 230 Views
Pilot Findings and Field Test: An Update. Presentation to the Before and Afterschool Advisory Committee California Department of Education January 11, 2011. California Afterschool Outcome Measures Project. Last report to the Advisory Committee —February 24, 2010
E N D
Pilot Findings and Field Test: An Update Presentation to the Before and Afterschool Advisory Committee California Department of Education January 11, 2011
California Afterschool Outcome Measures Project • Last report to the Advisory Committee —February 24, 2010 • Pre-Pilot activities and Key Findings: • Completed pre-pilot test of initial versions of measures and online administration • Measures performed well in online format; preliminary outcomes comparable to previous studies • Minor modifications made to measures and procedures based on pre-pilot findings • Changes to wording in items, scales removed and added • Survey administration instructions revised—clear step-by-step • Special instructions created to support ELL students and slow readers
Highlights of Today’s Report • Phase 2—Report on the Pilot Test • Creation of project website • Results of the spring 2010 pilot work • Phase 3—Update on the Field Test • Fall survey administration experience • Dissemination plans
Phase 2: PILOT PHASE • Project website created: afterschooloutcomes.org • Piloted online measures of positive behavior change and skill development: • 27 sites (13 in Southern and 14 in Northern CA) • 5 urban; 10 rural; 12 suburban • 18elementary; 6 middle school; 3 center-based serving K-8 • 10 regions across the state • Range of school/program sizes in Pilot sample: • School enrollments 225 to 1,963 • ASES enrollments 43 to 574 • Demographically diverse students (ethnicity, ELL and FRL)
Pilot DataNumber of Respondents & Time to complete Survey • 1,656 Students completed surveys • 1,175 Elementary; 481 Middle • On average less than 10 minutes to complete survey (observed) • 51% elementary students report they took less than 10 minutes • 61% middle school students report taking less than 10 minutes • 105 Staff completed 769 student reports • 54% report taking less than 5 minutes per student report • 46% report taking 5 to 9 minutes • 70 Classroom Teachers completed 535 student reports • 81 % report completing each student report in less than 5 minutes • 18% report taking 5-10 minutes per report • 1% took more than 10 minutes
FIELD TEST (2010-2011) Overview of key elements: • Participation open to all programs in the state • Outreach effort to all regions: • Presented on Field Test at Regional Lead Meeting (October 13, 2010) • Sent Field Test materials electronically to RLs for dissemination to all grantees • Web-based surveys for students, program staff, and classroom teachers completed in fall 2010 and spring 2011 • Programs select which surveys to administer • Confidential summary report of survey results • students’ scores of positive behavior change and skill development provided to participating programs • All materials and instructions available on Project website • Technical assistance provided to programs • Support provided by email and telephone during pre and post survey administrations & in interpreting their scores at the end of Field Test • Measures modified based on Pilot results and feedback from field
FIELD TEST (2010-2011)Outcome Measures • Student—Elementary & Middle School—(5 subscales) • Work Habits • Misconduct • Reading Efficacy • Math Efficacy • Social Efficacy • Staff (5 subscales) • Work Habits • Aggressive Behavior with Peers • Prosocial Behavior with Peers • Task Persistence • Social Skills • Teacher(6 subscales) • Same as Staff plus Mock Report Card
FIELD TEST FINDINGS (Fall 2010 – Spring 2011) • Diverse programs across the state participate in fall 2010 field test • 156 Sites (elementary & middle school) • 23 Programs (10 rural, 8 urban, 5 suburban) • 8 Regions represented • Pre-Surveys Completed (November 1 through December 17) • 3934 elementary students • 1847 middle school students • 360 program staff completed 4611 student reports • 29 classroom teachers completed 185 student reports • Initial feedback from Field Test sites positive • Instructions clear • Online administration easy • Happy with the quality of technical support • Field Test psychometrics consistent with Pilot results and previous studies
FIELD TEST FINDINGS Time to Complete Survey Elementary and Middle School Students Reported Time to Complete Survey
FIELD TEST FINDINGS Time to Complete Survey Classroom Teacher and Staff Reported Time to Complete Each Student Report
NEXT STEPS • Can programs sign up now? • Yes, but they will not get a report of pre/post change scores, only the results of the spring survey administration. • Provide support to new participants enlisting in Field Test for spring 2011 • Contact current Field Test participants to prepare for spring survey administration (April - May 2011) • Dissemination of findings in summer 2011 in collaboration with California Afterschool Network Regional Leads and CDE