1 / 11

Presentation to the Before and Afterschool Advisory Committee

Pilot Findings and Field Test: An Update. Presentation to the Before and Afterschool Advisory Committee California Department of Education January 11, 2011. California Afterschool Outcome Measures Project. Last report to the Advisory Committee —February 24, 2010

eleanor-hanson
Download Presentation

Presentation to the Before and Afterschool Advisory Committee

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pilot Findings and Field Test: An Update Presentation to the Before and Afterschool Advisory Committee California Department of Education January 11, 2011

  2. California Afterschool Outcome Measures Project • Last report to the Advisory Committee —February 24, 2010 • Pre-Pilot activities and Key Findings: • Completed pre-pilot test of initial versions of measures and online administration • Measures performed well in online format; preliminary outcomes comparable to previous studies • Minor modifications made to measures and procedures based on pre-pilot findings • Changes to wording in items, scales removed and added • Survey administration instructions revised—clear step-by-step • Special instructions created to support ELL students and slow readers

  3. Highlights of Today’s Report • Phase 2—Report on the Pilot Test • Creation of project website • Results of the spring 2010 pilot work • Phase 3—Update on the Field Test • Fall survey administration experience • Dissemination plans

  4. Phase 2: PILOT PHASE • Project website created: afterschooloutcomes.org • Piloted online measures of positive behavior change and skill development: • 27 sites (13 in Southern and 14 in Northern CA) • 5 urban; 10 rural; 12 suburban • 18elementary; 6 middle school; 3 center-based serving K-8 • 10 regions across the state • Range of school/program sizes in Pilot sample: • School enrollments 225 to 1,963 • ASES enrollments 43 to 574 • Demographically diverse students (ethnicity, ELL and FRL)

  5. Pilot DataNumber of Respondents & Time to complete Survey • 1,656 Students completed surveys • 1,175 Elementary; 481 Middle • On average less than 10 minutes to complete survey (observed) • 51% elementary students report they took less than 10 minutes • 61% middle school students report taking less than 10 minutes • 105 Staff completed 769 student reports • 54% report taking less than 5 minutes per student report • 46% report taking 5 to 9 minutes • 70 Classroom Teachers completed 535 student reports • 81 % report completing each student report in less than 5 minutes • 18% report taking 5-10 minutes per report • 1% took more than 10 minutes

  6. FIELD TEST (2010-2011) Overview of key elements: • Participation open to all programs in the state • Outreach effort to all regions: • Presented on Field Test at Regional Lead Meeting (October 13, 2010) • Sent Field Test materials electronically to RLs for dissemination to all grantees • Web-based surveys for students, program staff, and classroom teachers completed in fall 2010 and spring 2011 • Programs select which surveys to administer • Confidential summary report of survey results • students’ scores of positive behavior change and skill development provided to participating programs • All materials and instructions available on Project website • Technical assistance provided to programs • Support provided by email and telephone during pre and post survey administrations & in interpreting their scores at the end of Field Test • Measures modified based on Pilot results and feedback from field

  7. FIELD TEST (2010-2011)Outcome Measures • Student—Elementary & Middle School—(5 subscales) • Work Habits • Misconduct • Reading Efficacy • Math Efficacy • Social Efficacy • Staff (5 subscales) • Work Habits • Aggressive Behavior with Peers • Prosocial Behavior with Peers • Task Persistence • Social Skills • Teacher(6 subscales) • Same as Staff plus Mock Report Card

  8. FIELD TEST FINDINGS (Fall 2010 – Spring 2011) • Diverse programs across the state participate in fall 2010 field test • 156 Sites (elementary & middle school) • 23 Programs (10 rural, 8 urban, 5 suburban) • 8 Regions represented • Pre-Surveys Completed (November 1 through December 17) • 3934 elementary students • 1847 middle school students • 360 program staff completed 4611 student reports • 29 classroom teachers completed 185 student reports • Initial feedback from Field Test sites positive • Instructions clear • Online administration easy • Happy with the quality of technical support • Field Test psychometrics consistent with Pilot results and previous studies

  9. FIELD TEST FINDINGS Time to Complete Survey Elementary and Middle School Students Reported Time to Complete Survey

  10. FIELD TEST FINDINGS Time to Complete Survey Classroom Teacher and Staff Reported Time to Complete Each Student Report

  11. NEXT STEPS • Can programs sign up now? • Yes, but they will not get a report of pre/post change scores, only the results of the spring survey administration. • Provide support to new participants enlisting in Field Test for spring 2011 • Contact current Field Test participants to prepare for spring survey administration (April - May 2011) • Dissemination of findings in summer 2011 in collaboration with California Afterschool Network Regional Leads and CDE

More Related