Evaluation 101: After School Programs

1. Evaluation 101: After School Programs February 1, 2007 Region 3 After School Technical Assistance Center Conference

2. Evaluation 101 • ASES Evaluation—student data and how to get it • Prop 49 evaluation requirements in SB638 • Suggested Timeline • Roles and Responsibilities

3. ASES Evaluation • Annually by October 1 funded programs must submit to CDE data on participating students • Two years data required: • 2006/2007 follow up/program year • 2005/2006 baseline/prior year • Data submitted to CDE in Excel file • Evaluation narrative

4. Prop 49 Evaluation SB638 • Programs must submit individual student data: • Program attendance – participation • Regular school day attendance • STAR test results • Program effectiveness measures (1 or more from list) • STAR test data • Positive behavioral change* • Homework completion rates* • Skill development* • Additional measures to be determined by CDE * Measured by ASP teachers or regular day teachers

5. Prop 49 Evaluation SB638 • State Board of Education will define performance measures by November 2007 • Funded programs will comply with state evaluation requirements determined in collaboration with Advisory Committee

6. Terms • ASP: After School Program • CHKS: California Healthy Kids Survey • Participation: after school program attendance • Regular attendance: regular school day attendance • Baseline: prior year 2005/2006 • Follow up: program year 2006/2007 • Continuing students: those who participated the previous year in After School Program

7. ASP Evaluation Keys to Success • Four distinctive types of student information needed • Build evaluation database throughout year • Clear roles and responsibilities • Good ongoing communication • Document process and sources

8. Evaluation data: Four Types Needed on Students • Participation in after school program: • number of days attended program in the school year • Regular school attendance: • days enrolled and days absent in the school year • California Standards Test • proficiency level math and language arts • California Healthy Kids Survey • questions at entry and exit from program (4 questions)

9. Sources of Data

10. Sample Excel File in Folder

11. Who Needs to be involved? • ASP director and site staff • District data managers, or evaluation department, or fiscal department with regular school attendance information on students • District testing and curriculum staff who obtain the CST results annually and disseminate them to schools • Staff with expertise in Excel (or database software)

12. Tips • Create a roster of key people involved in ASP evaluation with their phone numbers, faxes and email addresses. • Do ASP evaluation update every 2-3 months to keep all informed on progress. Will raise ASP evaluation visibility • Immediately thank via email and/or phone call anyone who gives you data • Write it down – for next year!

13. Participation Data

14. Participation Data • Prior to program beginning: • Decide on how to gather and record program attendance: after school program software, Excel spread sheets or Access database • Find last year’s evaluation report in Excel format to get baseline data (ongoing programs) • Decide on roles and responsibilities for data entry • Who records attendance • Who enters data into computer

15. Participation Data • Record child’s name (first and last), grade, school, Student ID Number and days attended by month • As new children enter, record them in the data system on an ongoing basis • Continuing participants, gather baseline for participation, regular school attendance and CST from prior year’s Excel file

16. Tips • Keep track of Student ID Numbers – they are the key to obtaining regular school attendance and CST results • Get child’s name as they are registered in school – no nicknames. James Jones not JJ Jones or Jimmie Jones • Don’t record attendance in Word , you will need to be able to add figures and sum them up

17. Regular School Attendance

18. Regular School Attendance • Data needed: • Days enrolled, days absent current school year 2006/07 and prior school year 2005/06 • Where is it kept: • District data system • Link by Student ID Number and school

19. Regular School Attendance • What you need to do: • Identify where data is kept in district • Contact person who can get you access to data and what you will need: initial information query in fall • Contact in March to obtain baseline prior year attendance for students new to program • Contact in June to obtain current year attendance for all participants

20. Tips • Don’t be surprised if there is no prior year attendance for new participants – student mobility means students may be new to the school in program year • Check data for new participants in March for reasonableness • Days enrolled should be around 170 (150-181) • Days absent should be between 1 and 20; rarely over 20

21. California Standards Tests

22. California Standards Tests • Results available July or August following end of school year • Math score and Language Arts Score • Scores are 1 through 5 or Far Below Basic, Below Basic, Basic, Proficient and Advanced • Best source for student demographics: gender, ethnicity, English Language Learner Status

23. California Standards Tests • Data needed • Proficiency level in math and language arts for program year (spring 2007) and baseline year (spring 2006) • Demographics on students • Where it is kept • Comes to district testing staff July or August on CD ROM from state testing contractor • For continuing ASP students, baseline data is on prior year evaluation report to CDE • For new ASP students, baseline data is prior year CST results 2005/2006

24. California Standards Tests What you need to do: • Identify test contact person in district • Contact person who can get you access to test data: initial information query fall • Contact in March to obtain baseline prior year proficiency for students new to program • Submit file with id numbers of new students by school with data request • Contact in June to remind them of upcoming summer request • Submit file with id numbers of all participating students in summer for new test data and new demographics

25. Tips • Don’t be surprised if there is no prior year CST results for new participants – student mobility means students may be new to the school in program year • Keep in touch with testing person for district, sometimes data comes in early • Don’t wait until the summer, many people are not working in summer months!

26. California Healthy Kids Survey

27. California Healthy Kids Survey • Train staff to administer survey when program begins and when students leave the program or near the close of school year • Enter initial (pre) surveys in evaluation database as they come in.

28. Survey Questions 1.  In the past 30 days, how often have you wanted to go to school? (Never, one or two times, About once a week, A few times a week, Everyday) 2.  In the past 30 days, how often have you studied for a test? (Never, one or two times, About once a week, A few times a week, Everyday) 3.  In the past 30 days, how often have you felt unsafe at school? (Never, one or two times, About once a week, A few times a week, Everyday) 4.  In the past 30 days, how often have your parents asked you about school? (Never, one or two times, About once a week, A few times a week, Everyday)

29. Tips • Be sure to gather pre surveys for students who enter the program during the year. • Train staff to look for red flags if a student reports they don’t feel safe, for example. • Gather the post follow up surveys (same 4 questions as on entry to program) beginning about one month before the program ends. • Enter into database.

30. Month before program starts Fall Decide on database format Set up participation data entry system Train staff on California Healthy Kids Survey and attendance procedures Find last year’s evaluation report Excel files Identify person in district with regular school attendance data and CST test scores and hold initial meeting Master timeline

31. October through March Check to see that you can generate list of students by school with id numbers and total days attended for first three months or six months Check that number of initial California Healthy Kids Surveys in database match number of participants Build baseline for ongoing students using ASP participation, regular school attendance and CST results from previous evaluation Excel file Master timeline, continued

32. May of school year Meet with district attendance data contact; provide list of id numbers of participating students; obtain prior year attendance for new students Review completeness of database, look for holes on continuing students and fill them Master timeline, continued

33. June July August Request current year attendance data for all participants, giving school list of students with ID numbers Request CST data and demographics from district test contact Finalize Excel file with Student ID, days of ASP attendance; regular school attendance; CHKS answers Master timeline, continued

34. September Add test scores from CST to Excel files Complete narrative Submit to CDE Master timeline, final