Program Evaluation

1. 2008 NCLB School Choice Leadership Summit Program Evaluation Cristy Davidson, Joe Sterensis, and Anke Toth Wednesday, June 25, 2008

2. Agenda • Legislative Requirements • Data Reporting • Ensure Quality of Data • SES Evaluation • Ensure Program Effectiveness

3. Legislative Requirements Federal Requirement – Section 1116(e)(4)(D): NCLB requires FDOE to develop, implement, and publicly report on standards and techniques for monitoring the quality and effectiveness of the services offered by approved providers to contribute to increasing the academic proficiency of students

4. Legislative Requirements (cont.) State Requirement – SBE Rule 6A-1.039, FAC: • The Department is required to evaluate the quality and effectiveness of SES providers. • The evaluation will measure academic proficiency in reading/language arts and mathematics for all students participating in SES.

5. Legislative Requirements (cont.) Increased academic proficiency means: “Demonstrating increase in academic proficiency as measured by 60% of students earning a minimum of 1 normal curve equivalency (NCE) point learning gain in reading/language arts and 70% of students earning a minimum of 1 NCE point learning gain in mathematics.”

6. Legislative Requirements (cont.) Senate Bill 1414 amends Section 1008.331,F.S.: • Requires the Department to annually evaluate and assign each state-approved SES provider a performance grade based on a combination of students learning gains and student proficiency levels. • Requires the Department to assign grades beginning with the 2007-2008 school year and to report them by March 1 of each year.

7. Legislative Requirements (cont.) Performance Grade •  “A,” providing superior service •  “B,” providing above satisfactory service •  “C,” providing satisfactory service •  “D,” providing below satisfactory service • “F,” providing unsatisfactory service

8. Data Reporting Uses of the Data • Funding decisions • Monitoring program implementation • Compliance with federal and state statutes • Needs for technical assistance • Provider evaluations

9. Data Reporting (cont.) Requirements • District and school information (Appendix W) • Student demographic information • SES provider identification number (Appendix T) • Contact hours by subject • Provider site number • Service start date

10. Data Reporting (cont.) Survey Dates • Survey 2: October 13-17, 2008 • Survey 9: December 11, 2008 • Survey 3: February 9-13, 2009 • Survey 5: August 7, 2009

11. Data Reporting (cont.) Total Student Participation in SES in 2006-2007 (Survey 5) by Grade Groups

12. Data Reporting (cont.) Total Student Participation in SES in 2006-2007 (Survey 5) by Ethnicity

13. Data Reporting (cont.) Total Student Participation in SES in 2006-2007 (Survey 5) by Sub-Groups

14. Data Reporting (cont.) Total Student Participation in SES in 2006-2007 (Survey 5) by Gender

15. Ensure Quality of Data:Supporting Documentation • Attendance Roster • Student or parent and tutor signature to show attendance on each date, including length of session • Progress Reports • Parent signature on student progress form • Submitted to district with monthly invoice • Includes progress from past months • Check the contract of the individual districts to verify required supporting documents

16. Ensure Quality of Data:District Procedures • Provider Trainings • Contract and Procedures • Database System • Student Learning Plans • Invoice Documentation

17. Ensure Quality of Data:District Procedures (cont.) • Helpful Provider Materials • Procedure handbook • Provide list of due dates • Quick Reference sheet for database tasks • Email at the beginning of each month with upcoming information • Website link to Provider forms

18. Ensure Quality of Data:District Procedures (cont.) • Facilitators • Observation Forms • On-site • Off-site • Check Rosters • Student Contact Info • Student Safety

19. Ensure Quality of Data:Provider Procedures • Employee Training • Provide clear and thorough expectations • Include Completed Sample Forms • Ensure open line for communication • Allow new tutors to observe a model session via DVD or in person

20. Ensure Quality of Data:Provider Procedures (cont.) • Observe Tutors • Lead teacher or team • Opportunity to review observation forms with tutors • Plans for redirection • Consequences for non-compliance • Keep district database up to date and accurate

21. SES Evaluation Data Collection and Reporting • Collect student information via Department’s data base system • Match provider and student information • Link providers to student learning gains

22. SES Evaluation (cont.) Measuring Instrument • State-administered Florida Comprehensive Achievement Test (FCAT) in Reading/Language Arts and Mathematics grades 4-10 • Growth of at least 1 normal curve equivalency (NCE) from one grade to the next • Another norm-referenced test (NRT) approved by FDOE grades K-3 • Growth of at least 1 NCE from one grade to the next

23. SES Program Determining whether the SES program offered to students contributes to an increase in student academic achievement SES Provider Evaluating quality and effectiveness of the services offered based on the academic achievement of the students served SES Evaluation (cont.)

24. SES Evaluation (cont.) Evaluation Plan for Effectiveness of SES Program • Dimension • Treatment Group = Eligible students participating in SES • Control Group = Eligible students NOT participating in SES • Calculated for each district with a state average

25. SES Evaluation (cont.) Evaluation Plan for Effectiveness of SES Program • Dimension • Calculate contact hours for reading/language arts and mathematics for increments of time • Determine an optimal amount of time for tutoring to maximize learning gains

26. SES Evaluation (cont.) Evaluation Plan for Effectiveness of SES Providers • 60%of students showing growth of at least 1 NCE from one grade to the next in reading/language arts • 70% of students showing growth of at least 1 NCE from one grade to the next in mathematics

27. SES Evaluation (cont.) Evaluation Plan for Effectiveness of SES Providers • Students who received a minimum of 10 hours of contact per provider • Students who received services in reading/language arts, mathematics, or both • Cell size = 10 or more students in the same subject area

28. First Year: Implement a system improvement plan Second Year: Removal for the state-approved provider list Ineligible to apply to be a provider for two years SES Evaluation (cont.) Consequences for SES Providers

29. Ensure Program Effectiveness • Tutor Recruitment, Hiring and Retention! • Tutor Training! • Tutor Oversight! • Tutor Evaluation!

30. Ensure Program Effectiveness (cont.) • Hourly rates - Money vs. time – The age old question! • Group size - Effectiveness vs. profit! • Length of sessions - Effectiveness vs. profit! • Student mix - Grade level, behavior, subject - Same ethical dilemma!

31. Ensure Program Effectiveness (cont.) • 6 - 9 month revenue cycle! • Year round full time administrative staff – expensive but essential! • Technology - expensive but essential! • Additional seasonal administrative support!

32. Ensure Program Effectiveness (cont.) • If we build it they will come! • If they come we will build it! • What if we build it and they 'do not' come? • What if we don't build it and they 'do‘ come?

33. Ensure Program Effectiveness (cont.) • How do we know we are good? • How do parents know we are good? • How does the State know we are good? • Who is better? • How can we know?

34. Resources Web link regarding NCLB School Choice Data Reporting (CWT,SES): http://www.fldoe.org/flbpso/nclbchoice/ses/sesdataelements.asp

35. Questions

36. Contacts Cristy Davidson Marion County Public Schools Phone: 352.620.7652 Email:  cristina.davidson@marion.k12.fl.us Joe Sterensis "A+" ALC - Advanced Learning Centers, Inc. Phone: 727.381.9722 ext 202 Email: joe@thinkalc.com Anke Toth Florida Department of Education Phone: 850.245.0479 Web address: www.fldoe.org/flbpso