Agenda – FLP Evaluation

1. Building an Effective Evaluation of Your Flexible Learning Program (FLP) Presenter:Sue Hawks Foster, FLP Education Program SpecialistGeorgia Department of Educationsfoster@doe.k12.ga.us(404) 656-2636

2. Agenda – FLP Evaluation • Federal Regulation Requires Evaluation of the Title I Programs • FLP Evaluation Requirements- • Federal, State, and District • Building an Effective Program Evaluation • LEA Plan for District FLP Evaluation • Performance Based Observation of FLP • Most Common Findings FY13 • Questions and Answers • References

3. Federal requirements for evaluating title I programs

4. Federal Regulation for Title I Programs Federal regulations requires that a school operating a Title I program annually evaluate the implementation of, and results achieved by, the Title I program ESEA Title I program compliance requires written program evaluation procedure and program evaluation summary

5. Federal Requirements for Evaluation The Title I program evaluation is an annual review of Title I activities to determine if they are contributing to the desired outcomes: • Improved student achievement • Effective family engagement • High quality professional development

6. Federal Requirements for Evaluation Evaluation Components: Measures efficacy and impact of the Title I FLP program Documents impact of Title I FLP services on student learning Informs school and district planning (cycle of continuous improvement) Is an advocacy tool Is conducted at end of program year and used for the next years planning

7. Federal Requirements for Evaluation Description of: • Data used • Constituents consulted • Process used to arrive at evaluation findings • Previous data • Surveys • How findings are utilized for planning and improvement

8. Federal Requirements for Evaluation Summary of: • Program strengths • Program weaknesses • Subsequent program changes • Reallocation of funds

9. Needs Assessment & Program Evaluation What are the differences and how do they relate to each other? • Program evaluation • Evaluates Title I program-FLP specifically • Completed after the fact (end of year) • Helps to plan Title I program for next year including FLP

10. Needs Assessment & Program Evaluation • Needs assessment • Looks at all school data, including Title I • Schoolwide data in all subject areas • Targeted Assistance data in all subject areas • FLP data • Completed before beginning of school year • Assesses priority needs in district/schools and determines where Title I FLP fits in, as well as, other resources

11. FLP evaluation requirements

12. FLP Evaluation Requirements • State Evaluation • Participant Feedback • Evaluation Question: What is the overall experience of participants with program? • Data Source: Student and Parent Surveys • Service Delivery • Evaluation Question: Is the LEAs and programs in compliance with laws and regulations? • Data Source: Annual monitoring data, Program documentation, Federal reporting, Public reporting, Technical Assistance, etc.

13. State Evaluation cont’d • State Evaluation • Effectiveness • Evaluation Question: Are programs contributing to increased student academic achievement and performance on state education standards? • Data Source: Student performance on state tests, Pre-post assessment measures of state standards and academic skills targeting by programs, Performance Flag data, and student growth in schools offering FLP • Evaluation results would be shared with stakeholders and the public and used to inform ongoing program improvement

14. StateEvaluationcont’d • LEA Evaluation • LEAs will be required to evaluate outcomes of FLP interventions implemented at schools within their district that have been identified • LEAs will submit their evaluation plan as part of their overall FLP plan for approval by GaDOE Title I Specialists

15. State Evaluation cont’d • LEA Evaluation The LEA must submit the following components of an evaluation plan with their FLP plan in the CLIP for each Priority, Focus, or Alert school within their district required to implement FLP: • Effectiveness target or overall quantifiable goal of the program • Assessment instrument(s) that will be used to measure this target/goal • Plan for collecting and maintaining data • Data analysis plan

16. State Evaluation cont’d • What do the LEA evaluation requirements mean? • Focused, purposeful intervention • Tailor evaluation needs of program to match program goals and student needs • Expand on broadband state assessments to quantify benefits of FLP participation • Collect meaningful information for decision making and continuous program improvement

17. Building an effective program evaluation plan

18. Building Effective Program Evaluation • Determining Purpose of Evaluation • Why should we evaluate our program? • What is our program trying to do? • What information do our funders expect? • What questions do we want to answer?

19. Building Effective Program Evaluation • Performance Measures • Measures of effort • Measures of effect • Strong performance measures should: • Have strong ties to program goals, inputs, and outputs • Be compatible with age and stage of your program • Consider if the data you need are available/accessible • Yield useful information to the program

20. Building Effective Program Evaluation • Assessing Capacity for Evaluation • Who should be involved in the process? • What resources must be in place? • Who will conduct our evaluation?

21. Building Effective Program Evaluation • Selecting Evaluation Design • What type of evaluation should we conduct? • Formative to understand the changes that have occurred • Summative demonstrate they have achieved the standard required • What type of data should we collect? • Qualitative • Quantitative

22. Building Effective Program Evaluation • Selecting Evaluation Design (cont.) • What is our time frame for collecting data? • Causal • Experimental • Quasi-experimental • Non-experimental • Descriptive • Pre-experimental

23. Building Effective Program Evaluation • Choosing evaluation instruments • Construct • Audience • Format • Reliability • Validity

24. Building Effective Program Evaluation • Analyzing data • How will data be analyzed and shared? • Quantify central evaluation question (and supporting questions) • Communicating results • How? • Where? • Who?

25. LEA Plan for District flexible learning program (FLP) evaluation

26. LEA Plan for District FLP Evaluation List the effectiveness target(s) or overall quantifiable goal(s) of your programs What are the measureable outcomes that your intervention is designed to improve? Must include at least one academic achievement indicator

27. LEA Plan for District FLP Evaluation List the assessment instrument(s) that will be used to measure each listed effectiveness target/goal Include a plan for administering assessment instruments and how you will collect and maintain data

28. LEA Plan for District FLP Evaluation When will you administer pre- and post- assessments? How will you maintain student assessment information to ensure confidentiality? How will you determine program effectiveness based on program goals and measurement instruments listed above?

29. LEA Plan for District FLP Evaluation FLP Program Evaluation – Includes Academic Data and Narrative FLP Program Summary Chart Parent, Staff and Student Summary Results

30. True Test for FLP • Are graduation rates improving? • Are CRCT scores increasing? • Are EOCT pass rates increasing? • Is our CCRPI increasing?

31. Performance based observation (PBO)

32. Performance Based Observation • FY13 PBO in Review • 12 Districts were selected for PBO as a pilot • 16 schools were observed • FY14 Review • 26 Districts have been selected for review • 26 School will be observed

33. Performance Based Observation • Components of Performance Based Observation(PBO) • Review of Priority, Focus, and Alert School List • Select number of districts to be reviewed based on the ESEA Flexible Waiver • Selection of schools in district to be monitored

34. Performance Based Observation • Components of PBO cont’d • Notifications Sent to LEAs • Call to Title I Director • Follow up Letter to LEA – Superintendent, Title I Director and Principal of selected schools with brief outline of expectations

35. Performance Based Observation • Components of PBO cont’d • Agenda Developed by Title I Office • Follow up call to LEA by PBO Team Lead before visit • Addresses and list of staff reviewed • Selection of Teachers/Tutors for Interview

36. Performance Based Observation • Actual Visit to LEA • Interview with LEA Staff • Superintendent • Title I Director • School Principal • School FLP Coordinator • School FLP Instructors/Tutors (at least 2) • Other vital staff as needed

37. Performance Based Observation • Actual Visit to LEA • Classroom Observation -20 minutes per classroom • Talk with students and staff in classroom • Observe student and FLP staff interactions • Depart Building- Inform building staff we are leaving the building

38. Performance Based Observation • Follow-up Letter • Review of the PBO findings for the district • Non-Compliance • If a LEA is found out of compliance the district will be notified immediately by Program Manger • If corrective action is required it will be provided in writing and followed up with the FLP Specialist and/or the Program Director for FLP

39. FY13 FLP Most Common Findings

40. FLP’s Most Common Findings • Models used for FLP • After school programs were used by Middle and High Schools • Elementary and Middle schools extended the day or rotated the scheduled day • High schools used Saturday sessions • Summer Sessions were held when extra funds were available to extend the learning year

41. FLP’s Most Common Findings cont’d • Technology Usage Increased • Increase Professional Learning • Increased Student Instruction Time • More contact with parents • Reported Increase in Test Scores

42. FLP’s Most Common Findings cont’d • Frustration of operating the program by some districts because it was so new • Large amounts of carryover because of FLP funds were not used last year • Special Education Departments not connecting to FLP in various districts • Lack of staff in some districts to serve students

43. FLP’s Most Common Findings in Plans • Lack of understanding of FLP purpose • Did not provide performance goals for plans • Determining the needs assessment was not fully understood to determine subject area • Many did not identify all subgroups within the schools • Schools did not understand providing data for more than one year in order to provide a trend

44. FLP’s Most Common Findings in Plans • Special Education Departments were not really aware of what was going on with Students with Disabilities (SWD) in FLP. • Student ratio needs to be smaller • Individual Learning Plans needed to be developed and followed up • Documentation of Multiple Selection Criteria was lacking in some districts • Lack of understanding Professional Learning

45. Questions and Answers

46. References • Afterschool Evaluation 101: How to Evaluate an Expanded Learning Program, Erin Harris, Harvard Family Research Project, 2011 http://www.hfrp.org/ • Doing What Works Clearinghouse, USED -http://ies.ed.gov/ncee/wwc/ • Preschool for California's Children Hfrp_pubs@gse.harvard.edu • Harvard Family Research Project, 2013 http://www.hfrp.org/evaluation/overview • Georgia Department of Education, FLP Report http://www.gadoe.org/School-Improvement/Federal-Programs/Pages/default.aspx

47. Contact Information Sue Hawks Foster FLP Education Specialist sfoster@doe.k12.ga.us 404-656-2636