Maine 21 st Century Community Learning Center Program Evaluation 2011

1. Maine 21st Century Community Learning Center Program Evaluation 2011 Findings January 30, 2012 Prepared by: Hart Consulting, Inc. www.hartconsultinginc.com

2. Evaluation Methods Approach Goal based evaluation Capacity, process and outcome measures Multiple years of data No formal control or comparison sites Limitations Outcome data are aggregated by center and do not track progress of individual students Student characteristic data (gender, socio-economic status, etc) are not linked to individual student performance outcomes

3. Evaluation Framework • Program Inputs • Program Guidance • Funding • Partners and Collaborators • Staff • State/federal oversight • Evaluation • Activities • Academic/Enrichment Activities • Student Engagement • Staff Training • Capacity Building • Outcomes • Improved Learning • Improved Behaviors • Quality Programs • Sustained Student Attendance • Sustained Programming Evaluation Data Sources Student Assessment Records Teacher Surveys Site Visit Reports Program Database Program Records Site Visit Reports Program Database Healthy Environment Survey

4. Data Sources Site visit reports- grantee capacity Teacher surveys-behavioral and academic learning outcomes(99% response rate for regular attendee pop.) Assessment scores-learning outcomes (74 centers completed) 5210 Healthy Environment Survey- 68 centers completed) Program records-student, adult participation; cost information Program offerings-enrichment activities

5. Center Characteristics 2011 33 grantees with 80 centers Serving 9,062 students 711 paid staff, 652 volunteer (school year) 370 paid teachers (school year) 50% in second year of grant, 50% in third

6. Disposition of Students Enrolled

7. Student Characteristics 51% boys, 49% girls 48% middle school, grades 5-8 43% regular attendees (30+ days attendance) 70% of all attendees are low performing 80% of all low performing attending regularly (30+ days)

8. Majority of Students in Middle and High School Grades

10. Cost per RLP Target Range

12. Traits of Quality Afterschool ProgramsReaching Potential Through Quality Afterschool Programs, Maine Children’s Cabinet, Afterschool Workgroup, LD 1369 Resolve 41. Published in 2008. • Effective organizational capacity • Sound instruction/programming • Maintaining quality staff, including hiring, compensating and retaining qualified staff • Positive, caring, respectful relationships among youth and adults • Thoughtful, intentional relationships with other individuals and groups • Assuring physical and psychological safety by creating structured, healthy, safe spaces and a culture of peer and adult support. • Building effective partnerships that use community resources effectively

13. Maine 21st CCLC Program Goals Address Traits of Quality Programs 1. Academic Improvement 2. Health & Wellness 3. Cultural Enrichment & Multi-Cultural Education 4. Parent Education and Engagement 5. Sustainability

14. In 2011, Maine 21st CCLC Grantees Reaching audience of focus-low performing low income students Majority of the centers are meeting 4 program goals, almost all are making progress toward goals All are meeting the 5th goal on sustainability Seeing improvements in all areas Sustained student academic and behavioral success, measured in aggregate year over year

16. Goal #1 Academic Improvement –Sustained Success

17. Goal #1 - Classroom Behaviors are Improving Among Regular Attendees

19. Goal #2 - Health and Wellness- New Measure Shows Accomplishment, Areas for Improvement

20. Goal #3 - Community Based Organizations and Other Partners Provide Enrichment Opportunities

21. Goal #4 – Parent Education and Engagement 544 event opportunities for parent participation 1,658 parents attended the opportunities

22. 5th Goal- Sustainability is Grant Requirement 3-part Sustainability Plan includes the following components: • An Advisory Board made up of school and community partners that meet a minimum of 4 times per year. • Strategies for raising money to sustain the program through additional grants, fundraising, fees, partners assuming programming costs, etc. • Development and implementation of a communication and advocacy plan that demonstrates effective outreach and awareness about program outcomes.

23. Underlying Differences among Center Performance in RLP Learning OutcomesDetailed findings on next slide-analysis of variance using t-statistic Centers showing higher proportions of improved RLP academic assessments are: • Centers with fewer enrollees overall <100 (95% confidence level, p<.05) • Centers that have higher average daily attendance >60 days, (90% confidence level, p<.10) Insufficient data to analyze • program maturity, • Specific student demographic differences-gender, age, socio-economic status Attribute showing no difference in RLP outcomes: • Cost per student **Caution-learning outcomes are shown as aggregated student data and do not show longitudinal progress of individual students.

24. Analysis of Differences among Center Performance by Center Characteristics Analysis using t-statistic comparison of means

25. Center Specific Determinants of Student Outcomes Regression analysis of center inputs and characteristics explain much of the variation in the numbers of students improving learning outcomes across centers (78-79% fit). • Key explanatory variables are: • # of paid teachers, • # of volunteers, • Avg daily attendance of student, • Avg daily attendance in homework activities (all students) (not statistically significant but improves overall explanation of variation)

28. Summary of Findings • Maine’s 21st CCLC Program requires its grantees to adopt the key traits for Quality Afterschool Programs best practices and ensures implementation through careful monitoring, site visits, and work reports. • Almost all of Maine’s grantees are meeting the 5 program goals, which align with Quality Program traits. • Smaller centers and those with higher average daily attendance have a higher proportion of regularly attending students showing academic improvements. • Variations in per student funding did NOT make a difference in student outcomes. • Paid teachers, volunteers, increases in daily attendance, attendance in homework activities will increase the numbers of students showing academic improvement.

29. Recommendations • Maine should continue its monitoring and evaluation activities with a focus on sharing feedback to grantees and stakeholders regularly. • Share findings on the key drivers of success with grantees. • Identify Maine’s best practices that promote the key drivers of success in sessions such as professional development workshops or peer-to-peer sharing. These could include ideas to increase daily attendance, increase the number of paid teachers, as well as volunteers. • Adopt the Let’s Go! 5210 measures as indicators of a healthy afterschool environment. Link with the Let’s Go! program to share strategies on implementing activities and policies to promote healthy eating and increased activity.

30. Evaluation Recommendations • Explore ways to improve reporting on enrichment activities. For example, create structured definitions for each category of activity and reduce the number of activities. • Explore the idea of having grantees submit academic outcome data for individual students rather than entering outcome data aggregated by center. Would the individual data lead to program improvements or is it “nice to have”?