21 st CCLC The Afterschool Experience Conference: Year 1 AER Review Findings

1. 21st CCLC The Afterschool Experience Conference: Year 1 AER Review Findings Thursday, May 30, 2019 Jonathan Tunik, Project Director Lily Corrigan, Project Associate Nora Phelan, Project Assistant Dr. Nina Gottlieb, Senior Research Consultant

2. Presentation Objectives • Purpose of the reviews • Selection criteria • Variations in Program Objectives • Implementation and Outcomes • Conclusions • Announcements

3. Purpose of the AER Reviews • To help program staff become better informed consumers of evaluation • To identify patterns of strengths and challenges • In program implementation • In program evaluation • In communication of findings Caveat: These reviews are based entirely on the spring 2018 Annual Evaluation Reports (AERs). The information obtained was not verified through site visits, monitoring or additional document review.

4. Selection Criteria • AERs representing: • Region (NYC/RoS) • Grantee (LEA, CBO, OCS) • Locale (urban, suburban, rural) • Program size • Grades served • Evaluation firm • Goal is all programs reviewed over 5 years

5. Variations in Program Objectives

6. Types of Implementation Objectives Used in Each Program

7. Types of Implementation Objectives Used in Each Program • level of activity: Included in almost all reviewed programs (93%) -- # activities, days/hours per week, #s served, etc. • implementation quality/fidelity:Large majority of programs (87%). However, • Mostly referencing activity schedules and consistency with proposal • Very few defining quality in terms of best practices • •Representativeness of targeted population: included in only 1 in 5 programs • “Regular” participation (90+ hours): included in only a third of programs

8. Types of Outcome Objectives Used in Each Program

9. Types of Outcome Objectives Used in Each Program • Most reviewed programs had a wide variety of outcome objectives • All reviewed programs had at least some kind of academic objectives in various subjects • Most programs had Social Emotional objectives (87%) and Behavioral objectives (87%) • Many programs had objectives for parent services/participation (73%) and community service/collaboration (67%)

10. Defining Usable Program Objectives • We looked at whether program objectives and performance indicators were: • Meaningful and interpretable • Showing alignment between goals, objectives, indicators and activities • Meeting all criteria of a SMART goal: • Specific • Measurable • Achievable • Relevant • Time-bound

11. Do Objectives Meet All SMART Criteria?

12. Do Objectives Meet All SMART Criteria? • Most or All objectives met all criteria in ~60% of programs • Missing criteria: primarily “Specific” and/or “Measurable”, some “Relevance” • Not “Specific”: “…program will provide social-emotional support services to students in need of intervention for at least 3 hrs/wk." • Not “Measurable”: “…100% of students will participate in enrichment activities including…” • “Relevance”: “50% of students improve their math and ELA grades fall to spring”.

13. Are Activities Aligned with Objectives/Indicators?

14. Activities Aligned with Objectives/Indicators • Most or All activities were aligned with objectives and indicators in 72% of programs • Examples of apparent mis-alignment (or incomplete reporting): • Objective not supported by activities: e.g., “All sites provide project based learning”; but activities don’t describe PBL • Specific activities (e.g. career readiness) with no associated objectives • Evaluation Plan/Logic Model/report structure do not indicate which activities expected to achieve which objectives

15. Published/Standardized Assessments Used to Measure Objectives

16. Implementation and Outcomes

17. Strategies for School Day Alignment

18. Strategies for School Day Alignment • The most common alignment strategy by far was through collaboration with teachers (83% of programs) • Just over half the programs asserted or described ensuring appropriateness for targeted grade level(s), identified needs of targeted students, and/or the grade-specific curriculum • Half the programs asserted alignment with state and local standards

19. Stakeholder Involvement in Optimizing Implementation

20. Stakeholder Involvement in Optimizing Implementation • About half of programs (50%-56%) described collaboration roles for 21st CCLC staff and school administrators • 2 in 5 (39%) described collaboration roles for school-day teachers • Only 1 in 5 (22%) described collaboration roles for students • Those that did have collaboration roles were most often deciding or acting together • Most collaboration described was with program and school staff • For the majority of reports we were not able to determine if collaboration took place

21. %s of Implementation Objectives Met Across Programs, by Type of Objective 92 objectives 77 objectives 7 objectives 9 objectives

22. %s of Implementation Objectives Met Across Programs, by Type of Objective • Where results were presented, twice as many “met” than did not meet quality objectives • But mostly about scheduling • Less than half of targeted population objectives were met • Where established, strong majority “met” objectives for “regular” participation

23. %s of Outcome Objectives Met Across Programs, by Type of Objective

24. %s of Outcome Objectives Met Across Programs, by Type of Objective • 77% and 81% “met” ELA and math objectives. [add something about ‘any academic’] • 61% “met” their parent objective(s) • 51% “met” SEL objectives • 41% “met” other behavioral objectives • 50% “met” community involvement objectives

25. Conclusions • All reviewed AERs included recommendations for action. • Ideal is when developed collaboratively, with explicit plans to follow up • Y2 AER will require reporting on status, and impacts of recommendations • Preliminary evidence of academic progress • Many reported making SEL/behavioral progress

26. Conclusions Making better use of Evaluation: • Ongoing, detailed attendance reporting facilitated strategies to increase participation • Programs that emphasized participatory evaluation and data-informed decision making were well positioned for program improvement and enrollment/attendance strategies.

27. Conclusions Additional Recommendations: • Strengthen quality review; population representativeness where applicable • Disaggregate results by activities/dosage • Ensure all objectives/indicators are measured (only 56% measured all) • Ensure that all stakeholders are informed of findings and recommendations, and involved in solutions • To inform ongoing improvement, strengthen focus on existence of and reasons for program modifications and drift.

28. Announcements Evaluability Checklists • Now required Year 1 only; Years 2-5 replaced by annual review/update of Logic Model, reflecting fidelity review Year 2 Student Attendance Rosters, and Year 2 AER Guide • Drafts to be discussed at Evaluators’ Track session Workshop Surveys! • Important feedback to State and Facilitators • Included in on-line post-conference survey • Please sign in to your workshops!