Washington 21st Century Community Learning Centers : From State to Local Program Evaluation

1. Washington 21st Century Community Learning Centers : From State to Local Program Evaluation June 2011

2. The Evaluation Team American Institutes for Research • Recent merger with Learning Point Associates • Responsible for the development and maintenance of the Profile and Performance Information Collection System (PPICS) • Demonstrated 21st CCLC and afterschool content knowledge • Other statewide evaluations of 21st CCLC in New Jersey, Texas, and Oregon

3. The Evaluation Team David P. Weikart Center for Youth Program Quality • Developers of the Youth and School Age PQAs • Working to build program quality systems in 20 states, including 12 statewide 21st CCLC implementations, including Washington • Rigorously tested intervention strategy for improving the quality of youth-serving programs

4. Evaluation Objectives • Provide an Assessment of the Current State of 21st CCLC Program Impact • Support the PPICS Reporting Process and the Collection of Student Level Data • Document the Extent to Which 21st CCLC Programs Are Meeting Local, State, and Federal Targets and Goals • Identify Characteristics Associated With High-Performing Programs • Increase the Capacity of Grantees to Meet Their Program Improvement and Evaluation Obligations

5. Overview • Provide an overview of the 21st CCLC program at the state level • Provide an overview of the 21st CCLC program at the local level • Recommendations for future directions

6. Assessment of the Current State of 21st CCLC • Define the characteristics of the current 21st CCLC programs in Washington • Assess program outcomes with participating students • Examine local evaluation reports and alignment with state level data • Examine the quality of local evaluation reports to support state guidance

7. Data Sources • 21st CCLC Profile and Performance Information Collection System (PPICS) • Grantee and center level • From APR 2010 • 45 local evaluation reports prepared for the 2009-10 reporting period

8. Characteristics of 21st CCLC Programs in Washington • 46 active 21st CCLC grantees (21 new, 23 mature, 2 sustaining) • A total of 172 centers • Median first-year award: $333,886 • 54 percent of grants has a school district as a fiscal agent • 96 percent of centers located in schools • 34 percent offer summer programming

9. Characteristics of 21st CCLC Centers in Washington During Academic Year: • Average of 9.9 hours of programming after school each week • Average of 4.4 days per week over 32 weeks During Summer: • Average of 20 hours of programming per week • 4.4 weeks of programming

10. Characteristics of 21st CCLC Centers in Washington • 68 percent of students were regular attendees • Multiple grade levels served, 38 percent are “elementary only” • Of the staff, 25 percent are paid school teachers, 35 percent are volunteers. • Multiple types of activities, 39 percent of the centers were “mostly enrichment “, 19 percent were “mostly homework help”

11. Characteristics of 21st CCLC Centers in Washington

12. Characteristics of 21st CCLC Centers in Washington

13. Characteristics of 21st CCLC Centers in Washington

14. 21st CCLC Program Student Outcomes • Is higher levels of attendance in 21st CCLC programming related to the desired academic and behavioral outcomes? • Are particular center and student characteristics associated with student academic and behavioral improvement?

15. State Assessment Outcomes

16. Attendance & Program Outcomes • Significant positive relationship between # of days in the program and improved behavior (based on teacher surveys). • Higher level of program attendance was not significantly related to increased performance in state assessments in reading and mathematics in 2009-10 (among students who scored below proficiency in 2008-09).

17. Program Characteristics & Outcomes School-based centers : • More likely to be associated with teacher reports of higher improvement rates. • More likely to demonstrate significant improvement in elementary students’ mathematics proficiency level. • Students that scored in the lowest proficiency category in the previous year (e.g., Well Below the Standard) were more likely to demonstrate improvement than higher performing students.

18. Program Characteristics & Outcomes • In centers classified as “mostly tutoring”, elementary students’ mathematics assesment scores were more likely to improve. • In centers classified as “mostly recreation” teachers rated lower rates of improvement in student behaviors. • In centers staffed by “mostly teachers” teachers reported higher levels of improvement in motivation, attentiveness and motivation, and homework completion and quality.

19. Local Evaluation Reports • What type of student outcomes do the programs target, and how do they measure these program outcomes? • What evidence can be obtained from the local evaluation reports regarding how programs may impact student outcomes?

20. Student Attendance • Grantee goals: All grantees took attendance but only 20 identified increasing attendance and retention as a program objective. • Attendance was highest in programs serving elementary school students and averaged 80 percent or higher. • Many programs seeking to achieve a high percentage of regular attendees were not able to achieve their objectives.

21. Academic Performance • Grantee goals: All grantees aimed to increase academic performance. • PPICS teacher survey and change in state assessment scores are the most commonly used measures of academic performance. • Many reports stated objectives to achieve a specific percentage of increase in student achievement but only few reported the percent change in achievement.

22. Academic Performance • Almost all reports shared findings descriptively. • Findings were more positive for teacher reports of improvement as compared to reports of change in state assessment scores. • A large number of reports shared only the percentages of students that improved while ignoring those whose behaviors declined or those who did not need to change.

23. Student Behavior and Attitudes Grantee goals: • Increase positive attitudes towards and sense of connection to school (most common) • Decrease referrals and negative behavior • Increased student skills • Student exposure to enrichment and community involvement activities • Exposure to career and post-secondary education opportunities.

24. Student Behavior and Attitudes • Data collected from student surveys, teacher surveys, student focus groups, parent surveys. • Teacher reports suggested that the majority of students improved although significant variations were commonly reported. • Evaluations that targeted more specific skills and behaviors reported positive change (e.g., reduced discipline referrals, fewer missed days, increased knowledge of careers).

25. Parental Involvement Grantee goals: • Increase parental involvement in children’s education • Parent attendance in program activities • Increasing parent skills and knowledge in English, literacy, and community resources

26. Parental Involvement • Data collected through parent surveys, interviews, or informal feedback but primarily reports of parent attendance in program activities. • Only a small number of programs achieved their goals on parental involvement. • Mostly reports of attendance in social events. • Few reported increased parent involvement with student’s education • Few reported increased level of skills and knowledge in parenting, ESL, GED.

27. Analysis of Local Evaluation Reports • Largely descriptive analysis to assess program impact • Assessment of outcomes based on PPICS teacher surveys and district-level assessments • Lack of meaningful assessment of student behavioral growth common • Assessment of parental involvement based on attendance • Results not rigorously compiled

28. Quality of Local Evaluation Reports

29. Quality of Presentation • Average is 3.0 • A large number of reports were rated as Above Standards ; • Clearly organized • Materials presented in an order that described what the program goals and objectives are • Activities described in a clear manner to provide an overview of the program.

30. Comprehensiveness of Content • Average is 2.8 • A large number rated Acceptable or Marginal Quality • Varying degrees of detail • Inconsistent reporting of data tools

31. Relevance of Content to Goals & Objectives • Average is 3.0 • A large number rated as Acceptable or Marginal Quality • Did not reflect program action theory • Objectives are not presented by SMART criteria • Assessments were not aligned with program goals and intended outcomes.

32. Rigor of Evidence • Average is 2.4 • More than half of the reports are rated as Marginal Quality • Did not triangulate findings from different resources to support evidence. • Descriptive reporting provide little evidence that outcomes are due to program participation. • Limited information on number of respondents (e.g., survey response rates, focus group participants) • Little insight into program implementation and quality

33. Alignment with Report Requirements • Average is 3. • In general, reports followed state requirements and guidelines. • Weakest aspect is meaningful discussion of findings and recommendations for future decision making.

34. Recommendations on Evaluation Design • Identification of evaluation questions • Developing a logic model to guide program evaluation • Consider the most rigorous evaluation design that resources allow • Align data collection tools with goals and objectives rather than what’s available • Need to use measures that better assess student behavioral outcomes.

35. Recommendations on Program Implementation and Quality • Incorporate program monitoring into annual evaluation • Adoption of a self-assessment tool on program supports and quality of activities • Developing an evaluation team

36. Recommendations on Reporting • Use a template to report findings • Developed and disseminated by OSPI • Require more insightful recommendations in evaluation reports that will guide action plans.

37. State-Level Recommendations • Consider the adoption of one or more measure(s) to assess social-emotional functioning and other behaviors related to academic functioning (e.g., task persistence, organizational skills, etc.) • Leveraging local assessment data to both (a) inform the design and delivery of programming and (b) assess student growth and development • Consider adopting indicators that ask programs to measure within-year student growth on formative assessments employed by the districts they are working with

38. State-Level Recommendations • Further examine the relationship between student recruitment and enrollment policies and the achievement of desired outcomes • Find ways to connect the leading indicators being developed currently with local evaluation efforts

39. Activity - Pair or group with the participants sitting next to you. - Reflect on the findings that were presented. - Share with group • How can the local evaluations supplement state data rather than duplicate the findings? • What are the most feasible short-term and long-term recommendations for improving local evaluations? • How can the state can provide guidance to implement the short-term and long-term recommendations?

40. Manolya Tanyu , Ph.D Researcher American Institutes for Research P:312-288-7611 E-Mail:mtanyu@air.org 20 North Wacker Dr. Suite 1231 Chicago, IL 60606 General Information: 800-356-2735 Website: www.air.org Neil Naftzger Principal Researcher American Institutes for Research P: 630-649-6616 E-Mail: NNaftzger@air.org 1120 East Diehl Road, Suite 200 Naperville, IL 60563