The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward

1. The Community Schools Evaluation Toolkit:Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community Schools

2. Agenda • Research summary • Community Schools Evaluation Toolkit • Discussion • Work time: work on your evaluations and network www.communityschools.org

3. Research Summary Are community schools effective at improving outcomes for students, families, and the community? • Collected 153 studies of community school and community school-like initiatives • CS models: lead-agency, community agency/CBO, university-assisted, and staff-initiated • Scale: boutique, local, state, national • No studies in peer-reviewed journals www.communityschools.org

4. Characterizing the Literature • Mix of internal/external evaluators • Great variation in study design • Process/implementation studies (e.g., number of students served) • Outcome studies (e.g., improved achievement) • Number of outcomes depends on the unique strategies of each community school – found at least 30 outcomes • Tied to theory of action • Selected 22 quasi-experimental studies (comparison group, interrupted time series, controls) www.communityschools.org

5. Achievement The most rigorous studies indicate that community schools are improving student achievement for some students. • High-implementing CIS schools scored +6% than comparison non-CIS schools in percent proficient in grade 8 math (p<.01) and +5.1% in grade 8 reading (p<.05) • Mixed results out of Chicago for school-level data • Student-level demonstrates that OST participation matters • More students who participated in Children’s Aid Society (CAS) after-school programs demonstrated a steady increase from 2004 to 2007 in their math performance levels as measured by the state assessment compared to students who did not attend CAS activities (p<.05) www.communityschools.org

6. Attendance The most rigorous studies indicate that community schools are improving attendance. • CIS: High-implementing elementary schools had higher attendance levels than comparison group (+0.2%, p<.05). So did high-implementing high schools (+0.3%, p<.01) • CAS: students who participated in CAS after-school programs for 3-4 years had better attendance than students with lower or no participation (p<.05) • SF Beacons: participants who attended 30+ days of Beacon after-school had 3.9% less total days of unexcused absences than those who participated less than 30 days (p<.001) www.communityschools.org

7. Graduation Rate While a focus of many community schools, there is little evidence of improving the graduation rate. • CIS: high-implementing community schools had a significantly better graduation rate than comparison schools (+4.8 percent, p<.01) • Compared to other large-scale dropout programs, high-implementing CIS schools had the highest effect size for graduation rate (ES=.31) and second-highest for dropout rate (ES=.36) • CPS: schools that have been in CSI longer have better rates of ninth-grade students “on-track” to graduate than comparison group (p<.05) www.communityschools.org

8. Behavior While a focus of many community schools, there is little evidence of improving student behavior. • CPS schools had significantly less disciplinary incidents than their matched comparison group from 2002 to 2006 • CAS teachers reported that they saw greater improvement in students getting along with others for students with higher rates of participation in CAS than comparison students during the 2006-2007 school year (significance not reported) www.communityschools.org

9. Other Outcomes • Overall, few studies measure, evidence is weak, for the following outcomes: • Parental engagement • Relationships with adults • Student engagement • Impact on instruction www.communityschools.org

10. Future Directions for Evaluating Community Schools • Utilize high-quality study designs that establish causality • Why? Funding. Have to demonstrate effectiveness and grant-makers are increasingly aware of study design (e.g., i3) • Evaluate the CS strategy, not just programs • Community schools are more than after-school programming • Is the strategy or particular programming causing changes in outcomes? • Use appropriate unit of analysis (school, student, or both) • Measure: • level of implementation: fidelity to design matters (e.g., CIS) • longevity • intensity of the intervention (after-school participation, receiving health services, … • Evaluate outcomes for families and community; also student health • Develop standard measures: e.g., how do you measure student engagement? • Assess effectiveness of particular CS components to modify and strengthen the design (e.g., do CS coordinators matter?) www.communityschools.org

11. Community Schools Evaluation Toolkit • Rationale for Results Framework • John W. Gardner Center • Community School Evaluators • Coalition for Community Schools • JP Morgan Chase www.communityschools.org

12. What is in the Toolkit? • 35 Page step-by-step manual • Coalition for Community Schools Logic Model • Indicators for 5 short-term results • Indicators for 4 long-term results • Descriptions of school, city, county, and state data that is available to sites • 45 surveys –for sites that are ready to collect additional data • Examples from real sites www.communityschools.org

13. Goals of the Toolkit • Introduce the Coalition for Community Schools (CCS) Results Based Logic Model • Support sites to use existing data sources and collect additional data (if needed) • Support sites to identify areas of success and areas in need of improvement • Provide evaluation planning tools www.communityschools.org

14. 4 Parts to the Toolkit: • Before you Start: Begin with the End in Mind • Get Ready: Prepare to Evaluate • Get Set: Designing the Evaluation • Go!: The Evaluation Process www.communityschools.org

15. www.communityschools.org

16. Discussion • Describe your experience evaluating CS. • What challenges have you experienced? Successes? • What are your evaluation needs? www.communityschools.org

17. Work Time • Work on your evaluations and network www.communityschools.org

18. How to Get the Toolkit • Coalition for Community Schools Web site: www.communityschools.org • Questions? Comments? • shahs@iel.org www.communityschools.org

19. Contact information • Reuben Jacobson • reubenjacobson@gmail.com • Shital C. Shah • shahs@iel.org www.communityschools.org