Identification of Key Indicators of Quality in Afterschool Programs

1. Identification of Key Indicators of Quality in Afterschool Programs Denise Huang American Educational Research AssociationAnnual Meeting AERA San Diego, CA - April 13-17, 2009

2. The study questions • What are the basic core benchmarksfor quality afterschool programs? • What are the key indicatorsthat help define the core benchmarks in afterschool programs? • How can afterschool programs use a data-based system to identify their own strengths and weaknesses and strive for continuous improvement?

3. Study design • Synthesis of literature on quality indicators was conducted to extract commonly referenced indicators and benchmarks for quality programs • A theoretical model was constructed • Surveys, interviews and observation protocols were developed by CRESST to validate these benchmarks and indicators • Three local well established and high functioning afterschool programs were identified through a strategic recommendation procedure for validation • A statistical weighing system was utilized to develop the preliminary tool

4. Synthesis of literature • A wide search for relevant literature • The abstracts of the 1,269 citations were obtained then carefully reviewed and discussed among the research team members • When abstracts met the established criteria, the full articles (216) were reviewed • 54 studies met the criteria for inclusion • These studies were coded for benchmarks/indicators extracted from the synthesis

5. 13 Benchmark across categories

6. Participating programs • LA’s BEST (Los Angeles Better Educated Students for Tomorrow ) • Lawndale RAP (Lawndale Realizing Amazing Potential ) • Pasadena Learns (Pasadena Leading Educational Achievement – Revitalizing Neighborhoods) Each has been designated as a California After-School Partnership (CASP) Regional Learning Center and LA’s BEST was selected by World Hunger Year (WHY) as one of the top afterschool programs in the State of California.

7. Study Participants by Role and Afterschool Program

8. Data analysis-Weighing system Step 1 – Establishing the two-thirds rule • To make determinations as to whether the benchmarks/indicators were prevalent at the validation afterschool sites. • Data were analyzed at the site level. Step 2 – Establishing the weighting system • A checklist or numerical scoring tablet was created for each benchmark with all the indicators for that benchmark listed underneath • When all the literature under the benchmark was re-examined, a numerical score was calculated for each indicator • Statistical weighting was applied to give weight to all extracted indicators according to their importance as referenced in the literature • Weekly research team discussions on the appropriateness of assigned weight to the rating score were conducted until consensus had been reached on all items

9. Establishing the core benchmarks • Benchmarks that received a mean score of 7 out of 10 were considered as “core” components of quality afterschool programs • Benchmarks that received a mean score lower than 7 were considered as “additional” or “exemplary” benchmarks that afterschool programs could use to enhance their program quality • staff survey and observation was weighed 2:1

10. Formulating the “Quality Indicator System” Program Organization Management Staff Support Family Involvement Community PartnershipEvaluation Clear mission statement Staff/student input Collaboration with day school Staff-student ratio Staff competency Professional development Cultural diversity Communication with parents Parent involvement Parent feedback Partnership with communities Service projects Plan for community involvement Evaluation of staff performance Program activities Student engagement Student outcomes Continuous improvement Administration Staff Training Program policies Budgeting Sustaining Staff salary Staff feedback Staff orientation

11. Program Organization – Mean Scores, Core and High Quality Benchmarks

12. Program Environment Program Environment Safety Health Physical Space Positive Relationships Clean & secure Prevention strategies Proper supervision Promote healthy habits Nutritious snacks Minimize health risks Safe equipments Sufficient space Proper layout of space Provision for multiple activities Staff: student relationship Student: student relationship Staff: staff relationship

13. Program Environment – Mean Scores, Core, and High Quality Benchmarks

14. Instructional Features Instructional Features Quality of Implementation Variety of Activities Support Youth Development Academics Enrichment Socialization Appropriate activities Student engagement Cultural diversity Meets students’ needs Teaching & learning Opportunities for practice Youth development Personal responsibility Self-direction Leadership

15. Instructional Features – Mean Scores, Core and High Quality Benchmarks

16. Characteristics of the quality benchmark rating system • applicable to all programs serving students of different races, gender, and age groups. • applicable to programs with different program goals and approaches, such as academic achievement, enrichment, etc. • applicable to programs run by different organizations such as school districts, and community-based and religious based institutions, etc.

17. Using the rating system • Each major program component (organization, environment, instructional feature) has its own checklist for quality • If the indicator can be checked off– points will be allocated • In order for a program to a meet the benchmark, it required a minimum score of 7 out of 10 points • For programs that desired further improvement, they could examine the indicators that they did not check off and make plans on improving those weak areas • In order to reduce subjectivity, it was best to have at least 3 raters completing the same instrument • On benchmark ratings that did not reach consensus among the raters, discussions on those particular benchmarks and indicators would reveal insights and pinpoint areas of strength and weakness for program improvements

18. The score sheet