Assessing Social-Emotional Skills at Scale: Recent Developments in Policy, Practice, and Research

1. Martin R. West Harvard Graduate School of Education May 19, 2016 Assessing Social-Emotional Skills at Scale: Recent Developments in Policy, Practice, and Research

2. Policy Context: Required Accountability Indicators under ESSA • Academic achievement: proficiency rates (or alternative?) on state math/ELA tests • Another academic indicator: • High schools: graduation rates • Elementary/middle schools: academic growth (or another indicator of learning) • English language learners’ progress toward proficiency • At least one additional indicator of “school quality or student success”: Listed options include (but are not limited to) student engagement, educator engagement, access to and completion of advanced coursework, postsecondary readiness, or school climate and safety • Each indicator must receive “substantial” weight and the first three indicators together must receive “much greater” weight than the fourth

3. Policy Context: Requirements for “fifth indicator” Valid and reliable: draft regulations require that the indicator be predictive of achievement or graduation rates Statewide: measured in a comparable way across all schools serving the same grade span Disaggregated for schools and student subgroups Meaningfully differentiate school performance: states need use the indicator to assign each school to one of at least three performance levels Limited role (?): draft regulations state that the indicator can’t change identity of schools otherwise identified for comprehensive support

4. CORE Districts • A consortium of nine California school districts serving over one million students enrolled in more than 1,500 schools • Six of these districts have been operating since 2013 under an ESEA waiver to develop a new accountability system • School Quality Improvement Index (SQII) includes: • Academic domain: test score levels and growth; grad rates • Social-emotional domain: absences, suspensions, school climate and culture surveys, social-emotional skills • 2014-15: field test of social emotional measures on ~450,000 students in grades 3-12 • 2015-16: plan to report publicly and include as 8% of SQII ratings

5. Potential concerns with use of self-report surveys for accountability purposes • Reference bias: students’ normative expectations may vary across schools, undermining comparability of responses (West et al., 2015) • Faking: because self-report surveys are notably easy to game, increasing risk of corruption due to stakes (Campbell, 1976) • Imprecision: ability to differentiate school performance when aggregated is unknown (Kane & Staiger, 2002) • Instructional response: superficial measures could lead to superficial pedagogy

6. Selection of SEL Measures • Three criteria: evidence that the skills were (1) measurable (in <20 minutes total), (2) meaningfully predictive of academic and life outcomes, and (3) malleable through school-based interventions

7. The measures were piloted, field tested, and rolled out at scale over the course of three years 2013-14 2014-15 2015-16 ~9000 students participate in pilot of survey-based social-emotional measures All CORE schools administer survey-based social-emotional measures as partof theSQII 450,000+ students participate in field test of survey-based social-emotional measures

8. Internal reliability: full sample

9. Internal reliability: key subgroups

10. Relationship between combined SEL measure and math test scores, CORE middle schools

11. Student-level correlations of SEL measures and math test scores: overall and within-school, CORE middle schools

13. Tentative lessons from CORE • CORE Field Test presents a unique opportunity to learn about the properties of self-report measures of social-emotional skills when administered at scale • Evidence is generally encouraging with respect to reliability(above grade 4) and validity both within and across schools • Ability to differentiate schools with certainty based on average performance is limited, but comparable to that of test scores • Key caveats: Field Test data cannot address… • how the performance of self-report measures would change if stakes were attached; and • how the reporting of measures will change instructional practice and, ultimately, student achievement and life outcomes.

14. Key choice (1): assess status of social-emotional skills directly or the conditions we believe foster them? Key challenge: lack of knowledge of which conditions matter most Key challenge: lack of information on skills driving any differences in performance

15. Key choice (2): where in the pipeline should we focus measurement activity?

16. Alternative uses of measures of social-emotional learning and/or school climate in ESSA • Accountability: the fifth indicator • Transparency: element in school report cards/dashboards • Diagnosis: focus of required “needs assessment” for schools identified for comprehensive/targeted support and improvement • Research & Development: include in state longitudinal data system to facilitate research