Harnessing Untapped Potential: Using Assessment Data to Inform Next Generation Assessments

1. Harnessing Untapped Potential: Using International and National Assessment Data to Inform the Transition to Next Generation Assessment Systems Laura Egan, NAEP Support and Service Center, Westat Pamela Byrd, Arkansas Department of Education Mark DeCandia, Kentucky Department of Education Kate Beattie, Minnesota Department of Education Council of Chief State School Officers 2014 National Conference on Student Assessment

2. Outline of the Session • Overview of the Literature on Policy Uses of International Assessments (Jason) • Mapping the Terrain: Assessment Landscape and Policy Priorities (Pam) • The Kentucky Case (Mark) • The Minnesota Case (Kate)

3. Background • Ongoing project • Different subprojects in each state • Growing interest in non-state assessments • Shaped by different contexts of each state • Rapidly changing terrain

4. International Assessments in the Academic Literature • Increasing interest in international assessments • Come from different disciplinary backgrounds • Use different methodologies • Use different assessment data • TIMSS, PIRLS, PISA, regional assessments • Special studies (e.g., 1995 TIMSS International Curriculum Analysis, 1995 and 1999 Video Case Studies) • Variety of studies • Focus on different (sets of) countries

5. International Assessments in the Academic Literature • Example from the Literature • Growing body of literature linking performance on international assessments to economic performance

6. Relationship between system-level student achievement and economic performance • Assessments do not directly measure delivery of curriculum or the impact of an educational system given their design, cross-national nature • Instead, assessments like TIMSS, PIRLS, and PISA provide measures of cognitive skills, which is an indicator of human capital

7. Human Capital • Education, most often in the form of schooling, raises individuals’ productive value by providing knowledge, skills, values, and a way of analyzing problems (Becker, 1993) • Investments in education should therefore produce financial returns at the individual (e.g., salary) and societal (e.g., GDP) level

8. Human Capital • Often measured in terms of years of schooling, but this assumes that • 1 year of schooling delivers same increase in knowledge and skills regardless of education system • Formal schooling is the primary source of education and variations in the quality of non-school factors have negligible effect on education outcomes (Hanushek and Woessman, 2012) • Relationship between years of schooling and economic growth is loose, ambiguous

9. Assessment Data as a Measure of Cognitive Skills • International assessments can provide a better measure of human capital through measurement of cognitive skills – what kids know and can do • Human capital does not have to be gained through schooling, as long as effect is the same • Assessments provide data at a system level, so use GDP as economic indicator • Can combine TIMSS, PISA, PIRLS, regional assessments, prior assessments onto single scale with common standard deviation

10. Cognitive skills and economic growth • Hanushek and Kimko (2000), Hanushek and Woessman(2009, 2011, 2012) use econometric modeling to control for a number of variables, rule out omitted variables, etc., and demonstrate • “growth rates changing in a manner consistent with changes in cognitive skills” (Hanushek and Woessman, 2011, p 296) • Pattern is seen over time, across regions

11. Educational performance and economic growth across OECD Countries (Hanushek & Woessman, 2012)

12. Educational performance and economic growth across World Regions. (Hanushek & Woessman, 2012)

13. Conclusions • International large-scale assessments measure something that matters • Suggest that there is an economic value in reforms that lead to increased knowledge and skills as measured by these assessments • But…policy path is not immediately clear

14. Policy implications of international assessment data • Provide a great deal of policy-relevant information on the • context in which learning occurs • cross-national patterns in instructional practices • instructional practices and curricular focus of high performers (and low performers) • Do not provide policy prescriptions or demonstrate causal relationships between aspects of education system and its performance

15. Policy implications of international assessment data • Used to to advocate for change • Strong focus on curricular change - more rigor, focus, coherency • Findings/advocacy not always supported empirically

16. Implications for State Assessment Policy • Comparative data on instructional practices, which can be correlated to performance on cognitive items, may be useful for • starting conversations • identifying avenues for future research • identifying practices to be explored • Empirical relationship between assessment scores and economic growth fosters conversation with stakeholders outside of education arena

17. Implications for State Assessment Policy • Important to consider • Different assessments measure different constructs with different purposes • State contexts differ • System performance • Policy context