Standards, Assessments, and Accountability National Academy of Education White Paper

1. Standards, Assessments, and Accountability National Academy of Education White Paper Lorrie A. Shepard University of Colorado at Boulder Performance Assessment Briefing Washington, DC April 13, 2010

2. University of Colorado at Boulder/ School of Education Standards, Assessments, and Accountability Shepard, Hannaway, & Baker (2009), http://www.naeducation.org/ National Academy of Education Working Group on Standards, Assessments, and Accountability:

3. University of Colorado at Boulder / School of Education • Main Points • History of Standards-Based Reform and Its Theory of Action • Research on Teaching-the-Test Effects • Importance of Curricular Coherence • Significance of Performance Assessments for Recommendations 1 and 4 • Mention of Recommendations 2 and 3

4. University of Colorado at Boulder / School of Education RECOMMENDATION 2: The federal government should support research on accountability system indicators… …reflect both status and growth …set performance standards that are ambitious but realistic …communicate to the public …and students what is to be learned …report assessment results along the entire achievement continuum (not just at the cutpoint) RECOMMENDATION 3: The federal government should support the redesign and ongoing evaluation of accountability systems… …so that they contribute to school improvement …use test results to trigger closer investigations …verify results before sanctions or rewards …consider audit assessments, expert inspectorates, etc.

5. University of Colorado at Boulder / School of Education Testing, Teaching, & Learning Elmore & Rothman (1999), NRC 1994 Goals 2000: vision of coherent capacity building 1994 ESEA set forth an incentives theory of change – assumed that with sufficient motivation, teachers and other school personnel would find the means to improve instruction. Unfortunately, implementation studies then and now show that many schools do not understand the changes that were needed and lack capacity to make them happen.

6. University of Colorado at Boulder / School of Education • 20 years of standards have produced gains, especially in mathematics, but gains on NAEP are modest compared to state tests and have not accelerated under intense NCLB pressure. • Negative Effects of Teaching the Test • Curriculum distortion • Test-score inflation • Many policymakers believe that curriculum narrowing is a necessary trade off because reading and math skills are so essential. • But, a close look at research on test-score inflation shows that students cannot apply skills that they appear to know on the test, and • Excessive drill on worksheets resembling the test denies students opportunity to understand context and purpose that would enhance skill development.

7. University of Colorado at Boulder/CRESST 1. A series circuit consists of three identical light bulbs connected to a battery as shown here. When the switch S is closed, do the following increase, decrease, or stay the same? (a) The intensities of bulbs A and B (b) The intensity of bulb C (c) The current drawn from the battery (d) The voltage drop across each bulb (e) The power dissipated in the circuit 2. For the circuit shown, calculate (a) the current in the 2-Ω resistor and (b) the potential difference between points P and Q. Mazur (1997)

8. University of Colorado at Boulder / School of Education • The Importance of Curriculum • The NAEdStandards paper argues for the concurrent, coherent design of curriculum, assessments (both large-scale and classroom level), and teacher professional development. • Top performing countries in TIMSS have leaner, more hierarchically sequenced curricula leading to progressively more advanced topics and deeper understanding. (Schmidt et al., 2005) • Recent analysis of content standards in 14 states found they did not focus on big ideas or build from grade to grade. (Porter et al., 2009). Common Standards are a help but are still only a skeletal framework. • National control is not required for coherence (Schmidt & Prawat, 2006). Rather, coherence leads to effective outcomes if it is achieved at whatever level of governance has authority over policy instruments.

9. University of Colorado at Boulder / School of Education RECOMMENDATION 1: The federal government should encourage the redesign and clear connection of content and performance standards – and the curricula, teacher training, and high-quality assessments to go with them – with the goal of clearly articulating expected progressions of learning. • Instead of turning Standards over to test makers for implementation, the NRC report on state science assessments calls for: • Horizontal coherence (linking curriculum, instruction, and assessment) • Vertical coherence (shared vision at classroom, school, district, and state levels) • Developmental coherence (taking account of how understanding develops over time) •  To achieve coherence, state consortia might consider concurrent development of standards and assessments, or at least prototypes of assessment tasks, anchored to learning progressions.

10. Learning progressions or Progress Maps provide an underlying model of learning to coherently link classroom and large-scale assessments. “Progress maps describe skills, understandings, and knowledge in the sequence in which they typically develop: a picture of what it means to ‘improve’ in an area of learning.” (Masters & Forster, 1996). Learning progressions provide a substantive rather than a quantitative picture of developing competence. A criterion-referenced growth model. Learning Progressions Knowing What Students Know (2001)

11. Learning progressions are an effective tool for teacher professional development. How learning typically unfolds helps teachers know “what next” and how to “back-up” (though we must also be aware of natural variations and departures from the typical pattern). When embedded in units of instruction LPs can target typical misconceptions. Substantive and statistical growth trajectories are not the same, but with care they could be brought together. While much research is still needed, it is possible to start with judgmentally set sequences and then improve empirically. ooooo oooo oooo oooo oooo oooo oo oo o

12. An example of a typical misconception and challenge question: What happens if you cut the nail in half? - + - + - + - + + - - + + - Otero, Jalovec, & HerManyHorses. (2008).

13. University of Colorado at Boulder / School of Education RECOMMENDATION 4: The federal government should support an intensive program of research and development to create the next generation of performance assessments explicitly linked to well-designed content standards and curricula. Rather than alignment, embodiment might be a better term to characterize the more complete and substantive alignment that occurs when the tasks, problems, and projects in which students are engaged represent the range and depth of what we want students to understand and be able to do. Assessment tasks convey what’s important to learn as well as providing an opportunity to check on students’ understanding and evaluate achievement. New technologies make it possible to tap complex and dynamic aspects of cognition and performance. When conversations about student learning are engaged around rich tasks, there is greater opportunity for instructional insights.

14. University of Colorado at Boulder / School of Education To truly transform learning opportunities in classrooms, in ways consistent with what we know is possible from research, we will need to remove policy structures especially low-level tests that misdirect effort, provide coherent curricula consistent with ambitious reforms, and take seriously the need for capacity building at every level of the educational system.