Vertically Articulated Performance Standards

1. Vertically Articulated Performance Standards An Exploratory Examination of their Interpretability Steve Ferrara, Gary Phillips, Paul Williams, and Shannon Mahoney American Institutes for Research October 19, 2006

2. Overview for this talk • Background and purpose of the study • Vertically articulated standards • Definition, procedures • The assessment program • This study • Procedures • Results, interpretation, implications • Conclusion Vertically Articulated Standards (VAS)

3. Purposes of the study • Evaluate a system of articulated standards that we created for a state content area assessment • Examine definitions of Proficient performance and achievement growth as defined by test items around the Proficient cut scores • Consider implications for the panelist’s judgmental task in selecting cut scores and articulating standards Vertically Articulated Standards (VAS)

4. Vertically articulated performance standards • Coherent system of performance standards • When performance standards are articulated, they form a system of standards in an orderly progression across grades in the same content area and across content areas • An orderly progression of standards is indicated by the percentages of examinees at/above each performance level in each grade Vertically Articulated Standards (VAS)

5. Vertically Articulated Standards (VAS)

6. Vertically Articulated Standards (VAS)

7. Importance of vertically articulated standards • Discrepancies can cause confusion in the general public and media and enable misinterpretations of standards and performance • Misinterpreting cross-grade differences in performance standards as achievement differences • Mistargeting resources to a grade level or content area • Blaming or rewarding teachers Vertically Articulated Standards (VAS)

8. The assessment program • Typical state assessment program (e.g., MC and CR items) • Ambitions well beyond NCLB requirements (e.g., K-2 diagnostic assessments, other content areas, accountability system, growth modeling) • Reading and math, grades K-8 plus high school • Science, social studies, and writing in an elementary and middle school grade and at high school level Vertically Articulated Standards (VAS)

9. The assessment program (cont.) • Within grade assessments aligned with grade level content standards • Within grade scaling • Vertical articulation of standards across the within grade scales • Vertical scale for future score reporting (not considered in standard setting) • Linking items included from the lower adjacent grade only (“down linking”) • Previous year, math: Higher linking error when include upward linking items • Joint calibration, grades 3-8; also chain linking, other tryouts Vertically Articulated Standards (VAS)

10. Setting cut scores within grades • Mathematics grades 3-8 • Separate 3-5 and 6-8 panels • Reading grades 4-5 and 6-8 • Bookmark method • “Place your bookmark on the page that (approximately) two-thirds of those students who are just barely Proficient would be able to answer successfully” • RP criterion • Just Barely Proficient students Vertically Articulated Standards (VAS)

11. Proficiency level descriptions (PLDs) • In this program, the PLDs define Proficient performance explicitly in terms of growth from the previous grade Vertically Articulated Standards (VAS)

12. Grade 3 Mathematics PLD • Students performing at the Proficient level (1) show adequate progress by using grade 3 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) use place value concepts, apply basic measurement and geometry concepts to describe attributes of shapes or objects, and interpret graphs. They usually can use informal reasoning and make appropriate decisions about what procedure to use to solve routine problems. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language. Vertically Articulated Standards (VAS)

13. Grade 4 Mathematics PLD • Students performing at the Proficient level (1) show adequate progress by using grade 4 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) solve routine problems involving whole numbers, decimals and simple fractions; describe perimeter and area; compare geometric figures; write an equation to describe a situation; and describe data. They usually can use informal reasoning and make appropriate decisions about what procedure to use to solve routine problems. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language. Vertically Articulated Standards (VAS)

14. Grade 5 Mathematics PLD • Students performing at the Proficient level (1) show adequate progress by using grade 5 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) solve routine problems involving sums and differences of fractions or decimals, measuring angles, describing the probability of events, and converting units in the same measurement system. They typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students use informal and some formal reasoning to evaluate and justify the reasonableness of a solution. They communicate mathematical thinking and solutions using a combination of informal and mathematical language. Vertically Articulated Standards (VAS)

15. PLDs for Proficient • Each definition refers to growth from the previous grade • Each describes articulation of content and skill requirements across grades 3, 4, and 5 • Some features are constant across these grades Vertically Articulated Standards (VAS)

16. Achieving vertically articulated standards • Set cut scores for anchor grades in typical way • For interim grades, provided the target cut score (i.e., page number) that would provide an orderly progression of standards across grades • We asked panelists to try to make the bookmark judgment within three or so pages of the target Vertically Articulated Standards (VAS)

17. Achieving vertically articulated standards (cont.) • After all standards were set, table leaders participated in a moderation process to achieve articulated standards • Guidance: Cut scores that corresponded to orderly percentages of students reaching Proficient (and other levels) in each grade • Within content area, then across content areas • They chose not to adjust one standard Vertically Articulated Standards (VAS)

18. This study: Analytic procedures • Identified items that define Just Barely Proficient performance • Examined the knowledge and skills they • Target • Require for successful responses • We refer to this as Item Demands Analysis • Summarized and compared the knowledge and skill demands Vertically Articulated Standards (VAS)

19. Analysis (cont.) • Specifically, we focused on the items that define Just Barely Proficient performance at grades 3 and 4 • We examined those items and how they define: • Just Barely Proficient performance for standard setting panelists • Growth in achievement Vertically Articulated Standards (VAS)

20. Analysis (cont.) • We used mathematics in grades 3 and 4 from a state assessment program • We focused on the standard for Proficient performance, established using the Bookmark procedure • We focused on the concept of the Just Barely Proficient student because this is the concept that panelists use to make the cut score judgment Vertically Articulated Standards (VAS)

21. Analysis (cont.) • Items: RP 67 locations on the vertically linked scale • Just Barely Proficient: Items at the cut score, three items below, three items above Vertically Articulated Standards (VAS)

22. Example: grade 3 mathematics Vertically Articulated Standards (VAS)

23. Item demands analysis • Targeted content standards • Identified in the state content standards and test blueprints (see App. A) • Our judgments about: • Reading load • Content area knowledge demands • Content area skill demands/question types Vertically Articulated Standards (VAS)

24. Definitions: Reading load • High • Lots of text, complex text, interrelatedness of elements in the text, complex format • Moderate • Multiple steps, short phrases, scaffolded format • Low • Small amount of text, mostly mathematics, how to proceed and respond is explicit or obvious Vertically Articulated Standards (VAS)

25. Definitions: Content area knowledge • Prior knowledge • Supply or apply an arithmetic fact or taught algorithm • Conceptual understanding • Use or explain a mathematical concept • Recognition • Recognize and complete a numerical or other pattern, recognize a visual representation of a mathematical concept or procedure Vertically Articulated Standards (VAS)

26. Definitions: Content area skills • Use/apply • E.g., visual information given with the item • Answer and explain • E.g., defend the answer given • Analyze, categorize, hypothesize • E.g., observe and describe patterns in data, procedures, or results • Empirically supported (Ferrara, Duncan, et al., 2004); adapted from science Vertically Articulated Standards (VAS)

27. Selected results: Grades 3 and 4 Vertically Articulated Standards (VAS)

28. Selected results: Grades 3 and 4 cut scores • Standards targeted • More number sense at grade 3, more geometry at grade 4 • Scale location of cut scores and JBP items • CS grade 3 = -0.58, CS grade 4 = -0.01 • Locations grade 3 = (-0.76, -0.49), grade 4 = (-0.13, +0.26) • Item types • Similar Vertically Articulated Standards (VAS)

29. Selected results: Grades 3 and 4 (cont.) • Content area skills • Similar • Reading load • Grade 3 low-moderate • Grade 4 moderate-high Vertically Articulated Standards (VAS)

30. Content area knowledge requirements Vertically Articulated Standards (VAS)

31. Summary of results: Grades 3 and 4 • Item demands on JBP students similar for the grade 3 and 4 cut scores except in three areas • Location of cut score on vertical scale • Content area knowledge requirements • Reading load • Item demands for grade 3grade 4 scale • Similar to grade 3 except more emphasis on prior knowledge requirements Vertically Articulated Standards (VAS)

32. Interpretation of results: Three areas • System of articulated standards • Definitions/interpretations of what it means to be Proficient and what is growth in achievement • Implications for setting and articulating standards • We are interested in practical steps for setting performance standards that: • Are vertically articulated • Enable valid interpretations about what students know and can do as they progress through school Vertically Articulated Standards (VAS)

33. Interpretation: System of articulated standards • Articulating standards across grades using statistical considerations can enable meaningful interpretations from a content area demands point of view • Content knowledge, reading load, and difficulty demands increased in meaningful ways across grades 3 and 4 for this mathematics assessment • That increase is small and consistent with the grade level means Vertically Articulated Standards (VAS)

34. Student proficiency estimates, vertically linked theta scale Vertically Articulated Standards (VAS)

35. System of articulated standards (cont.) • A concern: The articulation worked out well, but it did not happen completely by design • Content standards, PLDs, and performance standards are articulated by design • Item locations are not articulated by design • Item writers aim at difficulty targets but cannot/do not control empirical item difficulty • A point being made often: Scientifically principled test design and development and item difficulty modeling • E.g., Ferrara & DeMauro, 2006; Gorin, 2006; Mislevy & Haertel (2006) Vertically Articulated Standards (VAS)

36. Interpretations of Proficient and growth in achievement • Articulating standards in a meaningful way enables inferences about growth in achievement; however, • Typical PLDs illuminate what that growth entails only in a very general way • “Students performing at the Proficient level show adequate progress by using grade 3 concepts and skills to solve familiar problems. …” Vertically Articulated Standards (VAS)

37. Interpretations of Proficient and growth in achievement (cont.) • Meaningful inferences about grade-to-grade growth in achievement require that four elements are articulated: • Content standards that are the basis for test design and development • PLDs that are the performance standards • The cut scores that effect those standards • The items around a cut score that panelists use to define the cut scores Vertically Articulated Standards (VAS)

38. Implications for setting and articulating standards • In item mapping procedures, panelists consider item demands to locate their cut scores • They may or may not consider whether the items they examine will result in articulated cut scores • Even when we give them articulation information • It seems likely that they do not • Consider using item demands summary tables as input to: • The standard setting process • Subsequent moderation processes to articulate standards Vertically Articulated Standards (VAS)

39. Conclusion • Exploratory examination; we need to look at additional grades and content areas • It seems unlikely that things will work out as fortunately in all grades and content areas Vertically Articulated Standards (VAS)

40. References Ferrara, S., & DeMauro, G. E. (2006). Standardized assessment of individual achievement in K-12. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: American Council on Education/Praeger. Ferrara, S., Duncan, T. G., Freed, R., Velez-Paschke, A., McGivern, J., Mushlin, S., Mattessich, A., Rogers, A., & Westphalen, K. (2004). Examining test score validity by examining item construct validity: Preliminary analysis of evidence of the alignment of targeted and observed content, skills, and cognitive processes in a middle school science assessment. Paper presented at the annual meeting of the American Educational Research Association, San Diego. Gorin, J. (2006 in press). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25 (4). Mislevy, R. J., & Haertel, G. D. (2006 in press). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25 (4). Vertically Articulated Standards (VAS)

41. Appendix A: Content Standards • Number, Number Sense and Operations Standard • Measurement Standard • Geometry and Spatial Sense Standard • Patterns, Functions and Algebra Standard • Data Analysis and Probability Standard • Mathematical Process Standard Vertically Articulated Standards (VAS)