The Design Phase: Using Evidence-Centered Assessment Design

1. The Design Phase:Using Evidence-Centered Assessment Design Monty Python argument

2. Workshop Flow • The construct of MKT • Gain familiarity with the construct of MKT • Examine available MKT instruments in the field • Assessment Design • Gain familiarity with the Evidence-Centered Design approach • Begin to design a framework for your own assessment • Assessment Development • Begin to create your own assessment items in line with your framework • Assessment Validation • Learn basic tools for how to refine and validate an assessment • Plan next steps for using assessments

3. The PADI Project – padi.sri.com

4. What is Evidence-Centered Design? • Approach developed by ETS • Uses “evidentiary reasoning” to design the underlying principles of an assessment • Answers these questions: • What complex of KSAs should be assessed? • What types of evidence would we need to show a test-taker has these KSAs? • What kinds of assessment items would allow us to gather this evidence? • Provides a set of tools for structuring a systematic approach to doing this

5. Phases of ECD • Domain Analysis Big-picture narrative of the knowledge domain • Domain Modeling Organization of the information in the domain in terms of: • The aspect of proficiency of the test-taker in the domain • The kinds of things the test-taker might do to provide evidence of their proficiency • The kinds of situations that might make it possible to provide such evidence • Conceptual Assessment Framework Blueprint for the actual assessment that takes all of this into account • The Assessment

6. Phases of ECD for Assessing MKT • Domain Analysis of MKT Big-picture narrative of the knowledge domain Rate of change MKT • Interpreting unconventional forms or representations • Choosing problems and examples that can illustrate key curricular ideas • Differentiating between colloquial and mathematical uses of language • Linking precise aspects of representations • Understanding implications of models and representations • Evaluating mathematical statements y=ax+b Proportionality

7. Phases of ECD for Assessing MKT • Domain Modeling Organization of the information in the domain in terms of: • The aspect of proficiency of the test-taker in the domain • The kinds of things the test-taker might do to provide evidence of their proficiency • The kinds of situations that might make it possible to provide such evidence Create a “Design Pattern”

8. Teacher Model Phases of ECD for Assessing MKT • Conceptual Assessment Framework Blueprint for the actual assessment that takes all of this into account

9. Types of Relationships to Curriculum/PD Assessment Development Curriculum/PD Development Assessment Development Curriculum/PD Development Assessment Development Curriculum/PD Development

10. Adapting Off-the-Shelf Instruments • Start with this process for your own assessment needs • Evaluate the fit of the instrument for your needs • Important: Do not simply mix and match!

11. Attributes of a Design Pattern

12. Examples of Design Patterns • MKT for SimCalc • Co-construct as a group • Do your own

13. Foundationsof proportionality Find the missing number Consider the function Proportionality includes: linearity, rate, function, slope in graphs, interpreting tables with an underlying rate

14. NCTM Curriculum Focal Points Students graph proportional relationships and identify the unit rate as the slope of the related line. They distinguish proportional relationships (y/x = k, or y = kx) from other relationships, including inverse proportionality (xy = k, or y = k/x). Grade 7: Developing an understanding of and applying proportionality, including similarity. Students extend their work with ratios to develop an understanding of proportionality that they apply to solve single and multistep problems in numerous contexts. They use ratio and proportionality to solve a wide variety of percent problems, including problems involving discounts, interest, taxes, tips, and percent increase or decrease. They also solve problems about similar objects (including figures) by using scale factors that relate corresponding lengths of the objects or by using the fact that relationships of lengths within an object are preserved in similar objects. Students graph proportional relationships and identify the unit rate as the slope of the related line. They distinguish proportional relationships (y/x = k, or y = kx) from other relationships, including inverse proportionality (xy = k, or y = k/x). Grade 8: Analyzing and representing linear functions and solving linear equations and systems of linear equations. Students use linear functions, linear equations, and systems of linear equations to represent, analyze, and solve a variety of problems. They recognize a proportion (y/x = k, or y = kx) as a special case of a linear equation of the form y = mx + b, understanding that the constant of proportionality (k) is the slope and the resulting graph is a line through the origin. Students understand that the slope (m) of a line is a constant rate of change, so if the input, or x-coordinate, changes by a specific amount, a, the output, or y-coordinate, changes by the amount ma. Students translate among verbal, tabular, graphical, and algebraic representations of functions (recognizing that tabular and graphical representations are usually only partial representations), and they describe how such aspects of a function as slope and y-intercept appear in different representations. Students solve systems of two linear equations in two variables and relate the systems to pairs of lines that intersect, are parallel, or are the same line, in the plane. Students use linear equations, systems of linear equations, linear functions, and their understanding of the slope of a line to analyze situations and solve problems. They recognize a proportion as a special case of a linear equation of the form y = mx + b, understanding that the constant of proportionality (k) is the slope and the resulting graph is a line through the origin. Students translate among verbal, tabular, graphical, and algebraic representations of functions.

15. KSAs • Common Content Knowledge • Proportionality concepts • Specialized Content Knowledge • Interpreting unconventional forms or representations • Choosing problems and examples that can illustrate key curricular ideas • Differentiating between colloquial and mathematical uses of language • Linking precise aspects of representations • Understanding implications of models and representations • Evaluating mathematical statements

16. Design pattern example#2, Co-constructed

17. Workshop Flow • The construct of MKT • Gain familiarity with the construct of MKT • Examine available MKT instruments in the field • Assessment Design • Gain familiarity with the Evidence-Centered Design approach • Begin to design a framework for your own assessment • Assessment Development • Begin to create your own assessment items in line with your framework • Assessment Validation • Learn basic tools for how to refine and validate an assessment • Plan next steps for using assessments

18. Activity #2Create a Design Pattern • Find Activity #2 in your binder • Pick a design pattern topic • Work on your own, with a partner, or with a small group • Complete the Design Pattern form • Feedback / Review Process • Discussion to follow • Show-and-tell of 2 or 3 Design Patterns • Your insights, questions, challenges

19. Some Useful References(all available on the web) • Baxter, G. P., & Mislevy, R. J. (2005). The Case for an Integrated Design Framework for Assessing Science Inquiry. PADI Technical Report 5. Menlo Park, CA: SRI International. • Embretson, S. E. (Ed) (1985). Test Design: Developments in psychology and psychometrics. New York: Academic Press, Inc. • Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to Evidence-Centered Design. CRESST Technical Paper Series. Los Angeles, CA: CRESST. • Mislevy, R. J, Hamel, L. et al. (2003). Design Patterns for Assessing Science Inquiry. PADI Technical Report 1. Menlo Park, CA: SRI International. • PADI Website: http://padi.sri.com

20. Feedback • What worked? • What didn’t work? • What do you still want to learn?

21. Welcome back! Nicole.shechtman@sri.com Teresa.lara-meloy@sri.com

22. Q1. Definition of MKT “By mathematical knowledge for teaching, we mean the mathematical knowledge used to carry out the work of teaching mathematics. Examples of this “work of teaching” include explaining terms and concepts to students, interpreting students’ statements and solutions, judging and correcting textbook treatments of particular topics, using representations accurately in the classroom, and effects of teachers’ mathematical knowledge on student achievement providing students with examples of mathematical concepts, algorithms, or proofs” (Hill, Rowan, & Ball, 2005). “We look across mathematical knowledge needed for or used in teaching, including “pure” content knowledge as taught in secondary, undergraduate, or graduate mathematics courses; pedagogical content knowledge and curricular knowledge (also described by Shulman) both possibly taught in mathematics methods courses; and what is more elusive, knowledge that, while also mathematical, is not typically taught in undergraduate mathematics courses and is not be entirely pedagogical. Mathematical knowledge for teaching, keeping the emphasis on mathematics and acknowledging that teachers may know and use mathematics that is different from what is required for other professions” (Ferrini-Mundy, et. al., 2008)

23. Question 2 Teacher’s attitudes about being evaluated

24. Q3. Adapting Off-the-Shelf Instruments • Start with this process for your own assessment needs • Evaluate the fit of the instrument for your needs • Important: Do not simply mix and match!