Designing a Pool of Items

1. Designinga Pool of Items

2. Workshop Flow • The construct of MKT • Gain familiarity with the construct of MKT • Examine available MKT instruments in the field • Assessment Design • Gain familiarity with the Evidence-Centered Design approach • Begin to design a framework for your own assessment • Assessment Development • Begin to create your own assessment items in line with your framework • Assessment Validation • Learn basic tools for how to refine and validate an assessment • Plan next steps for using assessments

3. Develop Pool of items Refine items Collect/ Analyze Validity Data Refine items Assessment Development Process Define item Template (Define Test Specs) Assemble Test Define item Specs Domain Analysis Document Technical Info Domain Modeling (Design Pattern)

4. From Design to Development • Use information from the Design Pattern to generate items • (Use the process of generating items to refine the Design Pattern)

5. Item Development Steps Based on the foundational design work… • Content experts develop draft items • Refine draft items – “hygiene” • Collect validity data on draft items • Use validity data to refine items • Assemble and document the assessment

6. Assessment Item Anatomy Multiple Choice Stem Essential parts of a butterfly are… • Wings, legs, teeth • Eyes, wings, legs • Nose, knees, ears • Fingers, frontal lobes, hair Butterflies are members of the _______ phylum. What are the closest relatives of the butterfly, and why? Choices Distracters Constructed Response (Closed-Ended) Prompt Constructed Response (Open-Ended)

7. Conducting Hygiene Reviewis essential • Clarity and precision • Accuracy • Grammar • No ambiguity • No unintended cues • Distracters believable, appropriate, same length • Placement of correct response varied

8. SimCalc Not a Trivial Process!

9. Other Resources forHunters and Gatherers • Standardized tests • International • PISA (http://nces.ed.gov/surveys/pisa/ ) • TIMSS (http://timss.bc.edu/ ) • National • NAEP (http://nces.ed.gov/nationsreportcard/ ) • State • TAKS released items

10. Other Resources forHunters and Gatherers • Other resources • Established curricular materials • Approved textbooks • TPD materials • Supplemental teacher resources • SRI’s online resources • Performance Assessment Links in Math (PALM): http://palm.sri.com • Integrated Performance Assessments in Technology: http://ipat.sri.com • Online Evaluation Resource Library (OERL): http://oerl.sri.com • GLOBE assessment tools: http://globeassessment.sri.com • Research literature (e.g., theoretical papers that provide example items) • Other research-based assessments • Mental Measurements Yearbook from the Buros Institute of Mental Measurements

11. Welcome to(mini) Item Camp!

12. Goals • Develop 1 to 3 draft items for your (or your neighbor’s) assessment • Have an experience of participating in an item camp

13. Starting with Item Templates • What are templates? • Templates are industry standard in assessment development • Templates outline the structure of an item, you fill in the content • Guidelines to systematically produce high-quality items • What are these templates? • Created based on the SimCalc design pattern to help seed thinking about the SimCalc items • Templates might be quite different for your design pattern • They are based on our observations about the structure of items that fit into types and “worked” in field testing • Heuristic tool, not theoretical framework • In this context, recommended but not required

14. Template Types Based onDomain Analysis of MKT • Unconventional forms or representations • Choosing problems and examples that can illustrate key curricular ideas • Differentiating between colloquial and mathematical uses of language • Linking precise aspects of representations • Understanding implications of models and representations • Evaluating mathematical statements

15. Structure of the Templates All multiple choice, with the following elements: • Context • Sets up a teaching task • Presents information necessary to answer the question • Question/Prompt • Distracters

16. Review Templates

17. Guidelines for distracters Be judicious! • Usually 4 or 5 distracters (may be more) • May use “Choose one” or “Choose all that apply.” • Responses may include “All of the above” and “None of the above.” • Whenever possible, distracters will reflect common errors or misunderstandings, naïve pre-conceptions, or other misconceptions. • Teachers should not be able to rule out a distracter or identify the answer simply because of superficial or trivial characteristics, syntactic complexity, or concept complexity. • Distracters will not be partially correct responses nor will they be designed to “trick” teachers in responding incorrectly.

18. Other Item Types • True/false • Matching • Open-ended • Calculations • Short responses • Generating representations • Generating explanations, stories, etc. Require a rubric for scoring

19. Filling Out the Item Form • Name • Connection to standard or curriculum • KSA assessed • Item • Correct answer and/or rubric • Notes

20. Consider this a brainstorm, so do not be overly concerned about IH However, keep in mind that this is the next part of the pipeline IH includes: Clarity, precision, correctness Grammar Non-ambiguity No unintended cues Distracters believable, appropriate, same length Placement of correct response varied “Item Hygiene”

21. Draw on Resources Available • Your Design Pattern • Your curriculum materials • MKT resources (e.g., prior assessments) • NCTM and Texas standards

22. Activity #3Mini-Item Camp • Find Activity #3 in Tab 5 • Work on your own, with a partner, or with a small group • Develop some MKT items (1 to 3) • Prepare 1 or 2 items for the Item Hygiene Review (see page 7) • On the item form, include: • Name • Connection to standard or curriculum • KSA assessed • Item • Correct answer and/or rubric • Notes

23. Time for Activity: 9:30-10:15 • On the item form, include: • Name • Connection to standard or curriculum • KSA assessed • Item • Correct answer and/or rubric • Notes

24. Activity #4 • Find Activity #4 on Tab 5 • Sit in groups of 3 for Hygiene Review • Use the Item Hygiene Guide to edit your teammates items • Fix your 1 or 2 items and prepare a clean copy for this afternoon’s activity • Discussion to follow • Show-and-tell of 2 or 3 items • Your insights, questions, challenges

25. Activity #4Conduct Item Hygiene Review • Conduct item hygiene on your set of items, using the Item Hygiene Guide. We suggest writing corrections directly on the item. • Following, we will discuss • Insights about development of assessment items • Questions and challenges

26. Some Useful Resources • Anderson, J. R. (2000). Cognitive psychology and its implications (3rd Ed.). New York: W. H. Freeman and Company. • Baxter, G. P., & Glaser, R. (1998). The cognitive complexity of science performance assessments. Educational Measurement: Issues and Practice, 17, 3, 37-45. • Pellegrino, J., Chudowsky, N., Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press. • Quellmalz, E., Hinojosa, T., & Rosenquist, A. (2001). Design of student assessment tools for the Global Learning and Observations to Benefit the Environment (GLOBE) program. Presentation at the annual GLOBE International Conference, Blaine, WA. • Snow, R. E., Federico, P. A., & Montague, W. E. (Eds.). (1980). Aptitude, learning, and instruction: Volume 2. Cognitive process analyses of learning and problem-solving. Hillsdale, NJ: Erlbaum.