1 / 30

Intuitive Test Theory

Intuitive Test Theory. Robert J. Mislevy CRESST/ University of Maryland September 9, 2004 Presented at the 2004 CRESST Conference, September 9-10, UCLA, Los Angeles, CA. This talk is based on joint work with Henry Braun. Where we’re going. How people think about tests

zlata
Download Presentation

Intuitive Test Theory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Intuitive Test Theory • Robert J. Mislevy • CRESST/ University of Maryland • September 9, 2004 • Presented at the 2004 CRESST Conference, September 9-10, UCLA, Los Angeles, CA. This talk is based on joint work with Henry Braun.

  2. Where we’re going How people think about tests Howard Gardner quote Intuitive physics Scientific test theory Some testing p-prims Conclusion

  3. Howard Gardner, 1993, p. 5 (1) In most domains of knowledge, we develop very powerful theories when we are very young. School and the disciplines are supposed to reformulate those theories and to make them more comprehensive and more accurate. As long as we stay in school, we can maintain the illusion that the effort has succeeded, but ... once we leave school, the illusion disappears and there is a 5-year-old mind dying to get out and express itself...

  4. Howard Gardner, 1993, p. 5 (2) No one has to tell a kid that heavy objects fall more quickly than light objects. It's totally intuitive. It happens to be wrong. Galileo showed that it was wrong. Newton explained why it was wrong. But, like others with a robust 5-year-old mind, I still believe heavier objects fall more quickly than lighter objects....

  5. Howard Gardner, 1993, p. 5 (3) The only people on whom these engravings change are experts. Experts are people who actually think about the world in more sophisticated and different kinds of ways. ... In your area of expertise, you don't think about what you do as you would when you were five years of age. But I venture to say that if I get to questioning you about something that you are not an expert in, the answers you give will be the answers you would have given before you had gone to school.

  6. Intuitive Physics • diSessa’s “p-prims” • Heavy objects fall faster than light objects. • Things bounce because they are 'springy’. • An object moves in a direction if and only if there’s a force on it in that direction. • Chi, Feltovich & Glaser’s problem-sorting • springs & pulleys vs. equilibrium & Newton’s laws

  7. “Scientific” Test Theory (1) • Items and scores are the springs and pulleys of testing. • Assessment as evidentiary reasoning. • Want to know what students know, can do, or have accomplished, • from a handful of particular things they’ve said, done, or made.

  8. “Scientific” Test Theory (2) • Probability models connect what you see with what you make inferences about. • Quantify accuracy and error. • Play “what if” games for design. • Quality control (e.g., large system w. raters) • Can tell when you’re wrong. • Can be any kind of data. • Can be any kind of knowledge or skill.

  9. Testing P-Prims

  10. A test measures what it says at the top of the page.

  11. Any two tests that measure the same thing can be made interchangeable, with a little equating magic

  12. A score is a score is a score.

  13. You score a test by adding up scores for items.

  14. 93% is an A, 85% is a B, 78% is a C, and 70% is passing.

  15. Multiple-choice questions only measure recall.

  16. What are the forces at the instant of impact? 20 mph 20 mph • A. The truck exerts the same amount of force on the car as the car exerts on the truck. • B. The car exerts more force on the truck than the truck exerts on the car. • C. The truck exerts more force on the car than the car exerts on the truck. • D. There’s no force because they both stop.

  17. What are the forces at the instant of impact? 10 mph 20 mph • A. The truck exerts the same amount of force on the car as the car exerts on the truck. • B. The car exerts more force on the truck than the truck exerts on the car. • C. The truck exerts more force on the car than the car exerts on the truck. • D. There’s no force because they both stop.

  18. What are the forces at the instant of impact? 10 mph 1 mph • A. The truck exerts the same amount of force on the fly as the fly exerts on the truck. • B. The fly exerts more force on the truck than the truck exerts on the fly . • C. The truck exerts more force on the fly than the fly exerts on the truck. • D. There’s no force because they both stop.

  19. Technology will solve testing problems by making it possible to get voluminous amounts of data. Cisco example, if time allows

  20. Conclusion (1) • Problems: • Assessment policy is based on intuitive theory. • Intuitive test theory is used with new forms of testing, like simulations.

  21. Conclusion (2) • Three ways of dealing with it: • Not-fun way: Tell people what they’ve already planned is wrong, and why. • Good way: Be involved in project planning early on. • Good & fun way: Existence-proof projects that demonstrate what can be done, outside familiar forms, using scientific test theory.

  22. Conclusion (2)

  23. An example: Cisco Learning Institute’s NetPass Prototype • Create on-line performance assessment of networking skills • Focus on learner feedback rather than high-stakes testing

  24. A text file is scored following detailed rules, which result in characterizations of the work

  25. Conclusion (2)

More Related