200 likes | 589 Views
Overview of the Rasch Measurement Model. Overview. Exam integrity, Limitations of Classical Test Theory (CTT), Advantages of the Rasch model, The partial credit model, Banking and equating, and. Challenges facing certification: Exam integrity. Integrity… Reliability, Validity, and
E N D
Overview • Exam integrity, • Limitations of Classical Test Theory (CTT), • Advantages of the Rasch model, • The partial credit model, • Banking and equating, and
Challenges facing certification: Exam integrity • Integrity… • Reliability, • Validity, and • Security • … of exam content • Controlling for compromises to integrity… • Statistically, and • By design
Classical Test Theory • Basis of conventional measurement methods. • Reliability (i.e., proportion of shared variance between items) • Item difficulty (i.e., proportion answering an item correct) • Item discrimination (i.e., how well items correlate with total score) • Based on a random (and hopefully representative) sample from the population.
Limitations of CTT • Item parameters are sample dependent. • Typically we don’t know whether our sample is truly representative. • Test calibration based on these groups would be misleading.
The Rasch measurement model • A persons’ measure on any trait is a simple function of their ability and the item’s difficulty. • Person ability can be conditioned (removed) from sample estimates making item calibration sample free.
Rasch Measurement: Sample-free equivalent estimates • The two characteristic curves are almost indistinguishable. • Our best estimate on how any one person would perform on the test – we’ve conditioned person ability from the sample.
Hi/Lo Scatter A scatter plot of item parameter estimates should converge around a straight line, with tighter convergence in the middle and more scatter at the extremes.
Advantages of the Rasch model • Used in conjunction with CTT • Item calibration is sample free • Sample sizes… • …as small as 100 examinees (generating standard errors about .12 logits) are generally enough. • Item banks and reliability • Since items can be transformed onto a common scale, they can be replaced with equivalent ones without jeopardizing the reliability of the exam. • Linking tests • Traditional test equating becomes obsolete. A linking system can be developed that can be translated into equivalent forms of an exam.
The Partial Credit Model • The number of steps into which an item is divided and their relative difficulties can vary between items. • The points must be awarded sequentially, so that each increasing value represents an increase in the trait being measured (0 = totally wrong, 1 = partially correct, 2 = almost completely correct, and 3 = completely correct).
Item banking • Calibrations are obtained by applying a probabilistic model for what should happen when an examinee attempts an item. • Comparability is maintained because any test formed from bank items is automatically equated onto the common scale.
Test equating • Links of common items can be embedded in different tests. • A translation constant is computed • Passing standards can be adjusted based upon the translation constant Form A • Separate samples • Relatively few link items • Requires pilot testing through item seeding 10 Link Items Form B
Test equating T=-2.0