High Quality Items for High Stake Tests

1. High Quality Items for High Stake Tests Delhi, 20 November, 2015

2. Consequences of using flawed test items • High-stakes decisions are based on test performance on a sample of items. • The quality and effectiveness of test items is determining factor of the quality of test. • The world is awash in poorly written items. • Trivial - Ambiguous • Irrelevant difficulty - Flaws benefitting testwise candidates • Low cognitive level - Window dressing • Extremely difficult or easy items - Inflammatory, offensive or upsetting It has been shown that flawed items are up to 15 percentage points more difficult than standard items testing the same content, with the median passing rates about 3.5 percentage points more difficult than standard items.

3. Validity What constitutes a high-quality test item? The ideal item assesses appropriate content accurately, consistently and fairly Testing Tenants: Focus on Measurement • Fairness • Reliability • - How well item measures the subject matter • - Whether the items address what the successful • candidates need to know • What the incorrect answers are measuring • + Are the items clearly written and unambiguous • + Does the item include content relevant to testing • Is the wording of the item free from sensitive issues • Is the content of the item causing invalid differences between groups The principles of effective writing are well delineated. These principles, based on a consensus of the educational measurement of textbook authors and empirical research studies can be considered the ‘evidence-based’ principles of effective item writing.

4. Item writing is both an art and science • The art of item writing: • Understanding the eligibility requirements to sit for the test • Skillful application of item writing principles to the content • Creativity • The science of item writing: • Knowledge of item writing techniques • The ability to follow item writing best practices • A sense of how item writing fits into the test development process. The creation of effective test items designed to measure content at an appropriate cognitive level is a nontrivial skill, requiring a great deal of professional time and effort from content experts and test developers.

5. Effective items are created, edited, reviewed, and analyzed • Characteristics and traits of effective item writers: • - Content expertise - Writing skills • Effective item writers are trained not born: • - Unique skill and must be learned and practiced • - Extensive knowledge of evidence-based item writing principles • Item editing is a craft which improve the quality of item: • - Improves the appearance, readability and fairness • - Item writer and professional editor work as a team • Item review by expert group to verify that • - It assesses appropriate and important information and is accurate • - Keyed answer is absolutely correct It is nearly impossible to estimate how an item will perform until it is actually used in a test. Item analysis provides information on what type of items tend to work well and what types tend not to work well in a particular test.

6. Thank you.