1 / 14

Lesson Seven

Lesson Seven. Item Analysis. Contents. Item Analysis - “investigates the performance of items considered individually either in relation to some external criterion or in relation to the remaining items on the test” (Thompson & Levitor, 1985, p. 163) Item difficulty (item facility)

driscoll-roman
Download Presentation

Lesson Seven

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lesson Seven Item Analysis

  2. Contents • Item Analysis - “investigates the performance of items considered individually either in relation to some external criterion or in relation to the remaining items on the test” (Thompson & Levitor, 1985, p. 163) • Item difficulty (item facility) • Item discrimination • Distrctor (efficiency) analysis Video http://www.le.ac.uk/education/testing/ilta/faqs/main.html

  3. Item Difficulty (1) • Item difficulty is simply the percentage of students taking the test who answered the item correctly. The larger the percentage getting an item right, the easier the item. • Also called item facility (Bailey 132-33) • Difficulty index: P or IF • In terms of how many people got this item correct (i.e., the percentage of Ss who correctly answer a given item).

  4. Item Difficulty (1) • P= The number of correct responses divided by the number of total test takers. • P= Ncorrect / Ntotal = (%U+ %L) / 2 • An item answered correctly by 85% of test-takers would have an item difficulty, or p-value of .85. However, an item answered correctly by 50% of the test-takers would have a lower item difficulty, or p value of .50

  5. Item Difficulty (2) • 0 ≤ P ≤ 1 (0= difficult item, 1= easy item) • Achievement Test (CRT): P=0 or P= 1  possible • NRT: P=0 or P=1  means the designer has to go back and check the question • The higher the difficulty index, the easier the item is understood to be. • How high should the P be? Depending on purpose of giving the test • If for mastery of the essential  P>= .8 • If for spreading people out  P should be lower

  6. Item Discrimination (1) • If the test and a single item measure the same thing, one would expect people who do well on the test to answer that item correctly, and those who do poorly to answer the item incorrectly. • A good item discriminates between those who do well on the test and those who do poorly. Two indices can be computed to determine the discriminating power of an item, the item discrimination index, D, and discrimination coefficients.

  7. Item Discrimination (2) • Discrmination index: D or ID • Indicate the degree to which the task differentiates correctly among test takers; the difference of the percentage passing between upper group and lower group

  8. Item Discrimination (3) • D= P (% of upper group) – P (% of lower group) OR D = • The higher the discrimination index, the better the item • How to interpret the discrimination index? -1<= D<= 1 • When D= 0 (i.e., %U = %L) • ok/great  if testing for mastery • terrible if testing for spreading people out • D>= .25 or .35 (Bailey 135) • D = .7 (realistic D): probably the best D that we could get Correct Answer in Upper Group – Correct Answer in Lower Group No. of people in each group (one group)

  9. Item Discrimination (4) • Definition of upper/lower group: • Sharper differentiation when using more extreme ends (e.g., 10% top & bottom) • But, reduce reliability of the results (because of small # of cases utilized) • Q: What’s the optimum percentage (used to define upper & lower group) so that the characteristics of differentiation & reliability can be maximized?

  10. Item Discrimination (4) • The optimum percentage = 27% (Anastasi 1988) • Any convenient number between 25% ~ 33% will do. (Yuan 1998) • The JCEE uses 33% • Now, do exercises on Bailey 146. and handout

  11. Distractor Analysis (1) • Analyzing the distractors (e.i. incorrect alternatives) is useful in determining the relative usefulness of the decoys in each term. • Also called distractor efficiency analysis • To investigate the degree to which the distractors are functioning efficiently (i.e., attracting Ss who don’t know the correct answer). • Specifically related to the multiple-choice format.

  12. Distractor Analysis (2) • Choices A B C D* U% 20 0 0 80 L% 10 40 0 50 • P = ? D = ? • P = (.8 + .5)/2 = 1.3/2 = .65 • D = .8 - .5 = .3 • Now, look at each of the distractors. • A  negatively discriminative • C  no appeal at all

  13. Distractor Analysis (3) • Choices A B C* D high .40 .0 .60 .0 P= .6 low .13 .07 .60 .20 D= 0 • Examine the original item carefully (in terms of its format and content) • Which choices should be revised?

  14. Distractor Analysis (4) • Choices A* B C D high .00 .80 .08 .12 P= .5 low 1.00 .00 .00 .00 D= - 1.00 • What does the result of distractor analysis tell us? • The item might be miskeyed? Should B be the correct answer?

More Related