1 / 14

SETTING & MAINTAINING EXAM STANDARDS

SETTING & MAINTAINING EXAM STANDARDS. Raja C. Bandaranayake. NORM-REFERENCED Relative Based on peer-performance Varies with each group Cut-off point not related to competence. CRITERION-REFERENCED Absolute Not related to peer performance Standard set prior to exam

ash
Download Presentation

SETTING & MAINTAINING EXAM STANDARDS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SETTING & MAINTAINING EXAM STANDARDS Raja C. Bandaranayake

  2. NORM-REFERENCED Relative Based on peer-performance Varies with each group Cut-off point not related to competence CRITERION-REFERENCED Absolute Not related to peer performance Standard set prior to exam Referenced to a definedlevel of performance NORM- & CRITERION-REFERENCED STANDARDS

  3. NEDELSKY (1954) METHOD: Example • Consider N judges and n MCQ items of 1 in 5 type • Judge A identifies 2 options in item 1 as those which a minimally competent examinee should eliminate as incorrect. • MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3 • Similarly, in item 2 he identifies 3 options, giving an MPLA2 = 1/(5-3) = 1/2 • He repeats this process for each item. • The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 + ………….MPLAn • Similarly, Judge B’s MPL [MPLB] is determined • The MPL for the exam (= cut-off score) is: (MPLA + MPLB + MPLC +….MPLN) / N

  4. ANGOFF (1971) METHODExample • N judges consider 100 minimally competent examinees taking an MCQ exam of n items. • Judge A estimates that, of these examinees, 50 should answer item 1 correctly, 20 item 2 correctly, 70 item 3 correctly, and so on to item n. • The MPL for Judge A [MPLA] = (0.5 + 0.2 + 0.7 + . xn) / n X 100 = (say) A%. • Similarly, for Judges B, C, D, E, …..N, the MPLs would be B%, C%, D%, E% ……N%, respectively. • The MPL (cut-off score) for the exam is: (A% + B% + C% + D% + E% +....N%) / N

  5. EBEL (1972) METHODExample • Assume that Judge A assigns items in a 200-item MCQ test to the cells of a “relevance-by-difficulty” matrix, as follows. • He then estimates the percentage of items in each cell of the matrix that a minimally competent examinee should be able to answer correctly (as indicated within the cell). • Each cell also includes the products of these two values. • EASYMEDIUMHARD ESSENTIAL 15 x 100% = 1500 25 x 80% =2000 10 x 60% = 600 IMPORTANT 20 x 80% = 1600 40 x 60% =2400 20 x 50% =1000 ACCEPTABLE 10 x 50% = 500 25 x 40% = 1000 5 x 10% = 50 QUESTIONABLE 10 x 30% = 300 15 x 20% = 300 5 x 0% = 0

  6. EBEL (1972) METHOD - contd.Example • The MPL for Judge A [MPLA] is then: • (1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000 + 50 + 0) / 200 = 56.25 % • Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD] …..N [MPLN] are determined. • The MPL for the exam (cut-off score) is: • (MPLA+MPLB+ MPLc+ MPLD + …..MPLN) / N

  7. PROPOSED EBEL MODIFICATION EASY MEDIUM HARDESSENT. 6x 100% = 600 12 x 80% = 960 7 x 50% = 350 IMPORT. 12 x 80% = 960 24 x 60% = 1440 19 x 40% = 760 ACCEPT. 5 x 60% = 300 12 x 50% = 600 3 x 10% = 30 MPL: =600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30 =6000/100= 60

  8. A fmax 20 Failure Rate% 15 B fmin 10 35 40 45 50 Cut-off score(%) cmin cmax HOFSTEE METHOD

  9. HOFSTEE METHOD Example A plot of cut-off scores for a given exam against resulting failure rates is given cmin = 40% cmax = 45% fmin = 10% fmax = 20% A = point representing cmin,fmax B = point representing cmax,fmin Line AB intersects the curve at a cut-off point of 42.5% Thus, operational cut-off score = 42.5%

  10. CUT-OFF SCORE FOR 1 IN 5 MCQ[FRACS PART 1] • Probability of guessing (=1 in 5) = 20% • ‘Total ignorance’ score = 20% • Maximum possible score =100% • Effective range of scores = 20% to 100% • Mid-point of this range = 60% • Additional factor (as PG exam) = 5% • Nominal cut-off score (60%+5%) = 65%

  11. CUT-OFF SCORES: “MARKER QUESTIONS” 1. Comparison of exam scores Mean score in this exam: 56.7% Average exam mean score over last 4 years: 59.4% Thus mean score in this exam is: 2.7% lower Assuming this candidate group is of same standard as in last 4 yrs, this exam is: 2.7% harder

  12. CUT-OFF SCORES: “MARKER QUESTIONS” - contd. 2. Comparison of “marker” scores Mean score in this exam on previously used questions (N=162): 62.5% Mean score on same questions when they were each last used: 60.5% Thus, when compared with previous candidates, this group of candidates, on these items, scored (62.5-60.5)% = 2.0% higher Thus this group of candidates is: 2.0% betterthan previous groups

  13. CUT-OFF SCORES: “MARKER QUESTIONS” – contd. 3. Estimating examination difficulty Thus it is expected that their mean score in this exam would be: 2.0% higher But their mean score in this exam is: 2.7% lower Thus this exam is really: 4.7% harder

  14. CUT-OFF SCORES: “MARKER QUESTIONS” –contd. 4. Determining cut-off score The cut-off level for an average exam is: 65.0% Thus the cut-off level for this exam should be (65 – 4.7)% = 60.3% Cut-off score = 60.3%

More Related