1 / 14

Statistics of EBO 2010 Examination EBO General Assembly Sunday June 21st, 2010 (Tallin, Estonia)

Statistics of EBO 2010 Examination EBO General Assembly Sunday June 21st, 2010 (Tallin, Estonia). Danny G.P. Mathysen MSc. Biomedical Sciences EBOD Assessment and Executive Officer

osric
Download Presentation

Statistics of EBO 2010 Examination EBO General Assembly Sunday June 21st, 2010 (Tallin, Estonia)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistics of EBO 2010 ExaminationEBO General AssemblySunday June 21st, 2010 (Tallin, Estonia) Danny G.P. MathysenMSc. Biomedical Sciences EBOD Assessment and Executive Officer Antwerp University Hospital, Department of OphthalmologyWilrijkstraat 10, B-2650 Edegem, BelgiumE-mail: danny.mathysen@uza.be

  2. Score calculation for Written Paper EBOD 2010 Candidate Population • Written examination (MCQs) • 310 candidates • Oral examination • 308 candidates • 1 candidate did not show up for Viva Voce • 1 candidate did show only for some of the Viva Voce topics

  3. Score calculation for Written Paper EBOD 2010 Scoring rules • Question Number (1  52) • Item Number (A  E) • T (True) • F (False) • D (Don’t know) Marks obtained? +1 In case ONLY the correct answer was completed 0 In case ONLY the D option was completed –0.5 In case ONLY the incorrect answer was completed In case T AND F werecompleted In case NOTHING was completed (blank item) In case D was COMBINEDwith T and/or F

  4. Score calculation for Written Paper EBOD 2010 Scoring rules • Candidate score for MCQ-1(simulation): • A True (Correct Answer: True) +1 • B False (Correct Answer: False) +1 • C True (Correct Answer: True) +1 +2.5 • D Don’t know (Correct Answer: True) 0 • E True (Correct Answer: False) –0.5

  5. EBOD 2010 Negative Marking • Advantages for EBO candidates of T/F items • Reliable in case of translation (English, French, German) choice of language will not result in being (dis)advantaged • Accessibility (e.g. dyslexia) not too complicated for candidates • Duration of the examination stress level of candidates can be kept to a minimum • Relatively easy to process results can be presented on-site • Disadvantage for EBO candidates of T/F items • Probability of guessing right = 50 % level of weakest candidates is overestimated ( oral examination)

  6. EBOD 2010 Negative Marking • Hypothesis on the influence of negative marking • Average scores will drop (punishment of incorrect answers) • Spread of candidate scores will enlarge ( room for discrimination) • Rit-value of individual items will increase • Reliability of EBOD will increase • Argument against negative marking expressed by European Board of Anaesthesiology • Negative marking is discriminating towards female candidates

  7. EBOD 2010 Spread of Scores NEGATIVE MARKINGAT EBOD 2010 2010 2009 • How to overcome the disadvantages of T/F items? • Introduction of negative marking • Increase of discriminative power of examination • Reduction of guess factor • wild guesses will be punished (weakest candidates) • guesses by reasoning (partial knowledge) will be rewarded 0 260

  8. Score calculation for Written Paper EBOD 2010 Statistical Output (SpeedWell)

  9. EBOD 2010 Degree of Difficulty • EBOD 2009 • Degree of Difficulty (P-value) of 0.79 (overestimated due to guessing) • Estimation of a large proportion of candidates guessing (> 33 %) • EBOD 2010 • Introduction of the “Don’t know” option reduction of wild guesses  used on average for 15 % of items (or 39 items) per candidate • Degree of Difficulty (P-value) of 0.66

  10. EBOD 2010 Point Biserial Correlation correlation betweenitem and total MCQ score 0.18 0.14 • Point biserial correlation coefficient (Rit) • Estimator of the correlation betweenthe individual item scores Xi(either -0.5, 0 or 1), andthe total MCQ scores Yi(ranging from 61.5 to 209) of the candidates -1 0 +1

  11. EBOD 2010 Internal Consistency internal consistencyof EBOD MCQ-test is good • Cronbach’s coefficient alpha (r) = 0.87 (2009: 0.78) • Estimator of the lower bound of the internal consistency(degree to which all MCQs leaves are measuring the same, i.e. knowledge of candidates) of EBOD 2010 (95% CI: 0.86 – 0.89)

  12. EBOD 2010 Male vs. Female Candidates • EBOD 2010 Written Examination • 310 Candidates • 168 Male Candidates • 142 Female Candidates • Percentage of candidates using the “Don’t know” option • Male candidates: used on average for 13% of items (34 items) • Female candidates: used on average for 16% of items (42 items) • Statistically significant (p = 0.02)

  13. EBOD 2010 Male vs. Female Candidates • Average absolute candidate scores • Male candidates: 148.21 • Female candidates: 143.36 • NOTstatistically significant (p > 0.05) • Distribution of converted candidate scores (1-10) • NOT statistically significant (p > 0.05) when comparing all scores • NOT statistically significant (p > 0.05) when comparing ≤ 5 versus ≥ 6

  14. EBOD 2010 Negative Marking: Conclusions • In general: • Average scores dropped (204.11  145.99) • Spread of results became larger (13.0  24.8) • Internal consistency (Cronbach-α) improved (0.78  0.87) • P-value was less overestimated due to D option (0.79  0.66) • Rit-value improved (0.14  0.18) • When comparing male and female candidates: • Female candidates (D option ticked for 42 items on average) are more prudent when guessing is concerned compared to male candidates(D option ticked for 34 items on average) (p = 0.02) • However, without negative impact on ability to pass EBOD 2010!

More Related