1 / 13

Lunelle L Pienaar & Gregory Doyle Education Development Unit,

Performance characteristic of computer based Extended matching items and One Best Answer: A pilot study . Lunelle L Pienaar & Gregory Doyle Education Development Unit, Faculty of Health Sciences, University of Cape Town. Background .

terri
Download Presentation

Lunelle L Pienaar & Gregory Doyle Education Development Unit,

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance characteristic of computer based Extended matching items and One Best Answer: A pilot study Lunelle L Pienaar & Gregory Doyle Education Development Unit, Faculty of Health Sciences, University of Cape Town

  2. Background • Multiple choice format remain one method of assessment that is most commonly used to test theoretical knowledge in medical education (Palmer EJ and Devitt PG. 2007, Mavis et al, 2001). • Increasingly computer based assessment is used more within the FHS UCT-student numbers, drive increase IT use

  3. MCQ Types • R-type (OBA) • Traditionally used more often and is written with question in the form of a clinical case scenario with 4 or 5 option, questions are thought to test recall of knowledge. • A-type (EMI) • The questions are written with a theme and questions are constructed around the focus theme (Beullens et al, 2002, Palmer and Devitt, 2007).

  4. Why EMI • EMIs is considered a better format to test problem solving and integration of knowledge( Swanson et al, 2008, Swanson et al, 2006). • EMI has demonstrated better discrimination between students as well as more difficulty compared to OBA (McCoubrie2004).

  5. study question • What is the performance of in house test administered to medical students during one of their clinical rotations in 5th year. • What is the test item difficulty and discrimination of EMIs and OBA used in summative assessment. • Is there any differences between item difficulty and discrimination between the EMI and OBA.

  6. Methods • The questions were administered to all 5th year students at the end of their clinical block. • We evaluated OBA items and extended matching items that were developed by the 5th year course lecturer and evaluated as appropriate for testing by the course convener(not the researcher). • The item difficulty and discrimination for the EMIs and OBA was determined

  7. Results • Results were from the end of block test with 5 groups of students. The same questions were administered to students all 5 groups of students. • The 193 students answered a computer based test containing five EMI sets with 34 items and 16 (OBA)-Type MCQs. • The OBA -had 4 options with each of the 5 EMIs sets had 9 options with 9 questions.

  8. Distribution of Test item Difficulty

  9. Distribution of test item discrimination

  10. Mean Test Item Difficulty and Discrimination

  11. Conclusion • The study did not show any statistical significance between the 2 types of MCQs – • small sample of test items. • Possible reason for this may also be that writing OBA format were used more often in the past. • Both EMI and OBA test items wide range of item difficulties • The results raises 2 issues • when questions are reused to select the best performing items. • evaluation of test post administration is important to determine how to improve quality of the assessment items.

  12. References • Beullens J, Van Damme B, Jaspaert H, Janssen PJ. 2002. Are extended-matching multiple-choice items appropriate for a final test in medical education? Medical Teacher 24 (4):390-5. • Mavis BE, Cole BL, Hoppe RB. 2001 A survey of student assessment in U.S. medical schools: the balance of breadth versus fidelity. Teaching and Learning in Medicine, 13 (2), 74-79. • McCourbrie P. 2004. improving the fairness of multiple- choice questions: a literature review. Medical Teacher, vol 26. 8 (709-712) • Palmer EJ and Devitt PG. 2007. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Med Educ; 7: 49. • Swanson, D.B., Holtzman, K.Z., Allbee, K. 2008. Measurement characteristics of content-parallel single-best-answer and extended-matching questions in relation to number and source of options. Journal of the Association of American Colleges, 83(10 Suppl):S21-4 • Swanson, D.B., Holtzman, K.Z., Allbee, K., Clauser, B.E. 2006. Psychometric characteristics and response times for content-parallel extended-matching and one-best-answer items in relation to number of options. Journal of the Association of American Colleges. Abstract 81(10 Suppl):S52-5

  13. http://edusum.edublogs.org/category/thoughts-on-assessment/

More Related