1 / 15

Computerised Free-text Assessment of Key Feature Problems

Computerised Free-text Assessment of Key Feature Problems. Adrian Molyneux Steve Capey School of Medicine. Project aims. Fulfil student requests for more formative assessment Fulfil need for practice of summative exam formats Provide quality feedback

psamsel
Download Presentation

Computerised Free-text Assessment of Key Feature Problems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computerised Free-text Assessment of Key Feature Problems Adrian Molyneux Steve Capey School of Medicine

  2. Project aims • Fulfil student requests for more formative assessment • Fulfil need for practice of summative exam formats • Provide quality feedback • Achieve this without a heavy marking load placed on staff

  3. Key-feature problems (KFPs) …an innovative assessment format that tests clinical decision making skills by focussing only on the critical steps - or 'key features' - of each decision, thereby increasing the number of problems that can be answered in an examination. - Liz Farmer, Royal Australian College of GPs

  4. SCENARIO - Jennifer Evans, a 38-year-old statistician, presents to the accident and emergency department with a two hour history of sudden onset of leftsided pleuritic chest pain shortness of breath and an episode of haemoptysis. She had returned from a conference in Malaga on a flight 48 hours earlier. On examination she was found to be slightly febrile, tachycardic and in atrial fibrillation. Her blood pressure is 98/64mmHg. Examination of the chest showed decreased expansion on the left and localised decreasing breath sounds with a pleural rub. A plain radiograph of the chest shows a very small trace of fluid in the left costophrenic angle to the lungs are clear. A presumptive diagnosis of pulmonary embolism is made.

  5. Example questions from scenario 1. What initial investigations should be requested? Name only two (2) 2. Long distance travel is considered a minor risk factor for venous thromboembolism – what are the major risk factors? Name only four (4) 3. What is the mechanism of action of heparin? 4. Shortly after the heparin was commenced, Jennifer was started on warfarin. What endogenous compound does warfarin inhibit to produce its anticoagulant effect? List only one (1) 5. How is warfarin therapy monitored?

  6. Research questions • How do we train a computer system to mark an open-ended freetext response? • How to deliver the assessment • How to feed back the results to students • How to author succinct KFPs • Can we develop a method for the most efficient creation of a computer-marked KFP?

  7. Methods • Generate 10 KFPs and marking guidelines • Generate computer markscheme • Build web interface • Recruit students and run exam • Computer-mark responses • Tutor-mark responses • Refine markschemes (iterative process) • Statistical analysis

  8. Exam delivery system and results Exam delivery Results

  9. Computer – Tutor comparison

  10. Markscheme improvement • Make first attempt at markscheme and run exam • Examine student answers for further valid responses • Include these in computer markscheme and mark a second time • Examine computer markscheme for badly-performing questions and adjust (maybe loosen). Mark a third time.

  11. Conclusions – practical limitations • Succinct answers should be encouraged – perhaps limit text entry • Expect unexpected answers from students and markscheme modification • Standardisation of human marking • Adherence of humans to marking guidelines • Evolution of computer markschemes to cover the whole range of student answers

  12. Conclusions - positives • Computer marking ideal for questions with narrow range of possible responses • Reduce marking hours spent on formative questions, whilst providing quality feedback in a realistic examination format • Perfect for low-stakes, on-demand assessment

  13. Future direction of work • Run exams with full year groups • Refine question authoring • Evaluate other freetext question formats currently used within the medical school • Investigate the possibility of distributing tests asynchronously throughout the curriculum

More Related