1 / 19

Opening up multiple choice: Assessing with confidence

Opening up multiple choice: Assessing with confidence. Jon Rosewell J.P.Rosewell@open.ac.uk. Self-assessment: strategies and software to stimulate learning HEA workshop, The Open University, 11 th June 2012. Pros objective marking reliable marking easy to implement Cons

varian
Download Presentation

Opening up multiple choice: Assessing with confidence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Opening up multiple choice: Assessing with confidence Jon RosewellJ.P.Rosewell@open.ac.uk Self-assessment: strategies and software to stimulate learning HEA workshop, The Open University, 11th June 2012

  2. Pros objective marking reliable marking easy to implement Cons distractors trivial distractors engender misconceptions working backwards Pros numeric easy to mark tests deeper learning can find misconceptions Cons Free text very difficult to mark reliably MCQ Open question

  3. Open question

  4. Multiple-choice question

  5. How (not) to do MCQ… • Give the name of a supervolcano: • Yellowstone • Redstone • Bluestone • Which is a plate boundary: • Conservative • Destructive • Liberal • Labour • What policy did the Chinese introduce to limit population: • One child policy • Two child policy • The child catcher Reported from BBC Bite Size

  6. Certainty-Based Confidence-Based Marking (CBM)

  7. CBM scoring Tentative & correct Confidently correct Cocksure – and wrong! Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment[http://www.ucl.ac.uk/lapt/REAP_cbm.pdf]

  8. CBM motivations • Rewards care and effort • Greater engagement • Encourages reflective learning • Encourages self-assessment

  9. Open CBM

  10. Open CBM advantages • Mechanically, question remains simple MCQ • answer matching is trivial • easy to implement • possible to reuse existing question banks • But transformed closed to open question • student must formulate answer to decide confidence • must decide answer without +ve or –ve clues. • cannot work backward • will not be led into misconceptions.  must answer the question as set.

  11. Pros objective marking reliable marking easy to implement Cons distractors trivial distractors engender misconceptions working backwards Pros open question reflection ...as MCQ Cons not always applicable intimidating? personality dependent? MCQ Open CBM

  12. Possible benefits • Students • better engagement • improved learning • more reflective learners • Staff • easier question setting

  13. eSTEeM project plan • Implement open CBM question type • in OpenMark • Trial in course assessment • controlled experimental design • pilot (formative) on T216 CISCO networking • Log measures • Assignment scores • Time spent on task • Structured interviews to probe attitudinal aspects • User-lab trials

  14. T216 Cisco networking • Questions from existing bank • Offered as revision quizzes • One set for each of four blocks • 25 questions drawn randomly from bank of ~100 • Randomly offered as either open CBM or MCQ format • Issues: • Some questions bad! • OpenMark report doesn’t show confidence • Too much randomness

  15. Uptake

  16. Demo questions – scores Easy Tricky! Difficult

  17. Demo questions – confidence

  18. Pros objective marking reliable marking easy to implement Cons distractors trivial distractors engender misconceptions working backwards Pros open question reflection ...as MCQ Cons not always applicable intimidating? personality dependent? MCQ Open CBM

  19. Jon RosewellJ.P.Rosewell@open.ac.uk www.open.ac.uk

More Related