“There’s no confidence in multiple-choice testing …”

1. “There’s no confidence in multiple-choice testing …” Phil Davies School of Computing University of Glamorgan South Wales

2. Who has NO confidence? • STAFF • “They can just guess the answer” • “It’s not the way it is in the real world” • “there are varying degree of knowledge” (Echternacht, 1972) • “don’t worry it’s easy, it’s multiple choice” • Possibly more importantly -- STUDENTS

3. The First Test • 40 Questions / Formative • Time Consuming / Difficult to create sensible distracters / Rocket Science • I thought … Average 1 minute per question • Actual … all finished in ten minutes … apart from one who took 50 minutes • Why? No reason to think

4. On-Line Assessment & Learning … OLAL • Level Two Degree .. Computer Communications & Networks • Failures High … Programming to Demonstrate .. Technical Subject .. Essay-type answers in examinations • Assess .. “Knowledge” “Understanding” “Learning Outcomes” …. Directed

6. On Line Assessment & LEARNING • Four Tests • Weeks 3,6,9,12 = 5%, 10%, 15% 20% • Cumulative • Two Passes of SAME test • Learning .. Short term? • Slowed Down .. Thought about WHEN WRONG .

7. What decides the QUALITY of a question? • In the recent World cup qualifier between England and Greece, who scored the all important equalizing goal? • David Beckham • Victoria Beckham • Brooklyn Beckham

8. Rewards? • Know it • On the tip of my tongue • If I see the answers I’ll know it • Deduction • Guess • Know it … oh no I’ve got it wrong .. misinformation

10. Allocation of Marks • Logarithmic +3, +2, +1, -3, -2, -1 • This is not the same!! -> not seeing answer • Students were involved / Provisional • “You can try this out but it must not have a negative affect upon our results compared with last year” • Reward knowledge +4, +2, +1, -2, -1, 0 • If we don’t know it, then why take off 1

11. 2002 OLAL 1/3 .. Better by approx. 6% +5, +3, +1, -2, -1, 0

14. Percentile Frequency Distribution

17. Proportion of Questions Correct • High Confidence => 84% correct • Fairly Confident => 58% correct • Not confident => 47% correct • 72% of all questions answered with high confidence

18. Based on Test 3, who were the most confident (60 Questions) • 75+ • 52 7 1 • 50+ • 44 13 3 • 25+ • 42 15 3 • 0+ • 33 19 8

19. Student Feedback Thoughts on Multiple Choice • “useful learning tool” • “second test is a really useful way of learning” • “immediate feedback helps me to learn” • “doesn’t reflect my lack of knowledge” • “not fair, I found it difficult to tell the difference between the answers” • “negative marking made me panic”

20. Student thoughts on Confidence • “Eliminates guesswork” • “I was waiting for a leggy blond to bring on my prize at the end” • “separated the lucky students from me” • “at least I didn’t lose marks for selecting no confidence” • “are you testing my confidence or knowledge”

21. Quality of a multiple choice question? • In the recent World cup qualifier between England and Greece, who scored the all important equalizing goal? • Teddy Sheringham • Michael Owen • Robbie Fowler • QUALITY OF QUESTION NOT DISTRACTERS

22. Question Format • “One or more questions were impossible to judge” • “I wish you’d put more detail in the question” • “Perhaps it would have been better to show the answers, ask for confidence, and then show the question”

23. Conclusions? • Assess subject area not individual’s confidence • Quality of questions => question NOT distracters • Allocation of Marks (based on past?) • Student Accept / Staff Accept • Time for tests increases “really had to think”