1 / 52

Paul Denny

Transforming Assessment Webinar 1 st August, 2012. Crowd-sourcing exam questions with PeerWise. Paul Denny. Department of Computer Science The University of Auckland New Zealand. Overview – 4 sections. PeerWise motivations and overview Participate hands-on demonstration

kalil
Download Presentation

Paul Denny

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Transforming Assessment Webinar 1st August, 2012 Crowd-sourcing exam questions with PeerWise Paul Denny Department of Computer Science The University of Auckland New Zealand

  2. Overview – 4 sections • PeerWise • motivations and overview • Participate • hands-on demonstration • A recent case study • The University of Auckland (n > 1000) • Getting started • tips and ideas 15 minutes 20 minutes 10 minutes 10 minutes

  3. Get ready!

  4. Section 1 PeerWise

  5. PeerWise • A web-based repository of multiple-choice questions created by students peerwise.cs.auckland.ac.nz

  6. Question authoring • A student contributing a new question would develop the question stem: Question Student authored question HUBS192, 2010 University of Otago

  7. Question authoring Alternatives • And a set of alternatives • These should target misconceptions • Of course, the correct answer must be indicated

  8. Question authoring Alternatives • And a set of alternatives • These should target misconceptions • Of course, the correct answer must be indicated

  9. Question authoring Explanation • And an explanation, in their own words • Useful for students who answer incorrectly

  10. Question authoring Explanation “You don't really understand how much or how little you know about a concept until you try to devise a good, original question about it” “The aspect I found truly useful was the creation of questions, which reinforced much of [my] understanding while also actively making me clarify and solidify my thought processes (especially the explanation parts)” • And an explanation, in their own words • Useful for students who answer incorrectly

  11. The main menu of PeerWise

  12. Finding questions to answer

  13. Selecting an answer

  14. Students answering questions are given immediate feedback on their selection:

  15. Peer comparison and the explanation

  16. “The biggest benefit of using PeerWise was getting an understanding of the kind of level of questions and knowledge that is possessed by other students” “Being able to see how other people answered was great as it allowed me to recognise at which level I was at compared to everyone else” Peer comparison and the explanation

  17. Critique and discussion

  18. Badges, points and leaderboards…

  19. Section 2 Participate Your turn: open a new window in your browser

  20. Google: peerwise, start typing: trans... Or go direct to the URL: http://peerwise.cs.auckland.ac.nz/at/?ta_workshop

  21. Registration

  22. Step 1 – choose a name

  23. Step 2 – enter a password

  24. Step 3 – enter the Course ID 6453

  25. Step 4 – enter your Identifier 3-digit number between 100 and 500

  26. That’s it!

  27. Log in

  28. Choose the “TA Workshop” course

  29. 1)Your questions: “view” 2) Create new question 3) Preview Creating a question 4)“Make changes” or “Save question”

  30. Section 3 A recent case study

  31. Recent case study • Population Health (POPLHLTH111) • Semester 1: March – June, 2012 • Students: 1031 • Deadlines • 30th March: author 1 and answer 20 for 1.5% • 18th May: author 1 and answer 20 for 1.5% • Mid-term tests • 28th March: worth 15% • 15th May: worth 15%

  32. Daily usage Test 2 Test 1 Deadline 2 Deadline 1 New questions authored per day Answers submitted per day Questions created: 4,196 Answers submitted: 188,590

  33. Student activity Answers per student: average = 167, median = 105 Most active 25% submitted 60% of all answers Three most active students: 10,000 answers

  34. Student activity 3919 answers (79% correct), 50 days active 3890 answers (78% correct), 66 days active 2043 answers (70% correct), 58 days active Answers per student: average = 167, median = 105 Most active 25% submitted 60% of all answers Three most active students: 10,000 answers

  35. Days of activity

  36. Perceptions Creating my own questions was an effective way of learning in this class (A + SA = 83%) Answering questions created by my class mates was an effective way of learning in this class (A + SA = 80%) n = 519 (50.3%)

  37. Perceptions What do you believe is the biggest benefit of using PeerWise? What do you believe is the biggest problem with PeerWise? Poor quality questions Learning by authoring

  38. Perceptions What do you believe is the biggest benefit of using PeerWise? What do you believe is the biggest problem with PeerWise? Poor quality questions Learning by authoring “Being able to get into the mindset of the exam writer - you have to know the subject very well to ask a good question” “When you write your own question you really have to understand what you are writing about which is really important” “The act of writing questions made you think about and absorb more course information. This is especially true when trying to explain why alternative answers are wrong.” “When writing the questions you have to learn and understand the information because you have to think about the question options and write an explanation. If your explanation is wrong or unclear then students tend to comment and correct you therefore you benefit and so does the other person.”

  39. Perceptions What do you believe is the biggest benefit of using PeerWise? What do you believe is the biggest problem with PeerWise? Poor quality questions Learning by authoring “Being able to get into the mindset of the exam writer - you have to know the subject very well to ask a good question” “When you write your own question you really have to understand what you are writing about which is really important” “The act of writing questions made you think about and absorb more course information. This is especially true when trying to explain why alternative answers are wrong.” “When writing the questions you have to learn and understand the information because you have to think about the question options and write an explanation. If your explanation is wrong or unclear then students tend to comment and correct you therefore you benefit and so does the other person.” “The quality of questions posted by my peers ranged from extremely poor to very good” “A lot of the questions were very shoddy. It didn't seem like those people put as much thought and effort into making them” “Bad questions, if you don't know the content as well, otherwise it's a great opportunity to correct someone and have more confidence in you knowledge” “The major problem I found is that for PopHlth 111 since Peerwise contribution is compulsory, the calibre of questions suffered seriously from people just trying to get their 1.5% for each module... even though there was maybe only 10% the number of questions for BioSci 101/107… their calibre was eminently higher”

  40. The “best” question Answered by: 233 Rated by: 187 Avg. rating: 4.22(/5)

  41. The “best” question Answered by: 233 Rated by: 187 Avg. rating: 4.22(/5)

  42. The “worst” question... Answered by: 19 Rated by: 11 Avg. rating: 0.73 (/5)

  43. The “worst” question... Answered by: 19 Rated by: 11 Avg. rating: 0.73 (/5)

  44. Section 4 Getting started

  45. Getting started Sign up!

  46. Creating a course

  47. Creating a course 123 234 345 456 567

  48. Creating a course • Step 1) • Step 2)

More Related