An interactive poster presentation for: “Bradford’s Got TALENT”

1. e-delivery of Formative Assessment and Feedback: How well does it work?Michael J Cox, Bradford School of Optometry & Vision Science An interactive poster presentation for: “Bradford’s Got TALENT” The 9th Annual Learning, Teaching and Assessment Conference, June 2010 Click to begin

2. e-delivery of Formative Assessment and Feedback: How well does it work?Michael J Cox, Bradford School of Optometry & Vision Choose an Option The Aims The Actions QMP – Pros and Cons Formatives in Action: Choose a Demonstration Evaluating Effectiveness: Facts and Figures Evaluating Effectiveness : User Comments Conclusions

3. e-delivery of Formative Assessment and Feedback: What are we trying to do? The Problem • A large (110+) cohort of students • A level 3, 20 credit module covering a broad and challenging area: Ocular & Systemic Disease • Formative assessments existed but providing individual feedback to these was too resource hungry to be practical The Aims • Develop and extend the breadth and depth of formative assessment available to each student • Provide individual feedback to students • Enhance the effectiveness of the formative assessment in assisting the learning of each student

4. The Module and the New Formative Assessment Structure • A 20 credit module, taught over 24 weeks and divided into 4-week blocks for the purposes of formative assessment • An assessment covers material from the latest block and revises some earlier material • An assessment takes 30 minutes, although students can take longer, if they wish as there is no limit to the number of attempts • The attainment on the assessments is entirely for the use of the student – it does not influence summative assessment marking

5. What can QMP do? • Present questions to students • Provide a rich variety of question types • Record their answers • Mark many of their answers and archive results • Provide feedback, dependent upon the student’s answer, to explain why a right answer is right and a wrong answer, wrong • Provide model answers, with a marking scheme, for questions with paragraph length answers so that students can self-mark • Provide a grading scheme and instructions for students so that they can determine their total mark and find out where this places them academically

6. Benefits when using QMP for Formative Assessment • Good integration with Blackboard • Flexible (well – bendable) in operation and availability • QMP can mark single word or multiple word answers for you • QMP will deliver feedback for you. It is immediate and can be answer (not just right/wrong) driven • Simple presentation of images/video in questions and feedback

7. Benefits when using QMP for Formative Assessment • “Sequencing” questions and “Hotspot” questions can be particularly powerful and automatically marked • Archiving of students’ answers allows the quality control loop to be closed • It makes you think hard about your assessments • Watch the demo to find out more

8. Challenges in using QMP for Formative Assessment • Not everyone is comfortable using a computer • Developing your design skills takes time • It can be an awkward tool to use • Development time can be extensive • You’ll want to make things perfect, so spend more time • UoB server version doesn’t yet work with Win 7 • It makes you think hard about your assessments • Watch the demo to find out more

9. Find out more:Choose a Demonstration VideoFull resolution versions available from the author on request. Best viewed in Full Screen Click me to watch the YouTube video demonstration about Accessing the Assessment(2 Minutes) Click me to watch the YouTube video demonstration about Using different Question Types(7 Minutes) Click me to watch the YouTube video demonstration about Feedback (5 Minutes) Click me to return to the main navigation screen

10. Initial Evaluation • All planned delivery took place • Uptake of the VLE delivered material was 80%+ (following the assessments with friends could mean this is even higher) • Summative assessments, also delivered through QMP, showed a small, but definite, improvement in the average student performance

11. Course Unit Evaluation Although not directed towards specifically evaluating this work, the CUEQs gave valuable information on the success of the work Written Comments • In “major strengths”, 25 out of the 33 individuals who commented on major strengths identified the formative assessment • In “how to improve the unit” three comments were made about formative assessment • More time to go through the formatives together in class • More formative assessments • Formatives not to appear online, but in classes, to force students to revise for them • 53% return rate • 4.15 score (1->5 point scale, 1 bad, 5 good) for “Assessment of my work was helpful in my learning” • Previous scoring for this area was 3.06 (2008) and 3.41 (2005), despite equally well received “overall satisfaction” at ~4.3 • “Learning objectives were clear to me” has also increased over this time, but less sharply

12. User Survey Results The Best All students registered on the module were invited to complete an online survey about the use of QMP for formative assessment • Mean satisfaction scores from 23 students that completed the survey. All except two had completed all four QMP formative assessments. Two had completed three of the four assessments • Error bars show 95% confidence interval • 0 – Strongly Disagree, 4- Strongly Agree Survey Questions Formative Assessments in OSD were useful in my learning QMP was easy to use The variety of question styles used was helpful Feedback from the questions was helpful Image quality in QMP was better than in paper based examinations I wish to see QMP continue to be used in formative assessment in the future The Worst

13. User Survey Comments (Likes) The variety of subjects was excellent. The questions were superb in the main, although one or two of the questions seemed a bit irrelevent to my actual understanding of OSD. The varied content of MCQ and short answer and photo recognition work very well. Instant feedback and explanations for the correct answers were very helpful. It was useful to have access to all the assessments so you can re-do the assessments at a later date and to recap all the information covered in the module. Mimicked exams taken in Jan very well. Good feedback points.

14. User Survey Comments (Likes) immediate feedback aided revision and the chance to repeat tests also allowed me to go back and test myself again on certain areas where i struggled. An absolute brilliant way of revision, and it really helped me in preparation for the exams in way of the format of the exam, it also made the exam enjoyable if that can at all be possible.It is a pity that at the moment not all exams are possible in this format. Formative assessments were extremely helpful in preparing for the osd exam. Was useful to get used to the layout and the feedback helped in letting us know how much detail to go into with the answers.

15. User Survey Comments (Likes) Very close to the exam format that we had in January and therefore prepares you well for what to expect. Liked variety of question styles rather than purely text based. Demonstrating knowledge through the identification of signs on the photos themselves ( remember a glaucoma one particularly) by placing marker over sign seemed useful and relevant! please make available every topic like this as class work conjunction with on line lecture. for every subj if possible the feed back! the variety of questions, extremly useful in testing my knowledge/ revision, and refelctes how the summative assesment will be

16. User Survey Comments (Dislikes) Increase the number of available questions. I did not feel that getting together and going over the assessments was always a good use of my time. I think gaining the feedback via my PC good enough Not being the best speller (and from an e-mail in the first semester i think i know your views on this one)some lee way on spelling or perhaps auto correction would help so long as it is not miles out. ps have used spell check on this. More feedback with wrong answers.

17. User Survey Comments (Dislikes) the quality of the images!, I dont like how our exams are done on computers, as takes alot longer to type and think using a p.cesp for people who do no not use a p.c for revision but a book instead however overall EXCELLENT WORK ! love it The automatic marking seemed a bit fussy, in that; if I had the correct answer but another word also it was marked as wrong. In every other way I think that the formatives were extremely useful. Allow feedback on unanswered questions! Otherwise, they were so useful the only way to improve them would be to offer more!!! The questions which involved scrolling up and down were not the easiest to follow, probably due to lack concentration more than anything.

18. Conclusions • QMP was used with Blackboard to deliver re-designed and extended formative assessments for the module Ocular & Systemic Disease • Assessments were successfully run and accessed throughout the academic year • Question design encompasses a broad range of different tasks • Automatic marking, self-marking and directed individual feedback were all used • The student response to the new delivery was very positive