Download
1 / 41
410 likes | 508 Views
The Unbearable Heaviness of Marking. What are the challenges of marking coursework?. Give a numerical value as a quantification of student attainment Clarifying what constitutes attainment Ensuring consistency In a way that is publicly defensible Under time pressure. But why do we do it?.
E N D
What are the challenges of marking coursework? • Give a numerical value as a quantification of student attainment • Clarifying what constitutes attainment • Ensuring consistency • In a way that is publicly defensible • Under time pressure
But why do we do it? • To give feedback • But feedback is a vast topic • Depends on the person and the task
Open University • Feedback is, put simply; additional tutoring that is tailored to the learner’s current needs. In the simplest case, this means that there is a mismatch between students’ and the tutors’ conceptual models, and the feedback is reducing or correcting this mismatch, very much as feedback is used in cybernetic systems.
The Principles of Amplification • Explain the student’s error • In a way suitable to this student
The Difficulties • We have to imagine this unknowing • We have to dismantle our own “taking-for-granted” knowledge • It needs to be rhetorically appropriate to the receiver
Then we have to give it a mark • In a way comparable to the other work we have received
We can be flexible with mistakes • But not too flexible
Time and Volume • 2 case studies • Marking >200 pieces of coursework for level 1 emedia design course • Marking 88 pieces of coursework for level M multimedia specification course
Marking Strategies • Rating Scale • Simple numerical value for each criterion • Checklist • Binary yes/no scores for properties of the submission • Rubric • A set of criteria with 3-5 attainment descriptors for each
EmediaMarksheet • Marking Scheme • BASICS • • Publish an swf file and upload it to studynet (5) • • Has correct number of pages with correct headings on each (5) • • Correct background colour (5) • • Correct width and height of the flash file (5) • • Correct number of buttons with correct colours for them (5) • • Make buttons navigate to correct frames using simple action script (5) • • Contains at least two images of you (5) • • Small filesize (5) • INTERMEDIATE • • Motion Tweening Of Position of Things in the animation welcome screen OR Motion • Tweening of Visibility of Things (for fade ins and fade outs) in the animation welcome • screen (5) • • Correct and Nice Positioning of Buttons and Content (5) • • Good easy on the eye content (text and image) ‐ not too little, not to much and all • relevant (10) • • Button clicks have small sounds associated with them (10) • ADVANCED • • Some very clever and visually elegant animation (using Shape Tweens, or Motion Guides • or Masking) in the animation welcome screen (10) • • Extremely well positioned and pleasant looking buttons (5) • • Extremely well judged content (5) • • An immediately visible and functioning background music toggle • Final Score = Σbasics + (Σbasics/40) * (Σintermediate+Σadvanced)
Evolution of criteria • In some categories the the marking was binary yes/no values - and in some cases "sort of". Explanation below: • SWF - Binary - If student submitted correctly 5 if not 0 • Pages - if student's work had correct number of pages, and the presentation stopped on those pages, student got 5 marks. If there was the odd one missing, or the odd one not stopping then sort of " mark was given - e.g 2.The background color was either right or wrong - 5 or 0. Width and height - usually 5 or 0. Button score varied: 5 if all buttons were of same style and were captioned according to which screen they went to then the score was 5. If the presentation used "next and forward" buttons rather than buttons going from every location to every other location (as specified by the brief) then deducationwas made. Button navigation: If all buttons went to the right screen and stopped there then student scored 5. If any misdirection then student scored less. If two images of the student were present, student scored 5 - if only 1 - then the score was 2. If the file was only marginally over the filesize limit then the student had a small reduction from 5. The presence of the tween was marked 5 or zero. If the buttons were fixed in the same position throughout the whole app, then the student scored 5. If they juddered a little, the student scored 2. If button positions moved randomly about then the scrore was 0.The student scored 10 if the content was well done in style and substance. If there was anything wrong with either the student scored only 5. If the content was severely found wanting - the score was 0.If the student had uniform sounds for all buttons, the score was 10. If there was uneven sound coverage for the buttons then 5 (and potentially lower if only one button say had sound). If no sounds 0.10 points for a cool animation which used advanced animation features of flash - with some complexity. If the animation, even if it used advanced features, but was just too trivial, then the score went down to 5If buttons were extremely elegant then an extra 5 points were given - and similarly if the content was extremely well judged.With the music toggle - 10 pts if it worked perfectly (the toggling and the the playing and muting of the music) - 5 points if the toggling didnt work perfectly.
The value of a rubric • helps faculty score students' work accurately, fairly, a reliably and quickly • requires faculty to be much more precise about criteria for evaluating student work • can lead to shared standards among faculty about what make a good essay or project • gives students a better idea about the qualities their work should exhibit • sets standards for .student achievement and make the standards explicit • allows students to self-assess work prior to submission and to offer feedback to peers • helps students understand the meaning of a grade; • improves student performance, particularly among lower achieving students • can lead to substantive conversations between faculty and students about the requirements for excellence within the field or discipline. Involving students in developing the rubric can be very beneficial
The drawbacks of rubrics • developing rubrics tends to be hard, time-consuming work • it is difficult to find just the right language to describe qualities of studentperformance that distinguish between mediocre and excellent work. If a student can achieve a high score on all the criteria and still not perform well at the task, you may have imprecise or inappropriate criteria • it is difficult to find the right language to describe differences among studentperformance levels for each important criteria or quality. It should be more than just that poor work has less of something than good work. • rubrics may require numerous revisions before they work "just right.“ Sometimes it is difficult to mark with rubric if the student work shows “incoherent attainment”. For instance very wise statements amid very absurd ones. In these cases both a high value descriptor and a low value one have elements of truth in them.
Write a Rubric for Participation in a Discussion Forum 1 2 3
Model Answer! From: Blended learning in higher education By D. Randy Garrison, Norman D. Vaughan
My View • As much as possible, the statements in rubrics should be expressed in subjective terms • Keep objective measures for checklists
Multimedia Specification Assignment Briefing You will be given a minimal content prototype developed by another group. You will evaluate it by producing a five minute long slide show containing screen captures of the application and an audio commentary by yourself. You can also produce slides using text and graphics in order to illustrate concepts and to emphasize points. The slide show should be a sequence of audio and graphics and be runnable in • Powerpoint • Smirk The slide show should be watchable in its entirety as a sequential presentation not requiring user interaction. However you should provide some navigational functionality.
But then you have to mark • Technological aids: Googledocs allows you to create forms which will write to spreadsheets • Studynet too – next academic year will allow the use of departmentally approved rubrics
Averages • Overall Time Taken: approx 31 hours • Breaks: 17 hours • Break understood as any gap longer than one hour between consecutive markings • Average time to mark (without breaks being factored in) approx 39 minutes • After factoring in breaks 11 minutes
Breaks • Sometimes a break represents genuine recovery time between markings • Sometimes leisure • Mentally I experienced it as necessary after every 6 markings
Very different from checklist • Between 20-30 a day • Did not feel tired after marking numbers • Easier to mark • More enjoyable artefacts • Did not require free text feedback
Quality of Feedback Number of Words of Freetext Feedback Given – average 79
Top Two • This was difficult to mark in that it was done in a way totally contrary to the spirit of the assignment - which states:"five minute long slide show containing screen captures of the application ...text and graphics ...The slide show should be a sequence of audio and graphics...should be watchable in its entirety as a sequential presentation not requiring user interaction."unfortunately there were no graphics, the presentation did require my intervention continually - and while there seemed to be a lot of audio files - only a subset of them seemed to play in powerpoint. Therefore I have decided to give onefor the technical correctness section - in that none of the descriptors really relate to what you have produced - and will judge the other categories on their merits.The point about the bars in the holistic evaluation section was well made - as was the positioning of about. However, you did not mention "aesthetic and minimalist design" which is the heuristic which most screams to me the problems of this site. You make a good point about the grouping and labelling of content in the information architecture section - and earlier in the holistic section you talked about the intended audience but this needed to be taken further - with comparison to other websites - and some deliberation on how the grouping and labelling of content might be better adapted to its presumed audience. • Overall not bad - you make a number of good points - the font and button background contrast for instance - also in your navigation through the site you expose some issues like the difficult to find rooms "gallery" button. You also make a number of good points in the heuristic section - but often under the wrong heading. For instance the non-linking "Taj" logo (which should go back to the homepage but doesnt). This, most clearly is a violation of "user control and freedom" (we cannot get out of unwanted states) - and more tangentially violates consistency and standards (typically logos positioned there takes us back to the homepage). However, you associated it correctly with the latter - but also with "flexibility and efficiency of use" - which is much less relevant (this heuristic is more for things like shortcut keys - i.e. doing the same thing in a variety of ways).Overall (this is the feedback I would like you to take forward into other work you do on this msc - particularly your final project report) - the points you made didn't really fit into an organized narrative. You said some valid things in the information architecture section - but you did so in a meandering way. Try in future to be much more organized in the categorization and sequencing of what you want to say in order that it is clearer and has more impact. You do say interesting things - so make sure their order is right!
HE Academy Recommendations • setting realistic targets for achievable turnaround time on assessed work • making the schedule of feedback on assignments clearer to students • timetabling of assignments more evenly through the academic year to avoid bottlenecks • instituting an effective monitoring system for the timing and placing of assessments • providing more timely feedback through activities such as assessment in class time • increasing the number of staff to ensure better staff-student contact on feedback issues • providing feedback in alternative forms • involving students in feedback processes, for example, using them as coordinators • introducing standardised feedback systems, such as standardised forms, hand-in and return procedures • ensuring feedback is given promptly and not delayed by external moderation • auditing practice within an institution and encouraging the spread of effective practice • ensuring that assessed work provides a basis for future improvement by • (a) indicating how future improvement might be achieved (b) ensuring • learning structures (such as semesterised modules) do not foreclose on • the possibility of improvement • using one-to-one tutorials for feedback on assessed work.
Assessment and feedback (Questions in the National Student Survey) Q5 The criteria used in marking have been clear in advance Q6 Assessment arrangements and marking have been fair Q7 Feedback on my work have been prompt. Q8 I have received detailed comments on my work Q9 Feedback on my work has helped me clarify things I did not understand
My recommendations • If marking <100 students use rubrics, if more, use checklists • Recognize the flight structure of marking • Take off, cruising, landing • Make rubrics contain primarily subjective criteria requiring interpretation • Make rubrics simple (no more than 4 criteria per artefact) • Revisit rubrics when writing module evaluation!
