1 / 30

SOLSTICE Conference 2015

SOLSTICE Conference 2015. Technology enhanced formative assessment: Participant experiences, relative efficiencies and tutor learning curves Dr W. Rod Cullen, Manchester Metropolitan University. 4 th & 5 th June 2015. Dr W. Rod Cullen Susan Gregory Dr Neil Ringan Mark Roche.

jtrimble
Download Presentation

SOLSTICE Conference 2015

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SOLSTICE Conference 2015 Technology enhanced formative assessment: Participant experiences, relative efficiencies and tutor learning curves Dr W. Rod Cullen, Manchester Metropolitan University 4th & 5th June 2015

  2. Dr W. Rod Cullen Susan Gregory Dr Neil Ringan Mark Roche

  3. Formative assessment... “... is designed to provide learners with feedback on progress and inform development, but does not contribute to the overall assessment.” QAA 2000 “I can’t get my students to do work that doesn’t get a mark” “My students don’t use the feedback I give them – sometimes they don’t even pick it up” “That all looks very impressive, but how long is it going to take me to do that?”

  4. Session outline Narrative account of evolution of “Enhancing Learning Teaching and Assessment with Technology” unit (ELTAT) • Delivery model • Feedback methods • Feedback efficiencies • Participant experiences • Conclusions and recommendations

  5. A brief history of ELTAT

  6. Learner Profile • Range of backgrounds • New and experienced academic staff • Academic support staff • Technical support staff • Majority working full-time but some part-time • Differing assessment requirements • PG Cert Academic Practice • MA Academic Practice • CPD for credits • CPD non-credits • Just like real students!

  7. Online Preparation In ClassSession 1 In ClassSession 2 Onlineformative activity Onlineformative activity Formative delivery model Week 1 Week 2 Online Preparation Feedback Feedback Mini (e)Portfolio

  8. Providing feedback – Text based

  9. MS Word file including annotations Audio feedback MP3 audio file

  10. Screen capture feedback

  11. Data Collection Tutor Activity – since 2007 • Kept a detailed feedback activity log • Worked out time spent engaged in each feedback activity. End-to-end process and each stage: Download, Annotate, Write/ Record, Publish, Upload Participant experience – 2007-2014 • Interviewed participants about experiences • End of unit survey • Submission data • Attendance • Informal anecdotal evidence Ongoing Opportunistic Action Research

  12. Early Experiences DCVLEs

  13. Feedback methods (DCVLEs)

  14. Producing formative feedback Issues for Susan 1. Getting up to speed with new technology (Pebblepad) 2. Technical problems with codecs and files sizes (common with web-based video) 3. Initially uncomfortable with recording process 4. Got up to speed with audio format more quickly Cullen W. R. (2011)

  15. Attendance and engagement with formative tasks 2007-10 Percentage MASummative Assessment 1. Reasonably high overall attendance 2. Generally high (70% +) submission rates for formative tasks 3. Use of the feedback is embedded in the online prep and F2F activities 4. Consistently lower submission rates in Week 4

  16. Ongoing development 2011 Programme Review: Designing Effective Online and Blended Learning (DEBOL) • Change in unit title • Neil and Mark – contributions as specialist lecturers 2013 Major Programme Review: Enhancing Learning Teaching and Assessment with Technology (ELTAT) • 30 Credits • Much longer delivery – 12 weeks • More time to complete tasks (and provide feedback) • Team approach to formative feedback – Tutors allocated group of student to work with throughout the course

  17. Recent experience (ELTAT)

  18. Same delivery and assessment model

  19. Feedback methods

  20. Producing formative feedback 1. Rod + Susan very similar times 2. Neil + Mark learning the ropes 3. Pronounced for video and audio About the task as much as the technology 4. Similar times for written feedback – new task 5. Similar times for TII (Written + Audio)

  21. Team overview 1. For team as a whole consistent ≈ 55 mins per submissions (end-to-end) 2. Variation in what the time is required for 3. Download times minimal 4. Upload of audio and video = minimal time but 2-3 mins more than written. 5. No down/upload for TII 6. Additional “publish” stage for audio 7. Reading through and annotating the submissions is most time consuming activity

  22. Efficiency of production • Written feedback: • Word count tool in MS Word • Divided by the total time (end-to-end) to produce the feedback • Screen capture and audio feedback: • Counted spoken words in 1 minute of recorded feedback = est. speaking rate. • Length audio and video recording X speaking rate = estimate of the total number of words in each piece of feedback. • Divided by the total time (end-to-end) to produce the feedback • Turnitin: • Pasted the general text comments into a word document – Word count tool. • Length of audio feedback X tutors speaking rate = estimate of number of word spoken. • Work count + Words spoken = estimate of the total number of words in each piece of feedback. • Divided by the total time (end-to-end) to produce the feedback Annotations (comments in MS word and Quickmarks in Turnitin) not included

  23. Estimated Feedback words/per minute 1. Audio and Video screen capture – learning curve? Task as well as tech 2. Written feedback - Personal preferences and expertise is an influence 3. TII least efficient – multi-media + suspect we made more comments in the text (Quickmarks) 4. Mark has felt need to reflect on his feedback on all tasks 5. Comparing efficiencies across technologies is more complex than anticipated

  24. Student perspectives Prefer audio and video to written feedback 50:50 split between audio and video Annotations important Utilised differently Personal and Engaging Video more impact Understandable Audio more reflective

  25. Conclusions • Take home messages • To embed formative assessment into a curriculum • Design it into delivery model – make the value of the feedback clear • Align with summative assessment and signpost this to students • Select feedback technology best matched to the type of tasks • Be prepared for the learning curve • Learning curves associated with • “Technical skills” • Feelings of self-consciousness • Familiarity with requirements of the formative task • Technical aspects not as time consuming as cognitive • Audio feedback can be produced quickly and simply • Model of delivery can influence engagement – need to add value to completing formative tasks • Participants indicated a preference for audio and video screen captures • Linked to characteristics of the task • Possible novelty aspect

  26. Just in case section

  27. Feedback methods

  28. Data Collection • All tutors kept details feedback activity logs relevant to the technologies being used

  29. Logistical considerations DCVLEs & DEBOL versionsCore Option 10 credits – Nominal 100 hrs Very limited F2F contact (4 x 3 hours sessions - 12 hours in total) ELTAT 30 Credits – Nominal 300 hours More F2F Over a long period

  30. Design considerations • Relevant • Purposeful

More Related