Paper 99 Can the computerized peer-assessment of digital stories fully assess student subject knowledge?

1. Paper 99Can the computerized peer-assessment of digital stories fully assess student subject knowledge? Dr Phil Davies University of Glamorgan Email: pdavies@glam.ac.uk Twitter: PhilWales

2. How to use Digital Storytelling? • Life stories using media – Dana Atchley • Centre for Digital Storytelling – Joe Lambert <inception 1993> • Social Sciences – interleaved with Narrative • Capture Wales Project <BBC> • How to assess knowledge/reflection/application of ideas/personalization? • The student becomes PART OF THE TASK not merely a REPORTER – reports in the 1st not 3rd Person • Synthesize ideas then present as if personal • THE IMPACT OF MULTIMEDIA

3. Cynical Essay Scenario • GOOGLE RULES OK? • Numerous links – actually follow some of them/even make a note of the reference and COPY into references section • HAVE TO PUT INTO OWN WORDS OTHERWISE THINK IT IS COPIED – takes approx 2 hours – do it properly! • Place in acceptable format – submit it – forget it – count 28 days for turnaround – like/dislike mark – may look at comments – difficult to work out comments/feedback as have forgotten what the essay was about in the first place • Did they really understand it? • Could they apply their knowledge? • Will they remember it for a long time? • Have they really thought about the implications of it? • Do they know what others in their group think about it?

4. How would we assess their knowledge? • In order for the student to ‘become’ that person they must perform significant background research in order to understand the needs of an individual in that position. • The presentation of the story and the associated research links provides a means of evaluating a student’s knowledge. I’ve described this in the past as ‘Technological Role-Play’.

5. Needs of Marking • Self-Assess own Digital Story • Peer-Assess others – Criteria and Framework • Generate Raw Peer-Assessed Median • Check if Under or Over Markers in General • Modify their Peer-Markings accordingly • Generate a Compensated Peer Average Mark • Reward the Peer-Markers for Showing Consistency: • Consistently high or low • How helpful were peer-comments • Were comments and marks provided consistent • Remove the Inconsistent Markers from Peer-Mark Generation and Re-Calculate the Peer Generated Average Mark • Allow Students to re-submit to tutor for grading after having peer-marked others and also having received peer-feedback not marks • Each Digital Story they view will be unique – not copied – personalized –produced Digital Story isNOTanonymous - however marking process is still ANONYMOUS • CAPODS – Computerized Assessment by Peers of Digital Stories

8. How the final mark was generated • The final grade (assignment worth 40% of module’s mark) awarded to the students comprised of: a) peer generated grade for their Digital Story (15%) b) a tutor generated grade for their Digital Story (15%) c) a tutor/system generated grade for their consistency shown in marking and commenting (10%).

9. Colour Blind Digital Story • Colour Blind Digital Story: Paul Stokes • ‘I never thought about how difficult it was for a student with a visual impairment to do the things on a PC that I find easy – it will really make me think about how I develop my web pages in future’.

10. EMPOWERING STUDENTS • Final Year student this year • Scott’s Family DSt – Wanted everyone to know! • Produce a digital story <to be peer-assessed> that will demonstrate how you learnt a particular skill (preferably computing) • A Student’s (Scott) perspective of studying a course at the University

11. Questions/Points Are they true? • Students don’t copy words – they copy images, audio, video, cartoons, etc • Only really matters if they are gaining marks for these images etc? • Does a student on writing an essay get more marks for using ‘big words’ or having neat hand writing or producing something that is grammatically correct – should only be if it is an assessable learning outcomes of the module • Require media skills to create Digital Stories • Not suitable for non-computing students • not true as a Digital Story is easily created using Powerpoint – computer literate –> Capture Wales?

12. Questions/Points Are they true? • Can only develop limited range of suitable Digital Stories for Higher Education • Anything where we can show the processnot just an end product by use of multimedia • Examples • Consider how a disability of your choice would affect your use of the web – how could web pages be developed that could make it easier for you? • Consider how in the past you’ve learnt a particular skill. Explain how you learnt that skill and then map it to Learning Styles/Theories • Consider nursing a terminally ill patient with X, how would you support them – could be modified to you are a terminally ill patient with X, show us what is it like for you? • What effect does Global Warming have upon a particular region of Y? => you live in Y, tell me what it it like for you living in this area since Global Warming occurred, etc.

13. Questions/Points Are they true? • Only surface knowledge – spend too much time on media development and not on learning • Again related to the marking scheme and module outcomes- same could be said of getting it grammatically correct • Students will not peer-assess ‘well’ and will give high marks to their friends • Not if they are being marked on their marking ability and consistency – marking is still anonymous • Students will not learn how to write essays and reports • Is this a learning outcome in ALL modules in HE?

14. Questions/Points Are they true? • Overseas students will be disadvantaged due to poor language skills • Easier to hide grammatical and spelling failings using voice/media presentation • Students do not learn how to cite references • Probably the opposite is true as they must cite using textual banners during the course of the Digital Story

15. Any Questions – main one > can we view these stories? http://www.comp.glam.ac.uk/pages/staff/pdavies/DSt/DSt.htm