Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis

1. Dr Richard Parsons Centre for Learning and Teaching University of Dundee r.parsons@dundee.ac.uk Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis

2. Overall position of CAA at University of Dundee Why self and peer assessment? Development story Key features Academic author sets exercise online Students use web, email and text messaging Robust reliable, secure Checks for copying Demonstration – create an exercise, live Using the system – key points Good question design Involve the students Teach the students Student activities and responses Marking moderation and plagiarism The future Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis

3. University of DundeeLearning Technology Systems • Blackboard VLE (5.5 Unix, upgrading to Enterprise 6.0 Unix) • QuestionMark Perception 3.4, dual server system • 1. Low stakes/formative server coupled to Blackboard (Oracle Unix database) • 2. High stakes/summative server runs stand alone (Oracle Unix database) • Provide full support for CAA including a “CAA Online” staff education course, initial authoring service • Have Policy & Procedure documents for CAA (based on BS 7988), (also for VLE and eLearning strategies) • Provide support for a range of additional services including online evaluation forms, plagiarism detection service, online staff eLearning tuition, publicise innovative solutions for supporting and enhancing learning (e.g Self and Peer Assessment Program)

4. University of Dundee MLE VLE Integrated Systems GroupWise Web Email Student Management System P3 QMP formative server CAA-01 User exists 3 weeks before appointment Completed daily Module enrolment Bb client Mark return Student exists Staff details also stored in SMS Module enrolments of students Module assessment mark storage LDAP client? Blackboard VLE – ELS 6.0 QMP Exam server CAA-02 Hardware 2 * 480s Version 6.0.8 Users created from SITS SMS system Module enrolment Staff assignments from web system LDAP client Module assessment mark storage Consultancy visit 12 May – 16 May Training teaching staff Special exam enrolment Mark return Library Catalogue access Resource management system LDAP client? e-Directory LDAP Create user name and email process LDAP authentication Remove user process Staff Assignment Record Novell Fileservers Student Desktop Secure web access Richard Parsons, CLT LEU

5. Why Self and Peer Assessment? • Assessment is one of the keys to learning • Being able to assess your own work is a vital skill • Developing peer assessment skills is important for future careers • By carrying out assessments, students learn about the assessment process • Shifts tutor work from post to pre-assignment • Web-based and innovative, so can interest the students

6. Development Story • First a system to replace some lectures when I was overseas • First self-assessment only, but peer assessment seen as important, results emailed and texted to students • Developed into a system that allowed online authoring • Added the ability to tutor mark as well as self & peer assessment • Made more robust based on usage • Remove the back button • Disable right click • Provide more guidance to staff and student users • Added a check for copying and plagiarism • Tidied up code to W3C compliance

7. Online Self & Peer Assessment of Text (Outline of the process) Instructor designs question, sets word limits and details criteria and scoring ranges Question provided, answer researched and composed All files created, URLs provided to instructor 1. Student submits their text answer (may copy and paste) 2. Self assessment completed by student using criteria offered Peer 1 assessment emailed anonymously to student author 3. Student completes an assessment of one of their peers Tutor can mark a selection of answers and moderate and collate results 4. Student completes an assessment of the second of their peers Peer 1 assessment emailed anonymously to student author Final marks are checked and transferred to register 5. Summary of marks presented. Student thanked for completing exercise, brief feedback completed

8. Technical Details HTML pages for input Author completes webform User interaction Perl files to process input Sets question & marking criteria Perl program creates And sets file protections HTML pages for output All operates on a Unix web server, with hyperlinks as required to Blackboard Author access to results Perl file to process output

9. Demonstration of the Program Create a NEW exercise, live and interactive? www.dundee.ac.uk/learning/ilt/

10. Using the System – Key Points • Good question design • Involve the students • Teach the students

11. Student Instructions – Online Self and Peer Assessment Exercise Week containing Tuesday 11 March Receive the first assignment, question and instructions Research the first topic, craft your first text answer Mark your first answer using the criteria provided and mark two peer’s answers Marks sent to student author Receive the second assignment and instructions Week containing Tuesday 15 April Tutor marking of exercise 1 Research the second topic, craft your second text answer Tutor marking of exercise 2 Mark your second answer using the criteria provided and mark two peer’s answers Complete a short evaluation form where your opinion of the exercise will be sought. Summary marks returned via VLE – Self, peer and tutor marks compared. Statistics provided overall to judge the success of the exercise Dr Richard Parsons r.parsons@dundee.ac.uk

12. Student Performance Environmental science exercise - Write a short piece of text that describes the major scientific reasons why global warming is almost certainly occurring at present. The total length of the text must be between 200 to 300 words. • Students performed well, range of marks awarded and obtained. Reasonable correlation between self, peer and tutor marking • Students recognised that they were learning about assessment. Positive feedback received • One of those exercises where you see the students learning

13. Criterion vs Norm-based Marking

14. Student comments demonstrate increased understanding “A good computer program that more classes should use” “A great learning tool for the students. Enjoyable too, better than the labs” “Useful for understanding what markers look for in a piece of work” “It was interesting to see how the marking process takes place, and how you can have an essay that in its own right is well written but does not contain the points needed by the marker” “Assessing others gives me comparison to my own” “Good to see how we can improve our work by our mistakes” “Useful for the ability to receive results via text message” “It was good to be able to compare your own work with others and offered an insight into the basic processes staff must use” “Makes you realise that you need more depth in your answer and to read the question” “It is quite funky”

15. Demonstration of plagiarism/copying/ collusion detection A check for copying is now embedded within the program http://www.personal.dundee.ac.uk/~rparsons/leu/zz02087process.htm Operates in the following manner: Recursive check for 6 word sets, ignoring most punctuation Tutor can insert text from web pages, textbooks and lecture notes as required No service is used to store the student’s work, so no complicated legal issues Will not pick up copying from year to year, or institution to institution

16. Random tutor marking, copying detection No tutor marking, copying detection Comparative tutor marking, plagiarism detection Checks Initial exercise limited training Training in assessment Students experienced at self and peer assessment Experience Little moderation of marking Anomalous marks adjusted Students gain marks for marking correctly Moderation Online Self and Peer Assessment Exercise Formative to Summative Assessments: Considerations Formative Summative Low stakes High stakes

17. Further development Should be able to soon offer • to include images • check submission step • present criteria to students before exercise • students receive marks for marking And perhaps eventually include • feedback on feedback • computer marking of free text

18. Self and Peer AssessmentWhat I leave you with • Online system is available • Relatively easy to use with supporting documentation • It has some features that can engage students (text messaging) and staff (plagiarism detection) • Academic issues remain important • Question design • Students trained in assessment techniques • Student understanding of what constitutes plagiarism • Policy for dealing with plagiarism (formative work) • Further developments are possible, suggestions welcomed