Dr Arlëne G. Hunter Science Faculty Academic Conduct Officer The Open University, UK

1. Text-comparison software for students:an educational development tool or quick ‘text checker’ – examining student use and perceptions of value Dr Arlëne G. HunterScience Faculty Academic Conduct Officer The Open University, UK

2. Why is this of interest? • increasing awareness of plagiarism in the news and research literature • attributed to ease of access to information on the internet • changing cultural expectations and practices • increasing pressures on students and their learning • potential for damage to an institute’s reputation • most UK HE institutes routinely check work with text-comparison software • most commonly used as a ‘deterrent tool’ • not an automatic outcome: requires academic input, checks and referral • recognition that use of software alone will not change student behaviour, attitudes or understanding • potential alternative or additional use of text-comparison software for formative and education purposes

3. Background – project origins • UK Open University (UKOU) implemented new plagiarism policy and procedures (2008) • improved detection • institutional integration of Turnitin and Copycatch text-comparison software • developed for academic use only • systematic penalty application • academic intervention versus disciplinary action by Academic Conduct Officers • learner experience, level of matching and number of occurrences taken into account • enhanced educational awareness: • creation of Developing good academic practices website • intelligent internal access remembers student progress • publically available on OpenLearn (http://openlearn.open.ac.uk) • most cases raised were low level ‘study skills’ issues rather than ‘serious plagiarism’ • Challenge: could text comparison software be made available to students for formative and reflective use?

4. Logistical issues to consider (1) • ~260 000 undergraduate and postgraduate students studying globally in an online and distance learning environment • assignment handling system separate to text-comparison software • mixed economy for assignment submissions • variable number and formats on and between modules • electronic, electronic or paper (student has the choice) or paper only • text-comparison software formally used on ~57% of all modules (324, n = 570; May 2012) • involves 1384 separate assignment submission dates (× student population) • 87% of assignments checked using Copycatch • 74% of assignments checked using Turnitin • 61% of assignment checked by both systems

5. Logistical issues to consider (2) • Provide one common access point via Developing good academic practices website • not all modules using text-comparison software • variable submission dates and number of assignments • not integrated into student’s core learning environment • Comparability with formal checks • cannot offer student access to Copycatch (data protection issues and relies on comparability checks with current cohort) • not all modules use Turnitin • different repositories used for Turnitin • Integrated support and advice on using system and understanding reports • cost and time implications of providing individual student support • need for generic information with relevance to level and discipline of study

6. Review of UK HE institutes’ usage • very variable practice across UK HE institutes • most common approach is ‘deterrent tool’ • students advised all (or some) assignments scanned by software • no student access to reports (unless there is an issue) • formative ‘feedback tool’ • access to report after submission • student required or ’encouraged’ to seek advice and guidance • use of reports to revise and resubmit work (‘second chance’ submission) • formative ‘educational tool’ • individuals can check draft assignments and complete revisions • student seeks advice and support as required

7. Main drivers for UKOU Text-comparison software system needs to: For students: • encourage students to take responsibility for checking and verifying the authenticity of assessed work • improve student awareness and application of good academic writing skills • enhance student confidence in their ability to write effectively • raise awareness of different types of plagiarism and how to avoid this through good academic practices • prevent minor slips in practice becoming formal issues of concern For staff: • reduce the number of cases requiring formal investigation (plagiarism) oracademic intervention (study skills) • save time (and resources) to invest in educational development

8. Turnitin for students (Tii4S) pilot 2011 Pilot set-up • involved two undergraduate (level 1) Arts and Health modules and eight postgraduate science modules • access system via module website • one submission allowed per assignment • specified open and close dates per assignment • only one submission option visible at a time • instant release of report to student • downloadable guidance on use and interpretation (written for PG) Issues • low uptake by students • very time intensive to set up and manage • students complained of lack of flexibility (e.g. early/late submission) • confusion over accessing final reports

9. Turnitin for students (Tii4S) pilot 2012 Pilot 2 Set-up • involved one undergraduate science module (honours level project) and eight postgraduate science modules • access via module website, downloadable guidance on use and interpretation instant release of report to student • Model 1 – restricted access • one submission allowed per assignment • all submission dates open at module start, closing sequentially on assignment due-dates • two ‘spare checks’ also set up open for duration of module • run with undergraduate and six taught postgraduate modules • Model 2 – unlimited access • multiple submissions allowed per assignment (once per 24hr period) • each report over-writes the previous one • used with two postgraduate skills and project modules only

10. Primary pilot objectives To determine: • if students would use the text-comparison software to check work prior to formal submission; if so: • what type of students would use it • when would they use it • ease of technical use of the system • perceived ease by which reports could be interpreted • impact of offering restricted versus unlimited access to the software • impact on student writing (perceived and actual changes in practice)

11. Student use of Tii4S

12. Levels of ‘raw matching’

13. ‘Raw matching’: possible concerns

14. Timing of student use of Tii4S

15. What type of student uses Tii4S • no differentiation in ‘type’ at undergraduate or postgraduate levels • students have moderate to good academic writing skills • majority of scripts exhibit ‘no issues of concern’ • confidence that some type of matching (even >20%) is ‘acceptable • majority of matching that occurs is minor • discrete strings of words from various sources • discrete chunks of text • patchwork plagiarism from a source • most students did not (need) to revise script prior to formal submission • majority with ‘potential matching of concern’ have revised their script prior to formal submission • most check their work a few days prior to submission • timing due to wide variety of reasons • some check on submission date (final confidence check?)

16. Perceived ease of use Based on questionnaire response (n=33; 15%) • 87% perceived Tii4S useful to very useful • 92% found it technically easy to use • 82% felt they were able to interpret the reports • but 18% would appreciate additional guidance to make sure • 55% felt using Tii4S had confirmed their academic practice • but an additional 33% felt it had had no impact on their practice • 12% stated using Tii4S had encouraged them to change their approach to academic writing • of which 3% believed it had made a significant change to how they wrote • 97% stated they would use it again • 67% of these said they would use it at every opportunity available • 30% indicated they would use it selectively or for a specific reason

17. Perceptions of usefulness • Open comments collected from students via the questionnaire and from module and programme specific forum postings • coded as positive, negative or suggestion • BUT equal numbers of comments received for all (further analysis needed!) • General positive perceptions • offers reassurance of good practice • develops confidence • confirmation that work does not contain plagiarism • good way to check referencing and writing is fine • helps with recognition of what could be seen as plagiarism and allow revision • surprise that software did not pick up all (known) matching, and need to be aware of own writing practices

18. Emotive responses to Tii4S • some students expressed very strong negative and emotive reactions towards Tii4S • generally linked to difficulties using the system and/or interpreting the results: • report does not discriminate between plagiarism and acceptable text • complete waste of time to get a 0% match • terminology of report confusing in terms of ‘originality’ versus ‘similarity’ • no use unless know what is and is not deemed ‘acceptable’ by university • unsure what to do with the report • or implications for academic integrity of students • good to know ‘cheats’ will be caught via formal checks, BUT concern Tii4S could be used to avoid detection (hence the need to limit access) • students at university should not need such tools to identify poor academic practices, but focus on developing their writing skills

19. Restricted versus unlimited access Restricted model • vast majority of students have used submissions for expected assignments (e.g. check 1 for assignment 1 etc.) • no student has used up all their checks • students with ‘high’ matches use one or both spare checks for revised script • late submissions generally using ‘spare checks’ • four students (on two modules) have used one or both ‘spare checks’ to check other module assignments • no one has attempted to register under multiple user-names • students reusing system even when getting very low matching • no difference between undergraduate and postgraduate behaviour • but, talk on undergraduate forum of checking random documents rather than assignments (from students deriding need for Tii4S)

20. Restricted versus unlimited access Unlimited model • Tii4S opened for PG skills module for final assessment only • majority of students submitted work once • Tii4S opened for PG project module since February 2012 • students had already submitted two assignments • assignment 3 due late March and assignment 4 due early July, with final project not due until October 2012 • all students have only submitted PG project work • vast majority of students have submitted once • only five students have submitted >1 draft script • one student submitted script once for assignments 3 and 4 • two students submitted assignment 3 twice (within 24 hours) prior to formal submission • one student has submitted x6 and one has submitted x7

21. Unlimited access: multiple submissions • Student who has submitted 6 drafts • two versions of assignment 3 submitted ~1 month and then 4 days prior to formal submission • three versions of assignment 4 submitted ~3 months, ~6 weeks, 11 days and 8 days prior to formal submission • level of matching is high due to chunks, strings and patchwork plagiarism • similarity index decreased from 49% to 20% (removal of chunks) • script still contains plagiarised material • Student who has submitted 7 drafts • did not submit script for assignment 3 • assignment 4 submitted • twice in May (6 days between submission) • five time in June (2 days between each submission) • level of matching is low (<10%) but contains patchwork plagiarism and strings of matched text • student resubmitting extended drafts

22. Impact on students’ assignments • too early to judge full impact on the number of cases being referred for academic discretionary action or disciplinary investigation • general sense that Tii4S is confirming good practice and alerting students of minor slips in practice • restricted model encouraging students to ‘think’ about their work and appropriateness of writing • majority of students with unlimited access have not (yet) taken full advantage of system to check successive drafts • tendency to submit just prior to assignment ‘due date’ • change in practice leading up to end of module assignment?

23. Concluding findings • students are generally positive about the formative use of text-comparison software • boosts confidence • enhances awareness of good practice (and possible plagiarism) • encourages students to think more about their writing practices/ approach • students are generally using the system ‘as expected’ • i.e. no abuse by setting up multiple accounts or checking documents that do not belong to the student • uptake is generally low (average ~20%), possibly due to • ease of finding Tii4S (even though linked on module website) • lack of motivation to use system • ‘fear’ of what may be uncovered • confounding pressures on studies • reflective comments indicate appreciation and support for restricted access over unlimited access model • promote individual reflection • prevent ‘others’ abusing system

24. Dr Arlëne G. HunterScience Faculty Academic Conduct OfficerThe Open UniversityMilton Keynes, England, UK. MK7 6AAA.G.Hunter@open.ac.uk