Using Natural Language Processing for Automatic Plagiarism Detection

1. Using Natural Language Processing for Automatic Plagiarism Detection  Miranda Chong*, Lucia Specia, RuslanMitkov Research Group in Computational Linguistics University of Wolverhampton, UK. *miranda.chong@wlv.ac.uk 23rd June 2010 4th International Plagiarism Conference Northumbria University, Newcastle upon Tyne, UK.

2. Overview • Introduction • Challenges • Aims • NLP Explained • Experimental Setup • Findings • Discussion • Further Developments • Summary

3. Introduction • What is plagiarism? • What is plagiarism detection? • As humans it is easy to judge “similar” passages. • But can computers perform this judgement?

4. Challenges • Existing methodologies: Limitations Lexical changes: synonymy, related concepts Structural changes: active/passive voice, word order, joining/splitting sentences Textual Entailment: sentence paraphrase & other semantic variations Multi-source Plagiarism Multi-lingual Plagiarism

5. Vector space model (or term vector model) is an algebraic model for representing text documents (and any objects, in general) as vectors of identifiers, such as, for example, index terms. It is used in information filtering, information retrieval, indexing and relevancy rankings. Its first use was in the SMART Information Retrieval System. The vector space model has the following limitations: 1. Long documents are poorly represented because they have poor similarity values (a small scalar product and a large dimensionality) 2. Search keywords must precisely match document terms; word substrings might result in a "false positive match"

7. Aims • Current research focus: • Proposed framework: Existing approaches + NLP = Improve accuracy • Scope of research: Tackle genuine plagiarism cases External plagiarism Monolingual (English) Free text Document Level

8. NLP Explained [ Natural Language Processing ] Computer system to analyse Written/ Spoken Human Speech Machine Translation Question Answering Linguistics Computer Science NLP Information Extraction Document Summarisation Artificial Intelligence …etc Mathematics

9. Experimental Setup (1) • Corpus of Plagiarised Short Answers • Clough & Stevenson (2009) • Original source documents (wiki articles) : 5 • Plagiarised documents : 57 • Near copy : 19 • Light revision : 19 • Heavy revision :19 • Non-plagiarised documents : 38

10. Experimental Setup (1 cont.) • 4 levels of suspicious plagiarised documents • Near copy (copy & paste without changes) • Light revision (minor alteration) • Heavy revision (rewriting and restructuring) • Non-plagiarised (original text not given) • Alternatively, 2 levels of classification • Plagiarised (Near copy + Light revision + Heavy revision) • Non-plagiarised Note: The 2 level classification was not used in the paper. Please see poster presentation for a comparison.

11. Experimental Setup (2) • System architecture pipeline Suspicious Documents Original Documents Text Pre-processing & NLP Techniques Processed Text Raw Text Corpus Comparison Methodologies Classifier Machine Learning Algorithm Accuracy Score Features Sets

12. Experimental Setup (3) • Text pre-processing & NLP techniques: • Syntactic processing techniques: Dependency parsing Chunking Part-of-speech tagging Baseline Sentence segmentation Stop-word removal Punctuation removal Tokenisation Number replacement Lowercase Lemmatisation Stemming

13. Experimental Setup (4) • Comparison methodologies: • Comparative baseline: Ferret Plagiarism Detector (Lyon et al., 2000) • Machine learning algorithm: Trigram similarity measures Language model probability measure Longest common subsequence Dependency relations matching Naïve Bayes Classifier

14. Sentence segmentation • Determine sentence boundaries • Split text in document into sentences • Allow sentence level matching [ “To be or not to be– that is the question: whether 'tis nobler in the mind to suffer the slings and arrows of outrageous fortune, or to take arms against a sea of troubles and, by opposing, end them.”] [“To die, to sleep no more – and by a sleep to say we end the heartache and the thousand natural shocks that flesh is heir to – ‘tis a consummation devoutly to be wished.” ] - Quote from William Shakespeare's Hamlet

15. Tokenisation • Determine words, punctuation symbols boundaries • Isolate punctuations from words “To be or not to be– that is the question:” ↓ [To] [be] [or] [not] [to] [be] [–] [that] [is] [the] [question] [:]

16. Lowercase • Substitute uppercase characters with lowercase • Generalise word matching “To be or not to be– that is the question:” ↓ to be or not to be– that is the question:

17. Part-of-speech tagging • Assign grammatical tags to each word • Analyse sequence of tags on syntactic level “To be or not to be– that is the question:” ↓ TO VB CC RB TO VB : WDT VBZ DT NN :

18. Stop-word removal • Remove irrelevant words • Keep content words (verbs, adverbs, nouns, adjectives) “To be or not to be– that is the question:” ↓ be or not be - question:

19. Punctuation removal • Remove punctuation “To be or not to be– that is the question:” ↓ To be or not to be that is the question

20. Number replacement • Replace numbers with dummy symbol • Generalise words “63.75 percent of all statistics are made up, including this one.” ↓ [NUM] percent of all statistics are made up, including this one.

21. Lemmatisation • Transform words into their dictionary base forms • Allow matching of similar words Produced Produce

22. Stemming • Transform words into their base forms Produced/ Product/ Produce Produc Computational Comput

23. Dependency parsing • Syntactic analysis of sentences • Stanford parser • Allow matching for related pairs of words at constituent level “To be or not to be– that is the question:” aux(be-2, To-1) cc(be-2, or-3) neg(be-6, not-4) aux(be-6, to-5) conj(be-2, be-6) nsubj(question-11, that-8) cop(question-11, is-9) det(question-11, the-10) parataxis(be-2, question-11)

24. Chunking • Shallow parsing generates parse tree • Keep only the identifiers and structure “To be or not to be” (S (VP (TO To) (VP (VB be)))) (CC or) (PP (RB not) (IN to) (VP (VB be)))) ↓ VP VP CC PP VP

25. Trigram similarity measures “To be or not to be” {“To”, “be”, “or”} {“be”, “or”, “not”} {“or”, “not”, “to”} {“not”, “to”, “be”} • Jaccard similarity coefficient - Ferret Plagiarism Detector Matching Trigrams in suspicious & original Documents ÷ All Trigrams in suspicious & original Documents • Containment measure – Clough & Stevenson Matching Trigrams in suspicious & original Documents ÷ All Trigrams in Suspicious Documents

26. Longest common subsequence • Calculates the longest sequence of matching words between sentences Sentence 1: to be or not to be– that is the question. Sentence 2: should we trust our new PM? that is the question for many voters. LCS = “that”, “is”, “the”, “question” = 4

27. Language model probability measure • N-grams statistical model • SRILM – language modelling toolkit (Stolcke, 2002) • Calculates level of similarity between document pairs • Combining probabilities of n-gram overlaps • Unigrams, Bigrams, Trigrams (tokenised corpus) • 4-grams & 5-grams (chunked corpus) Tokenisation Chunking

28. Dependency relations matching • Count number of matching parsed data between documents • Dependency = Overlapping relations ÷ Number of relations in Suspicious doc = 2 ÷ 4 = 0.5 Original doc Suspicious doc aux(be, to) cc(be, or) neg(be, not) aux(be, to) conj(be, be) nsubj(question, that) cop(question, is) nsubj(question, that) cop(question, is) det(question, the) parataxis(be, question)

29. Machine learning algorithm • WEKA – machine learning toolkit (Hall et al., 2009) • Use feature scores for training • Naïve Bayes classifier to learn a model • The model classify documents according to their level of plagiarism What does a classifier do?

30. Findings (1) Comparison results of feature sets Feature Sets Pre-processing Techniques Comparison Methodology • Trigram containment measure: baseline dataset • Ferret: baseline dataset • Ferret: baseline + lemmatisation • Ferret: baseline + stop-word removal + punctuations removal + number replacement 5. Language model: Bigram perplexity 6. Language model: Trigram perplexity 7. Longest common subsequence 8. Dependency relations matching

32. Findings (2) • Naïve Bayes classifier 10-fold cross-validation 70% accurate • Trigram Containment Measure: Baseline • Ferret: Baseline + Lemmatisation • Ferret: Baseline + Stop-words removal + Punctuation removal + Number replacement • Language model: Bigram perplexity • Language model: Trigram perplexity • Longest Common Subsequence • Dependency Relations Matching Best Features Set Ferret Baseline 66% accurate • 41 features in total All features 60% accurate

33. Discussion (1) • NLP enhances existing approaches • Effective : distinguish between plagiarised & non-plagiarised documents • Deep NLP Techniques (Parsing) + Machine Learning = Promising Framework Accuracy of best features set on two levels (plag/non-plag): 94.74%

34. Discussion (2) • Final human judgement needed to establish cases • Potential educational purposes • Identify suspicious cases for further investigation • Pre-emptive tool to detect incorrectly referenced materials

35. Further Developments • Identify paraphrased texts • Wordnet : correlation 0.72 • Future plans: • Passage level • Integrate Wordnet with current framework • Perform experiments on other corpora (METER, PAN) • Address multi-lingual plagiarism detection Parse tree dependency relations 0.67

36. Summary • Plagiarism detection methodologies can be improved using NLP • These tools can identify possible plagiarised cases • Human intervention will always be required to judge plagiarised cases

37. THE END References • Clough, P., & Stevenson, M. (2009). Developing a corpus of plagiarised short answers. Language Resources and Evaluation, LRE 2010. • Ferret (2009). University of Hertfordshire. [Accessed: 21/3/2010] Available at: http://homepages.feis.herts.ac.uk/~pdgroup/ • Gumm, H. P. (2010). Plagiarism or “naturally given” ? Decide for yourself …. Philipps-Universität Marburg. [Accessed: 17/5/2010] Available at: < http://www.mathematik.uni-marburg.de/~gumm/Plagiarism/index.htm>   • Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I. (2009). The WEKA Data Mining Software: An Update. In ACM Special Interest Group on Knowledge Discovery and Data Mining, SIGKDD Explorations, (11)1. (pp.10-18). • iParadigms (2010). Turnitin [Accessed: 11/5/2010] Available at: <http://turnitin.com/> • Lyon, C., Barrett, R., & Malcolm, J. (2001). Experiments in Electronic Plagiarism Detection. [Accessed: 21/3/2010] Available at: <homepages.feis.herts.ac.uk.> • Stolcke, A. (2002). SRILM- An extensible language modelling toolkit. In Proceedings of the Seventh International Conference on Spoken Language Processing, 3, (pp. 901-904). • ZEIT Online. (2010). AbrechnungimNetz. [Accessed: 17/5/2010] Available at: <http://www.zeit.de/studium/hochschule/2010-05/mathematik-plagiate>

38. …