CAA-2003

1. CAA-2003 Genre Analysis and the Automated Extraction of Arguments from Student Essays Emanuela Moreale e.moreale@open.ac.uk Maria Vargas-Vera m.vargas-vera@open.ac.uk

2. Introduction • Automatic extraction of arguments from student essays • Aim • “Proof of concept” • Provide feedback & aid assessment • “Qualitative” • Aims for limited “understanding” • Theory-based (v.s. stats approaches) • Who for? • Mainly tutors • Also students CAA2003 - Genre Analysis & Arguments in Student Essays

3. Topics of Discussion • Argument Modelling in Academic Papers • Ontologies / Schemas • Swales, Teufel, Hyland, ClaiMaker • Our own categorisation • Experiments in using NLP (IE techniques) for argument extraction • Gate, MnM, Amilcare (Annie) • Student Essay Viewer (SEV) + our categorisation • SEV: architecture, GUI, feedback and assessment • Results / Future Work CAA2003 - Genre Analysis & Arguments in Student Essays

4. Argumentation Modelling in Papers Research into - supporting rhetorical argument construction / writing process - tools for “making thinking visible”: • Writer’s Assistant • Belvedere • SenseMaker Research on Paper Structure • Swales’s CARS model • Teufel • Hyland • ClaiMaker CAA2003 - Genre Analysis & Arguments in Student Essays

5. Argument Construction / Writing • Writer’s Assistant • Text and Argument views • Belvedere and SenseMaker • Develop scientific argumentation skills • Users: unpracticed beginners • Focus on rhetorical relations between items: evidence, claims, explanations Interesting but… • is this generic enough? • Our users: tutors and university students CAA2003 - Genre Analysis & Arguments in Student Essays

6. Argumentation in Papers: Swales • Swales – CARS model (1990) • Research paper introductions • Three moves with 3 or 4 steps each: • Establish territory • “There has been a great interest…” • Establish a niche • “This approach fails to…” • Occupy niche • “The purpose of this paper is to…” • “The paper is structured as follows…” • Influential model CAA2003 - Genre Analysis & Arguments in Student Essays

7. Argumentation in Papers (2): Teufel • Teufel et al. (1999) • Extend CARS model to whole essay • Add a few new moves • Bias  automatic summarisation • Focus: mark the purpose of the paper in relation to previous literature • Types of sentences: • Background (background knowledge), other (outside this paper), own (author’s new contributions); • Aim - main research goal of this paper • Textual – structure of the paper • Contrast – own work to other work • Basis – work used as basis to this work • Human annotators (not an implemented system) CAA2003 - Genre Analysis & Arguments in Student Essays

8. Argumentation in Papers (3): Hyland • Hyland’s Metadiscourse Schema(1998) • Metadiscourse in academic texts; 1. Textual metadiscourse - allows the recovery of the writer’s intention by explicitly establishing preferred interpretations: • Code glosses (namely, in other words…) - relates propositions to each other and to other texts: • Logical Connectives (but, therefore, thus…) • Endophoric Markers (noted above, see Fig 1) CAA2003 - Genre Analysis & Arguments in Student Essays

9. Argumentation in Papers (4): Hyland Hyland’s Metadiscourse Schema(cont.) • Metadiscourse in academic texts: 2. Interpersonal metadiscourse alerts readers to the author’s perspective to both the information and the readers themselves (expresses the writer’s persona). - Hedges: might, perhaps, it is possible…; - Emphatics: in fact, definitely, it is clear…; - Relational markers: Frankly, note that, you can see that - Attitude markers: Surprisingly, I agree, X claims - Person markers: I, we, my, mine, our CAA2003 - Genre Analysis & Arguments in Student Essays

10. Mixed Approaches We also looked at SchoolOnto, a project aiming to: • model arguments in academic papers • devise an ontology for scholarly discourse The team have also produced a tool called ClaiMaker Claim types: general, problem-related, evidence-related, similarity and causal CAA2003 - Genre Analysis & Arguments in Student Essays

11. ClaiMaker • ClaiMaker is mainly meant for academic papers: • Academic paper = set of interlinked parts • Statements: in one paper -> in another paper • Result: network of cross-referencing claims • Use - Taken a complete paper, claims are annotated manually by the reader; - In the future, paper authors may publish: paper AS WELL AS claims; - ClaiMaker could be used when writing a paper (claim search, issue focusing) CAA2003 - Genre Analysis & Arguments in Student Essays

12. Towards Our Categorisation • Interesting categorisations/tools but imperfect fit • ClaiMaker (tool): • academic papers (not student essays) • currently, claims are entered manually • Belvedere/Sensemaker (tool): • Unpracticed beginners in scientific argumentation • Argumentation Categorisations so far: • Target academic papers • Swales: introduction only • Teufel: manual system, no implementation • ClaiMaker: ontology OK for manual editing but for automated system? CAA2003 - Genre Analysis & Arguments in Student Essays

13. Towards Our Categorisation (2) • With these concerns in mind, we identified categories of possible arguments in student essays. • Preliminary manual analysis of essay texts • Some categories may have been “influenced” by ClaiMaker • Our initial categories were: • Definition • Comparison • General • Critical thinking • Reporting • Viewpoint • Problem • Evidence • Causal • Taxonomic • Content/expected • Connectors CAA2003 - Genre Analysis & Arguments in Student Essays

14. Towards Our Categorisation (3) • A review of the first categorisation led us to: • Reduce number of categories • Cognitive overload • Ease of display • Reassess main categories • Main elements in a university-level essay: • Showing knowledge of background / research area • Reporting other people’s views • Demonstrating analytical thinking • Contrasting & comparing viewpoints • Define and “relate” terms / concepts CAA2003 - Genre Analysis & Arguments in Student Essays

15. Our Categorisation CAA2003 - Genre Analysis & Arguments in Student Essays

16. Our Categorisation: Characteristics • Overall, remarkable similarities across categorisations • STRATEGY: • Teufel: Textual category • Hyland: Endophoric markers “see section 4” • Swales: M3, S1a: Purpose & M3, S1d: Structure • REPORTING: • Swales: M1, S3 verbs like “show, demonstrate, establish” • Teufel: Other • Hyland: Evidentials e.g. “according to X (1990)” • POSITIONING: • Swales: Move 2 (Establishing a niche) • Teufel: Contrast • Hyland: Emphatics, Attitude Markers, Person Markers CAA2003 - Genre Analysis & Arguments in Student Essays

17. Our Categorisation: Characteristics BUT • No AIM category (unlike Teufel’s schema): • Essay have implicit aim to answer the essay question(s) • No distinction between OTHER and OWN • Tricky distinction anyway • Not applicable to student essays • New category: Content/Expected • Relates to essay content (material covered) • Student essay-specific • Domain-dependent and Tutor-specified CAA2003 - Genre Analysis & Arguments in Student Essays

18. Natural Language Technologies • Problem to be solved: • How can Natural Language technologies help in the automatic extraction of arguments? • Requirements: • provide tutors with a tool that is easy to use • Analysed Tools: • Gate identifies entities (names of people, organizations, dates, money and locations). It uses gazetteers and regular expressions • Good tool for developers • Too difficult for tutors to use • MnM • Uses Amilcare, an Information extraction engine • Requires a training set from which it learn patterns. These patterns can then be recognized in new documents (from the test set). CAA2003 - Genre Analysis & Arguments in Student Essays

19. Natural Language Technologies • What is Information Extraction (IE)? • IE extracts facts about pre-defined types of information from documents; • Origin: IE research begun late 1980s, as a product of cold war: automatic extraction of information from naval messages. • Uses: • An IE system designed for terrorist domain might extract perpetrators, victims, physical targets, weapons, dates and locations (Riloff et al. 1993) • Domains: • IE has been used in scientific articles, bibliographic notices [Proux et al. 1997], medical records [Soderland et al. 1995]. CAA2003 - Genre Analysis & Arguments in Student Essays

20. Natural Language Technologies • First MnM implementation was able to fill templates such as the following: • “visiting-a-place-or-people” event • Template looks like this: • Visitor • Date of visit • Place being visited • Pattern learnt: • X visited Y: where • The type of X is Person and • The type of Y is Location CAA2003 - Genre Analysis & Arguments in Student Essays

21. Natural Language Technologies • Information Extraction seems very appealing for finding patterns • IE tools work very well in narrow domains BUT • In our domain (student essays), it is not easy to define in advance templates containing concepts and relations between them: • We cannot anticipate which concepts / relations students going to be use CAA2003 - Genre Analysis & Arguments in Student Essays

22. Proposed Solution • Our approach combines: • cue phrases with • a set of patterns without template filling. • We started off by defining gazetteers of cue phrases and patterns written as regular expressions. • The set of patterns is organised on the basis of our categories • We then developed a tool to show this “in use”: the Student Essay Viewer (SEV). CAA2003 - Genre Analysis & Arguments in Student Essays

23. Architecture – SEV Components CAA2003 - Genre Analysis & Arguments in Student Essays

24. About SEV • WHY: Gives a visual representation of argumentation within an essay • INTUITION: • essays with considerably more “highlighted text” contain much more argumentation (and “content”) THUS • they attract higher grades than essays with little highlighting CAA2003 - Genre Analysis & Arguments in Student Essays

25. SEV - Users • Main Target Group: Tutors • Context: assessment / feedback • SEV’s automatic counts indicator • Citation highlighting • Amount/distribution of highlighting • Possible Target Group: Students • Formative assessment • Essay has little highlighting -> revise • Needed type of argumentation CAA2003 - Genre Analysis & Arguments in Student Essays

26. SEV - Interface • SEV looks like a simple webpage • Top part • categorisation browser/selector • Main part • Essay • Initially not annotated (no highlighting) • After category/categorisation selected, the annotated essay is displayed • Automatic link counts (by type) also displayed CAA2003 - Genre Analysis & Arguments in Student Essays

27. SEV: Initial Presentation CAA2003 - Genre Analysis & Arguments in Student Essays

28. SEV: After Selecting “Definition” CAA2003 - Genre Analysis & Arguments in Student Essays

29. SEV: Highlighted Essay (Ours) CAA2003 - Genre Analysis & Arguments in Student Essays

30. SEV & Assessment 3 underlying assumptions: • 1) Number of annotations: Bad essays have fewer annotations than better essays • 2) Critical analysis and background are two essential elements in essays: corresponding annotations expected to closely correlate with grade; • 3) The relative importance of annotation categories may vary & depend on essay type CAA2003 - Genre Analysis & Arguments in Student Essays

31. Assumption 1: annotation count & score Correlation: r =0.878; N=12;p<0.01 ANOVA F-stat: F(1,10)=33.501; p<0.01 CAA2003 - Genre Analysis & Arguments in Student Essays

32. Assumption 2: essential elements in essay Corr(Positioning/Score): r =0.753; N=12;p<0.01 MANOVA Positioning/Background: F(1,10)=18.462; p<0.01 NOTE: Background = expected + reporting CAA2003 - Genre Analysis & Arguments in Student Essays

33. Assumption 3: Essay “Types” + Annotations • Analysis of 4 different assignments • Questions: • Say How and Why • Opinion about X • Describe and Discuss • Give example of X and critique X • Basic Idea: different essay questions require different “link/annotation profiles” • E.g. “Summarise X” will be answered by an essay with many “reporting” links • Better fit for some assignments (1) than others (4) CAA2003 - Genre Analysis & Arguments in Student Essays

34. Results - Summary • Student essay metadiscourse schema • compared with research essay categorisations • Links: Argument Extraction and Assessment • Total number of links correlated with score • Positioning and background most useful for score prediction • Different link profiles to answer different essay questions • Student Essay Viewer (SEV) • Highlights instances of our categories in an essay • Helpful to tutors • Quick visualisation of type of argumentation • Quick visualisation of amount/distribution of arg. • Helpful for students • Feedback about the essay (e.g. lacking categories) CAA2003 - Genre Analysis & Arguments in Student Essays

35. Results & Future Work • Real Data (actual postgraduate essays) • Higher number in the future • Encouraging results • Future Work • SEV could categorise longer linguistic units • e.g. sentences or paragraphs • SEV to provide reasons why a specific categorisation is assigned to a linguistic unit • Explanation in pseudo-natural language • Inclusion of “Essay Question Analysis” Tool • Particularly useful for students CAA2003 - Genre Analysis & Arguments in Student Essays

36. Conclusion • Generic metadiscourse schema for student essays • Links to similar schemas for research papers • SEV: Argumentation visualisation tools • Detects argumentation by using: • our annotation schema and cue phrases • Target users: tutors (and students) • Can aid Assessment • Link between annotation count and score • Positioning and background links most important in predicting score • Formative feedback to students CAA2003 - Genre Analysis & Arguments in Student Essays