Evaluating Legal Argument Instruction with Graphical Representations using LARGO

1. Evaluating Legal Argument Instruction with Graphical Representations using LARGO Niels Pinkwart, Vincent Aleven, Kevin Ashley, and Collin Lynch Clausthal University of Technology Carnegie Mellon University University of Pittsburgh

2. Outline LARGO (Legal ARgument Graph Observer): ITS to engage students in analyzing & reflecting about examples of expert Socratic reasoning • US Supreme Court Oral Arguments as learning resources • Diagrams to visualize argument as hypothesis testing: analysis and feedback • Evaluation of LARGO Evaluating Legal Argument Instruction with Graphical Representations using LARGO

3. US Supreme Court Oral Arguments • Important part of decision process • Attorneys propose a decision rule (“test”) to determine how to decide a case • Justices challenge these tests by posing hypothetical scenarios (meaning, consistency with past cases, legal and policy implications) Evaluating Legal Argument Instruction with Graphical Representations using LARGO

4. An Example Case • Example:Kathy Keeton v. Hustler Magazine, 465 U.S. 770 (1984) • Facts: Kathy Keeton sued Hustler Magazine for libel in US District Court in New Hampshire. Hustler is an Ohio corporation with its principal place of business in California. Ms. Keeton was not a New Hampshire resident and had almost no ties there. Each month, Hustler sold about 15,000 issues in New Hampshire, which was only state where Keeton was not time-barred under a statute of limitations. • Question: Did the courts in New Hampshire have personal jurisdiction over Hustler Magazine? • Conflicting principles:Fairness to defendant vs. right of state to redress in-state injury. Evaluating Legal Argument Instruction with Graphical Representations using LARGO

5. Example: Tests and Hypotheticals Test Hypo Response:Analogize Hypo Response: Distinguish, Modify

6. Educational Value • Training in skills of argument important part of legal argumentation • Transcripts realistic and valuable examples of expert Socratic reasoning • Analysis of transcripts: • can help learning these skills • provides opportunities for reflection • But: complex material, hard to understand! Evaluating Legal Argument Instruction with Graphical Representations using LARGO

7. A LARGO Argument Diagram LARGOArgument Diagram Palette Evaluating Legal Argument Instruction with Graphical Representations using LARGO

8. Intelligent Support: How to interpret diagrams? • Problem 1: legal argumentation is an ill-defined domain • Different views possible • Difficult to define “correctness” in this interpretive field • Problem 2: arguments consist of natural language texts – NLP based approaches seem error-prone here • LARGO approach: Attempt to find characteristics in argument diagram • Weaknesses (areas of potential problems) or • Opportunities for reflection Evaluating Legal Argument Instruction with Graphical Representations using LARGO

9. Context: links between graph elements and important passages in the transcript. Structure: graph portions that correspond to pre-defined “argument patterns”. Content: adequateness of students` free-text descriptions in diagram boxes. Example: No “test” elements linked to transcript passage which contains important test Detected by graph grammar. Example: Diagram contains hypothetical not related to any fact or test element. Detected by graph grammar. Example: A description of a proposed test in a “test” diagram element is of poor quality. Detected by collaborative filtering. Diagram characteristics Evaluating Legal Argument Instruction with Graphical Representations using LARGO

10. A rates the test descriptions provided by B and C after having entered his own. Detection of Content Weaknessesthrough Collaborative Filtering Student A Student C Student B Evaluating Legal Argument Instruction with Graphical Representations using LARGO

11. Feedback: Self Explanation Prompts • Principle: Use detected characteristics for self explanation prompts • Research results: SE beneficial for learning, effective metacognitive strategy • Offer opportunities for reflection about specific parts of Socratic reasoning examples: • the presumably weak parts of student’s argument analysis, and • the good parts still worth reflection Evaluating Legal Argument Instruction with Graphical Representations using LARGO

12. Feedback Example Evaluating Legal Argument Instruction with Graphical Representations using LARGO

13. LARGO v. Cognitive Tutors • Study of expert argument examples • No cognitive model of diagram construction • LARGO does not insist that students create a “correct graph” • Because of ill-structure, it may not be feasible to identify a comprehensive set of correct graphs • Getting the diagram right is not the main thing: we want students to reflect on the quality of the arguments • Student may be more deliberate and thoughtful without corrective feedback Evaluating Legal Argument Instruction with Graphical Representations using LARGO

14. Experiment (Fall 2006) Goal: Evaluate LARGO, compared to standard note taking Hypothesis:UsingLARGO improves • understanding hypothetical reasoning method, • ability to recognize/reason about examples in near and far transfer legal domains. Participants:28 volunteer law students in 1st-semester Legal Process course. Task: Read SCOTUS oral arguments; represent hypothetical reasoning. Experimental condition: Use LARGO graphical argument representation and feedback to identify/relate elements of hypothetical reasoning. Control condition: Same oral arguments and focus on hypothetical reasoning, but use tool with text-based word-processing and highlighting (no feedback). Procedure: Over a four-week period: 1. (2h): Pre-test & tool introduction 2. & 3. (2x2h): Apply tools to Burnham & Burger King (personal jurisdiction) arguments 4. (3h): Post-test Evaluating Legal Argument Instruction with Graphical Representations using LARGO

15. Control Condition Tool Evaluating Legal Argument Instruction with Graphical Representations using LARGO

16. Pre/post test measures Evaluating Legal Argument Instruction with Graphical Representations using LARGO

17. Overall test scores Evaluating Legal Argument Instruction with Graphical Representations using LARGO

18. Test scores by item Evaluating Legal Argument Instruction with Graphical Representations using LARGO

19. Divide sample by LSAT score *p< .05 Evaluating Legal Argument Instruction with Graphical Representations using LARGO

20. Post-test scores in Low LSAT Group * *p< .05 Evaluating Legal Argument Instruction with Graphical Representations using LARGO

21. Post-Hoc analysis: Classify questions in terms of argument model. One category: evaluating hypotheticals with respect to a proposed test Assume that Mr. Grutman’s proposed test is as follows: …. The following hypotheticals either were or could have been posed in the oral argument. Each of them is followed by four explanations why the hypothetical is or is not problematic for Mr. Grutman’s proposed test. For each hypothetical, please check ALL of the explanations that are plausible. [First hypothetical:] “ what if the plaintiff was totally unknown in the jurisdiction before the magazine was circulated?” • The hypothetical is problematic for Mr. Grutman’s proposed test. The decision rule applies by its terms, but arguably the publisher should not be subject to personal jurisdiction in the state under those circumstances. • The hypothetical is not problematic for Mr. Grutman’s proposed test. … • … • … [Second hypothetical:] … Result: In near- and far-transfer problem questions of this type, LOW and MED students in experimental group scored higher than those in the control group, and the difference was significant (t(1,17)=2.73, d=1.00, p<.05, 1-sided). Evaluating Legal Argument Instruction with Graphical Representations using LARGO

22. LARGO advice usage • LARGO feedback is on-demand • Student requested help 10.1 times per transcript (1 hr). • LOW 12.3 • MED 6.2 • HIGH 17.9 • Advice usage did not decrease over time: • 7.3 and 9.8 in the first 2 transcripts • 12.2 and 8.6 in the last 2 transcripts • In 75% of requests, students selected one of the three short hints and read detailed feedback • Conclusion: Feedback was appreciated Evaluating Legal Argument Instruction with Graphical Representations using LARGO

23. Conclusion & Outlook • LARGO: an ITS for the ill-defined domain of legal argumentation. Uses examples of expert Socratic reasoning to “ask good questions” about student-created argument diagrams. • Experiment showed LARGO did not lead to better learning across the whole sample. But LARGO helps lower ability students learn argumentation skills and understanding the role of hypotheticals. • Ongoing and future work: • Predictive value of argument graphs • Extend LARGO so students make arguments • Collaborative approach to argument construction Evaluating Legal Argument Instruction with Graphical Representations using LARGO

24. Please visit our project website:http://www.cs.cmu.edu/~hypoform • Email:niels.pinkwart@tu-clausthal.de Evaluating Legal Argument Instruction with Graphical Representations using LARGO