1 / 27

Using Artificial Intelligence to Support Peer Review of Writing

Using Artificial Intelligence to Support Peer Review of Writing. Diane Litman Department of Computer Science, Intelligent Systems Program, & Learning Research and Development Center. Context. Speech and Language Processing for Education. Learning Language (reading, writing, speaking).

shelly
Download Presentation

Using Artificial Intelligence to Support Peer Review of Writing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Artificial Intelligence to Support Peer Review of Writing Diane Litman Department of Computer Science, Intelligent Systems Program, & Learning Research and Development Center

  2. Context Speech and Language Processing for Education Learning Language (reading, writing, speaking) Processing Language Using Language (to teach everything else) Readability Tutors Tutorial Dialogue Systems/ Peers Peer Review Questioning & Answering Scoring CSCL Discourse Coding Lecture Retrieval

  3. Outline • SWoRD • Improving Review Quality • Identifying Helpful Reviews • Summary and Current Directions

  4. SWoRD [Cho & Schunn, 2007] • Authors submit papers • Reviewers submit (anonymous) feedback • Authors revise and resubmit papers • Authors provide back-ratings to reviewers regarding feedback helpfulness

  5. Some Weaknesses • Feedback is often not stated in effective ways • Feedback and papers often do not focus on core aspects

  6. Our Approach: Detect and Scaffold • Detect and direct reviewer attention to key feedback features such as solutions • Detect and direct reviewerandauthorattentiontothesis statementsin papers andfeedback Improving Learning from Peer Review with NLP and ITS Techniques (with Ashley, Schunn), LRDC internal grant

  7. Feedback Features and Positive Writing Performance [Nelson & Schunn, 2008] Solutions Summarization Understanding of the Problem Implementation Localization

  8. I. Detecting Key Feedback Features • Natural Language Processing (NLP) to extract attributes from text, e.g. • Regular expressions (e.g. “the section about”) • Domain lexicons (e.g. “federal”, “American”) • Syntax (e.g. demonstrative determiners) • Overlapping lexical windows (quotation identification) • Machine Learning (ML) to predict whether feedback contains localization and solutions, and whether papers contain a thesis statement

  9. Learned Localization Model [Xiong, Litman & Schunn, 2010]

  10. Quantitative Model Evaluation

  11. II. Predicting Feedback Helpfulness • Can expert helpfulness ratings be predicted from text? [Xiong & Litman, 2011a] • Impact of predicting student versus expert helpfulness ratings [Xiong & Litman, 2011b]

  12. Results: Predicting Expert Ratings (average of writing and domain experts) • Techniques used in ranking product review helpfulness can be effectively adapted to peer-reviews (R = .6) • Structural attributes (e.g. review length, number of questions) • Lexical statistics • Meta-data (e.g. paper ratings) • However, the relative utility of such features varies • Peer-review features improve performance (R = .7) • Theory-motivated (e.g. localization) • Abstraction (e.g. lexical categories) better for small corpora

  13. Changing the meaning of “helpfulness” • Helpfulness may be perceived differently by different types of people • Average of two experts (prior experiment) • Writing expert • Content expert • Student peers

  14. Content versus Writing Experts • Writing-expert rating = 5 • Content-expert rating = 2 • Writing-expert rating = 2 • Content-expert rating = 5 Your over all arguements were organized in some order but was unclear due to the lack of thesis in the paper. Inside each arguement, there was no order to the ideas presented, they went back and forth between ideas. There was good support to the arguements but yet some of it didnt not fit your arguement. First off, it seems that you have difficulty writing transitions between paragraphs. It seems that you end your paragraphs with the main idea of each paragraph. That being said, … (omit 173 words) As a final comment, try to continually move your paper, that is, have in your mind a logical flow with every paragraph having a purpose.  Argumentation issue Transition issue

  15. Results: Other Helpfulness Ratings • Generic features are more predictive for student ratings • Lexical features:transition cues, negation, suggestion words • Meta features: paper rating • Theory-supported features are more useful for experts • Both experts: solution • Writing expert: praise • Content expert: critiques, localization

  16. Summary • Artificial Intelligence (NLP and ML) can be used to automatically detect desirable feedback features • localization, solution • feedback and reviewer levels • Techniques used to predict product review helpfulness can be effectively adapted to peer-review • Knowledge of peer-reviews increases performance • Helpfulness type influences feature utility

  17. Current and Future Work • Extrinisic evaluation in SWoRD • Intelligent Scaffolding for Peer Reviews of Writing (with Ashley, Godley, Schunn), IES (recommended for funding) • Extend to reviews of argument diagrams • Teaching Writing and Argumentation with AI-Supported Diagramming and Peer Review (with Ashley, Schunn), NSF • Teacher dashboard • Keeping Instructors Well-informed in Computer-Supported Peer Review (with Ashley, Schunn, Wang), LRDC internal grant

  18. Thank you!Questions?

  19. Peer versus Product Reviews • Helpfulness is directly rated on a scale (rather than a function of binary votes) • Peer reviews frequently refer to the related papers • Helpfulness has a writing-specific semantics • Classroom corpora are typically small

  20. Generic Linguistic Features

  21. Specialized Features

  22. Lexical Categories Extracted from: • Coding Manuals • Decision trees trained with Bag-of-Words

  23. Discussion • Effectiveness of generic features across domains • Same best generic feature combination (STR+UGR+MET) • But…

  24. Results: Specialized Features • Introducing high level features does enhance the model’s performance. • Best model: Spearman correlation of 0.671 and Pearson correlation of 0.665.

  25. Students versus Experts Critique • Student rating = 7 • Expert-average rating = 2 • Student rating = 3 • Expert-average rating = 5 The author also has great logic in this paper. How can we consider the United States a great democracy when everyone is not treated equal. All of the main points were indeed supported in this piece. I thought there were some good opportunities to provide further data to strengthen your argument. For example the statement “These methods of intimidation, and the lack of military force offered by the government to stop the KKK, led to the rescinding of African American democracy.” Maybe here include data about how … (omit 126 words) praise

  26. Sample Result: All Features • Feature selection of all features • Students are more influenced by meta features, demonstrative determiners, number of sentences, and negation words • Experts are more influenced by review length and critiques • Content expert values solutions, domain words, problem localization • Writing expert values praise and summary

More Related