Understanding Cue Reliability in Language Using Naïve Bayes and the Competition Model

1. Naïve Bayes, Cue Reliability, and the Competition Model Roman Taraban Texas Tech University

2. Overview • I. Cue Reliability in Language and the Competition Model • II. Text Analysis • Background • III. Supervised Classification • IBM Watson vs Naïve Bayes • Cue Reliability and Naïve Bayes • Neural Networks • IV. Potential Applications to Other Analyses

3. Cue Reliability in the Competition Model

4. Competition Model Metrics • In our 1989 paper (Taraban, McDonald, MacWhinney), we suggested that the key to accounting for the extraction of gender/case/number paradigms in language acquisition depended critically on cue validity. • From the earlier work of MacWhinney et al, the cue measures were defined as: • Availability: the percentage of times a cue is available in the training input • Reliability: how often a cue contributes to linguistic category, based on the overall frequency of a cue • Validity: product of Availability and Reliability • Conflict Validity: how often a cue is present and correct in cases of conflicting cues

5. Competition Model Metrics Availability = P (cue) f (category  cue)Reliability = P(category | cue) = ------------------------ = Bayesian probability f (cue) Validity = Availability * Reliability = f (category  cue) = Likelihood function

7. Text Analysis

8. Language: A Manifestation of Spirit and Thought • Wilhelm von Humboldt (1767-1835) wrote: • “Language is the outward manifestation of the spirit of people: their language is their spirit, and their spirit is their language; it is difficult to imagine any two things more identical.” • Edward Sapir (1884-1939) believed that • “Language and our thought-grooves are inextricably interwoven, [and] are, in a sense, one and the same.”

9. Predictors in Speech/Text From Chung & Pennebaker, 2008: • “In a job or clinical interview, meeting an office mate for the first time, or talking to someone at a party, we usually ask others to tell us about themselves. Directly or indirectly, we elicit people’s descriptions of themselves to construct a coherent sense of them.” _________ Chung, C. K., & Pennebaker, J. W. (2008). Revealing dimensions of thinking in open-ended self-descriptions: An automated meaning extraction method for natural language. Journal of research in personality, 42(1), 96-132.

10. LIWC 2015 (Linguistic Inquiry and Word Count)http://liwc.wpengine.com/ • LIWC uses specialized dictionaries, and frequency counts from those dictionaries, to characterize a person’s cognitions, motivations, attitudes, and emotions. • Tracks 6400 words, word stems, and emoticons, organized into approximately 90 categories, and outputs frequency counts for categories • In the next analyses, we build our own dictionaries, based on Bayesian probabilities

11. LIWC 2015 (Linguistic Inquiry and Word Count)http://liwc.wpengine.com/

12. Applied Context • Our current projects are a multi-disciplinary collaboration between psychological sciences and engineering • Collaborator Bill Marcy teaches undergraduate engineering ethics -- typically has upwards of 150 students in a class each semester • Course: ENGR 2392: Engineering Ethics &Impact on Society • Center for Global Communications • https://www.depts.ttu.edu/globalcommunications/ • Ethical Engineer Website • Students in the course and from participating institutions contribute comments to case studies on the website. • https://EthicalEngineer.ttu.edu

13. Course Description • Course materials and assignments consider several ethics theories • intuitionism • utilitarianism • respect for persons • virtue ethics • and the Professional Engineers Code of Ethics (NSPE). • Course activities require students to analyze and respond to ethical issues in contemporary social settings involving engineering dilemmas.

15. Supervised Classification

16. Research Question and Models Tested Research Question: to what degree can machine tools reliably detect • kinds of ethical thinking Supervised models • IBM Watson, naïve Bayes Neural Nets with Naïve Bayes

17. Test of IBM Watson Natural Language Classifier • Students composed papers describing the ethical implications of engineering technology and ethics (autonomous vehicles; ethical hacking) • Human raters classified each sentence in students’ papers as involving an ethical vs technical statement.

18. Percent Agreement (SD in parentheses) Between Watson-NLC and Human Raters and Between Human Raters, by Test

19. Classification of Ethics Theories • Another test that we conducted trained Watson to identify the ethics theories in students’ compositions. • Possible classifications: General Moralizing, the NSPE code of ethics, or to an ethical theory, either Utilitarianism, Virtue Ethics, or Respect for Persons • Watson agreed with human classifiers at the 70% level.

20. IBM Watson vs Naïve Bayes • Naïve Bayes was trained on the same classifications of Ethics vs Non-Ethics statements as Watson in 60 random ethics-technology papers. • It was then tested on 10 new random papers • ____________ • Taraban, R., Marcy, W. M., LaCour, M. S., Pashley, D., & Keim, K. (2018). Do engineering students learn ethics from an ethics course? Proceedings of the American Society of Engineering Education – Gulf Southwest (ASEE-GSW) Annual Conference, Austin, TX..

22. Case Study Comment Instructions • Submit a Comment • As you read and analyze case studies your reflective comments are invited on some or all of the following. As part of your analysis include information on the stakeholders and how they are impacted both positively and negatively. • What knowledge and skills are needed to implement sophisticated, appropriate and workable solutions to the complex global problems facing the world today? • What interdisciplinary perspectives would help identify innovative and non-obvious solutions? • What insights can you articulate, based your culture and other cultures with which you are familiar, to help understand your worldview and enable greater civic engagement? • What is your position on the right thing(s) to do?

23. petroleum413 April 9, 2019 at 12:27pm In the case study “Which is more important- Environmental Concern or Economic Growth” by Dr. Majumdar, the situation examined is about an area in India known as Trombay economic growth and pollution due to big oil companies. Trombay and the surrounding areas economies began to expand rapidly due to the big oil companies drilling and refineries, but with this expanding company came many negative consequences. The environment and the surrounding communities were greatly affected by the pollution which was being created by the drilling sites and refineries. One way to help prevent these situations from occurring is for engineers and large oil companies to know the most effective drilling and refinement process which minimize negative environmental impact. Secondly, problem solving skills and the ability to communicate respectively to people of other cultures are an essential tool to solving the complex global problems created by big oil companies. Also, knowledge of safe disposal practices is an essential tool to solving the difficulties facing the world today. Third, by engineers having interdisciplinary perspectives such as knowledge about chemistry and economics would assist in detecting innovative, non-obvious solutions to balancing economic growth and the impact on the environment. Fourth, civic engagement is an essential device to understanding the balance between environmental concern and economic growth. In many cultures certain land is considered sacred, holy, or historical significance. In the event that there is holy, sacred, or historical lands is near drilling sites, engineers with knowledge about the locations of these land can enable superior civic engagement. Lastly, the balance between economic growth and environmental concern is an extensive ethical concern. I believe engineers should take precautions to prevent negative environmental impact. A more expensive piece of equipment may affect the company’s profits but will eliminate potential problems in drilling or refinement is worth the expense. Also, I believe countries who do not have strict environmental regulations should not be taken advantage due to less restrictive laws.

24. Data and Parameters • The following analyses are based on • 119 independent comments to a case study on the Ethical Engineer website: Which is more important - Environmental Concern or Economic Growth? Mr. Amit Mathur paid a farmer 10,000 INR to get permission to drill for oil in a farm in Trombay (Maharashtra, India) and promised to remove the black ooze that would damage the crops... • Inputs for Naïve Bayes, simple Bayes, neural networks: • 584 word stems • 1200 training sentences • 434 test sentences

25. Conditional Probabilities and Ranked Predictors • Naïve Bayes calculates a composite function:

26. .CSV File for Training and Testing

27. Confusion Matrix for New Classifications Total Accuracy = .611

28. Markup of Users’ Comments • Use most probable naïve Bayes classification of each sentence (Culture, Interdisciplinary, etc.) to mark up user’s submission. • Show relative distribution of classifications.

30. Markup of Users’ Comments • Calculate simple Bayesian probabilities (cue reliability) for each predictor (stem) for each category (stakeholder, interdisciplinary…) • Rank order the predictors for each category. • Use subset of ranked predictors (e.g. top 20) for each category in order to mark up the text, as one form of feedback to the user and the instructor.

32. Seeding Neural Networks with Bayesian Posteriors (Cue Reliability)

33. Supplementing the Bayesian Analysis • Naive Bayes estimates the probability of a classification (e.g., How likely is this text about stakeholders?) by treating each of the predictors (word stems) as independent predictors. • Naïve Bayes excludes the multivariate aspects of the estimate. • Question: Can neural networks be used to extend Bayesian analyses?

34. Neural Network with Hidden Layer

35. .CSV File for Training and Testing

36. Output Units: Categories Direct Input Output connections with Bayesian P (output | stem) Hidden Units Input Units: Word Stems

37. Single Layer Neural Network with Simple Bayesian Weights From Input to Output and Base Rates on Output Units Versus Starting with Small Random Weights

38. Neural Network with Simple Bayesian Weights From Input to Output, Base Rates on Output Units, and 128 Hidden Units Versus Starting with Small Random Weights

39. Summary • Assumption that written language is composed of concepts (cues) that underpin the meaning of the message. • Naïve Bayes can readily extract cues with 60-80% predictive accuracy – i.e., agreement with human raters. • In principle, one way to extend the predictive capacity of naïve Bayes may be by combining it with neural network methods. • MacWhinney’s cue reliability metric shows predictive validity alone and in conjunction with neural nets.

40. Potential Applications inOther Analyses

41. Cues in Language Acquisition • Slavic languages, like Ukrainian, Russian, Polish, Slovak and Czech use inflectional morphemes to convey syntactic, grammatical, or semantic features. • Nouns and adjectives are inflected for case, gender and number • Verbs are inflected for tense, aspect, mood, person, subject number, and gender • Inflectional morphemes must be coordinated across the construction of a grammatical sentence.

42. Weighted Cues • Analytic method outlined here may provide a means of tracking the emergence of cue coordination in the acquisition of a native or second language. • Use corpora to • classify grammatical and ungrammatical utterances • classify utterances at different ages • then identify lexical and morphological cues with highest reliability.

43. Cues in Academically Productive Discussions • Use students’ classroom work to classify utterances • as descriptive/analytic • productive/unproductive • high/low quality • then identify lexical cues with highest reliability.

44. Markup as Feedback • From an applied prospective, mark up methods may provide a means of providing feedback to language learners, classroom discussants, and instructors regarding cue use at various points in learning.

45. What We Need • we need more data

46. What We Need • we need more data • we need a talkbank • we need a grant • we need a nudge from Brian

47. Related Papers Mcgallian, J., Taraban, R., Marcy, W. M. (2019). Teaching engineering ethics using interactive computer scenarios. Proceedings of the American Society of Engineering Education – Gulf Southwest (ASEE-GSW) Annual Conference, Tyler, TX. Taraban, R., & Bandara, A. (2017). Beyond recursion: Critique of Hauser, Chomsky, and Fitch. East European Journal of Psycholinguistics, 4(2), 58-66. Taraban, R., Koduru, L., LaCour, M., & Marshall, P. (2018). Finding a common ground in human and machine-based text processing. East European Journal of Psycholinguistics, 5(1). Taraban, R., Marcy, W. M. (2018a). Using technology to develop ethical choice in engineering students. Proceedings of the American Society of Engineering Education – Gulf Southwest (ASEE-GSW) Annual Conference, Austin, TX. Taraban, R., Marcy, W. M. (2018b). Tools to assist with collection and analysis of ethical reflections of engineering students. Proceedings of the American Society of Engineering Education (ASEE) Annual Conference, Salt Lake City, UT. Taraban, R., Marcy, W. M., Koduru, L., Schumacher, J., & Iserman, M. (2019). Using machine tools to analyze changes in students’ ethical thinking. Proceedings of the American Society of Engineering Education (ASEE) Annual Conference, Tampa, FL. Taraban, R., Marcy, W. M., LaCour, M. S., Pashley, D., & Keim, K. (2018). Do engineering students learn ethics from an ethics course? Proceedings of the American Society of Engineering Education – Gulf Southwest (ASEE-GSW) Annual Conference, Austin, TX. Taraban,R., Marcy, W. M., LaCour Jr., M. S., & Burgess II, R. A. (2017). Developing machine-assisted analysis of engineering students’ ethics course assignments. Proceedings of the American Society of Engineering Education (ASEE) Annual Conference, Columbus, OH. Taraban, R., McDonald, J., & MacWhinney, B. (1989). Category learning in a connectionist model: Learning to decline the German definite article. In R. Corrigan, F. Eckman, & M. Noonan (Eds.), Linguistic categorization (pp. 163-193). Philadelphia: Benjamins.