If I Have a Hammer: Computational Linguistics in a Reading Tutor that Listens

1. If I Have a Hammer:Computational Linguistics in a Reading Tutor that Listens • Jack Mostow • Project LISTEN (www.cs.cmu.edu/~listen) • Carnegie Mellon University • “To a man with a hammer, everything looks like a nail.” –Mark Twain • Funding: National Science Foundation • Keynote at 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain 1

2. If I had a hammer… [Hays & Seeger] • If I had a hammer,I’d hammer in the morningI’d hammer in the evening,All over this land • I’d hammer out danger,I’d hammer out a warning,I’d hammer out love between my brothers and my sisters,All over this land. 2

3. Outline • Project LISTEN’s Reading Tutor • Roles of computational linguistics in the tutor • So… Conclusions 3

4. Project LISTEN’s Reading Tutor (video) 4

5. Project LISTEN’s Reading Tutor (video) • John Rubin (2002). The Sounds of Speech (Show 3). On Reading Rockets (Public Television series commissioned by U.S. Department of Education). Washington, DC: WETA. • Available at www.cs.cmu.edu/~listen. 5

6. Tutoring: Dr. Joseph Beck, mining tutorial data Prof. Albert Corbett, cognitive tutors Prof. Rollanda O’Connor, reading Prof. Kathy Ayres, stories for children Joe Valeri, activities and interventions Becky Kennedy, linguist Listening: Dr. Mosur Ravishankar, recognizer Dr. Evandro Gouvea, acoustic training John Helman, transcriber Programmers: Andrew Cuneo, application Karen Wong, Teacher Tool Field staff: Dr. Roy Taylor Kristin Bagwell Julie Sleasman Grad students: Hao Cen, HCI Cecily Heiner, MCALL Peter Kant, Education Shanna Tellerman, ETC Plus: Advisory board Research partners DePaul UBC U. Toronto Schools Thanks to fellow LISTENers 6

7. Computational linguistics models in an intelligent tutor • Language models predict word sequences for a task. • E.g. expect ‘once upon a time…’ • Domain models describe skills to learn. • E.g. pronounce ‘c’ as /k/. • Production models describe student behavior. • E.g. which mistakes do students make? • Student models estimate a student’s skills. • E.g. which words will a student need help on? • Pedagogical models guide tutorial decisions. • E.g. which types of help work best? • Theme: use data to train models automatically. 7

8. Language model of oral reading [Mostow, Roth, Hauptmann, & Kane AAAI94] once up a PrRepeat PrCorrect once upon PrTruncate PrJump . . . • Problem: which word sequences to expect? • Language model specifies word transition probabilities • Given sentence text (e.g. ‘Once upon a time…’) • Expect correct reading • But allow for deviations • With heuristic probabilities • Result: • Accepted 96% of correctly read words. • Detected about half the serious mistakes. 8

9. Using ASR errors to tune a language model [Banerjee, Mostow, Beck, & Tam ICAAI03] • Training data: 3,421 oral reading utterances • Spoken by 50 children aged 6-10 • Recognized (imperfectly) by speech recognizer • Transcribed by hand • Method: learn to classify language model transitions • Reward good  transitions that match transcript • Penalize bad  transitions that cause recognizer errors • Generalize from features (kid age, text length, word type, …) • Result: reduced tracking error by 24% relative to baseline 9

10. Domain model of pronunciation • Problem: what should students learn? • Data: pronunciation dictionary for children’s text • ‘teach’  /T IY CH/ • Method: align spelling against pronunciation • ‘t’  /T/, ‘ea’  /IY/, ‘ch’  /CH/ • How frequent is each grapheme-phoneme mapping? • ‘t’  /T/ occurred 622 times in 9776 mappings • ‘z’  /S/ occurred once (in ‘quartz’) • How consistently is each grapheme pronounced? • ‘v’  /V/ always • ‘e’  /EH/ (‘bed’), /AH/ (‘the’), /IY/ (‘be’), /IH/ (‘destroy’) • + ‘ea’, ‘eau’, ‘ed’, ‘ee’, ‘ei’, ‘eigh’, ‘eo’, ‘er’, ‘ere’, ‘eu’, … 10

11. Production model of pronunciation [Fogarty, Dabbish, Steck, & Mostow AIED2001] • Problem: Which mistakes to expect? • Data: U. Colorado database of oral reading mistakes • ‘bed’  /B IY D/ • Method: train G  P  P’ malrules for decoding • ‘e’ /EH/  /IY/ 11

12. Drop ‘s’. Drop ‘s’. Add ‘n’. Add ‘s’. Drop ‘n’. Result: predicted mistakes in unseen test data Context-sensitive rules improved accuracy. Later work: predict real-word mistakes [Mostow, Beck, Winter, Wang, & Tobin ICSLP2002] Top five G  P  P’ decoding errors 12

13. Student model of help requests [Beck, Jia, Sison, & Mostow UM2003] • Problem: when will a student request help on a word? • Data: 7 months of Reading Tutor use by 87 students • Average ~20 hours per student • Transactions logged in detail • Help request rate excluding common words: 0.5%–54% • Method: train classifier using word, student, history • Result: predict words that unseen students click on 13

14. Try to predict subset Grade 1-2 level 1-6 prior encounters Selected data 53 students 175,961 words 29,278 help requests Train predictive model Count help requests 5x Predict other kids’ data 71% accuracy Learning curves for students’ help requests 14

15. Features used • Information about the student • Help request rate, overall reading proficiency, etc. • Information about the word • Word length, position in sentence, etc. • Student’s history with reading word • Percent of times accepted by Reading Tutor, time to read, etc. • Student’s prior help on this word • Was the word helped previously? Earlier today? • How to get all this data?? 15

16. Data collection and translation word features 16

17. Structure of Reading Tutor database Reading Tutor Student Login List readers Session List stories Pick stories Story Encounter Show one sentence at a time Read sentence Sentence Encounter Listens and helps Read each word Word Encounter 17

18. The Reading Tutor beats independent practice… Effect sizes up to 1.3 [Mostow SSSR02, Poulsen 04] …but how? Use embedded experiments to investigate! 2003-2004 database: 9 schools > 200 computers > 50,000 sessions > 1.5M tutor responses > 10M words recognized Embedded experiments Randomized trials Project LISTEN’s Reading Tutor: A rich source of experimental data 18

19. Pedagogical model of help on decoding [Mostow, Beck, & Heiner SSSR2004] • Problem: Which types of help work best? • Data: 270 students’ assisted reading in the Reading Tutor • Method: randomize choice of help and analyze its effects • Result: detected significant differences in effectiveness 19

20. Within-subject experiment design:270 students, 180,909 randomized trials Student isreading a story Student needs help on a word Tutor chooses what help to give Student continues reading Time passes… Student sees word in a later sentence • (How) does the type of help affect the next encounter? ‘People sit down and …’ Student clicks ‘read.’ Randomized choice among feasible types ‘… read a book.’ ‘I love to read stories.’ Outcome: success = ASR accepts word as read fluently 20

21. Whole word: 24,841 Say In Context 56,791 Say Word Decomposition: 6,280 Syllabify 14,223 Onset Rime 19,677 Sound Out 22,933 One Grapheme Analogy: 13,165 Rhymes With 13,671 Starts Like Semantic: 14,685 Recue 2,285 Show Picture 488 Sound Effect Which types stood out? Best: Rhymes With 69.2% ± 0.4% Worst: Recue 55.6% ± 0.4% 180,909word hints(average success rate 66.1%) Example: ‘People sit down and read a book.’ 21

22. Compare within level to control for word difficulty. Supplying the word helped best in the short term… But rhyming hints had longer lasting benefits. What helped which words best? 22

23. So…. what can your computational linguistics model in an intelligent tutor? • What problem is important to solve? • Language models predict word sequences for a task. • Domain models describe skills to learn. • Production models describe student behavior. • Student models estimate a student’s skills. • Pedagogical models guide tutorial decisions. • … • What data is available to train on? • What method is suitable to apply? • What result is appropriate to evaluate? 23

24. …Well I got a hammer • Well I got a hammer,And I got a bell,And I got a song to sing, all over this land. • It’s the hammer of Justice,It’s the bell of Freedom,It’s the song about Love between my brothers and my sisters,All over this land. 24

25. Muchas gracias Molto grazie Obrigado Merci beaucoup Danke schön Dank U well Spaseeba Blagodaria Tak Todah rabah Shukra Efcharisto Xeh-xeh Arigato gozaymas Kop-kun krap Thank you! Questions? Conclusions…See papers & videos at www.cs.cmu.edu/~listen. Thanks 25