Technology Based Assessment of Language and Literacy Patti Price

1. Technology Based Assessment of Language and LiteracyPatti Price  - Intro   - Decoding   - Comprehension - Opportunities/Discussion PPRICE Speech & Language Technology www.pprice.com

2. Intro • Decoding • Comprehension • Discussion Gorillas Primates Humans Humans Stone tools Fire Shelter Spears • Language: ~ 50,000 years ago • Clothing • Burial • Hunting • Art Farming, 10,000 years ago Bronze, 5500 years ago Written language: ~5 – 6,000 years ago

3. Intro • Decoding • Comprehension • Discussion Pre-language: Information access by direct experience Multimedia: The power of direct experience, but not limited by time, place, language, learning style, sensory abilities… Oral language: Information access also from someone in same time and place with same language Written language: Information access not limited by time and place

4. Intro • Decoding • Comprehension • Discussion Written and Spoken Language are Different • These differences are important in learning to read. • How difficult it is to spell (many letter combos to one sound) • How difficult it is to read (many sounds for the same letter) • How closely your dialect matches written conventions “Got to say this for you, you got guts. Guts and no brains. But guts alone don’t mean nothin’. “ “I gotta say this for you-- you have guts. Guts and no brains. But guts alone mean nothing.” I’ve got to say this for you: You have guts. You have guts and no brains, but guts alone mean nothing.

5. Intro • Decoding • Comprehension • Discussion Differences Between Speech and Writing Vocabulary Distinct words Sentence length Sentence structure Grammar Prosody WrittenSpoken ‘things’… ‘stuff’ 20,000+ 2,000+ 10 – 30 words ? Complex shallow Conservative Liberal A . , ! ? ; : ()  Rich

6. Intro • Decoding • Comprehension • Discussion Example ~130 words (1) I’ve been talking – I’ve been multiplying matrices already, but certainly time for me to discuss the rules for matrix multiplication. (2) And the interesting part is the many ways you can do it, and they all give the same answer. (3) So it’s – and they’re all important. (4) So matrix multiplication, and then, uh, come inverses. (5) So we’re – uh, we – mentioned the inverse of a matrix, but there’s – that’s a big deal. (6) Lots to do about inverses and how to find them. (7) Okay, so I’ll begin with how to multiply two matrices. (8) First way, okay, so suppose I have a matrix A multiplying a matrix B and – giving me a result – well, I could call it C. (9) A times B. Okay. (10) Uh, so, l- let me just review the rule for w- for this entry. Transcript from a linear algebra lecture (From Glass, Hazen, Lee and Wang, Analysis and Processing of Lecture Audio Data, 2004) ~45 words We’ve been multiplying matrices. Now let’s discuss the rules for matrix multiplication. Note that there are many ways to do it and get the same answer. One way to multiply two matrices, A and B, to get a new matrix C is the following rule:

7. Intro • Decoding • Comprehension • Discussion Overview of TBALLTechnology Based Assessment of Language & Literacy • Child-friendly testing system • Measure & analyze child reading & pronunciation • 5-8 year olds • Native speakers of English &Spanish Multidisciplinary collaboration: • Engineering • Education • Computer science • Linguistics • Neuroscience UCLA, USC, UCB

8. Intro • Decoding • Comprehension • Discussion TBall Goals and Components • Reliable and objective automatic reading assessments • Comprehensive instructional framework for • Diagnosis • Intervention • Three main components: • Children’s interface • Teacher interface • Assessment module • Decoding (DAM) • Comprehension (BARLA)

9. Intro • Decoding • Comprehension • Discussion Decoding Comprehension Need to be balanced • In alphabet-based languages, hints about pronunciation • If you know the language you are reading, the hints may be enough • If your first language is another language, and/or if you know the LTS rules in another language, there may be problems. • MOSTLY depends on successful decoding • Much more to learn than LTS rules…

10. Intro • Decoding • Comprehension • Discussion Decoding • Necessary (but not sufficient) for reading comprehension • Necessary for reliable and efficient processing of text • Strongly related to speed of initial reading acquisition • A predictor of difficulties in comprehension development • Accounts for much variance in reading ability at all ages

11. Intro • Decoding • Comprehension • Discussion English Letter to Sound RulesExample: pronunciations of ‘o’ Pronunciations of the letter ‘o’ observed in the 100 most frequent words of English 9 OW: go, home, most, no, only, over, so (know, own) 6 UW: into, to, who, do, (too, you) 6 AH: from, of, other, another, (some, come) 6 AO: for, or, on, often, (your, more) 5 OU: (about, hour; down, how, now) 3 UH: (good, goodbye would) 3 -: (people, work, world) 2 AA: got, not 1 W AH: one

12. Intro • Decoding • Comprehension • Discussion Written Language, Complexity

13. Intro • Decoding • Comprehension • Discussion Decoding Assessment • Are the words read correctly? • Decodability: closer to ‘one letter <-> one sound’ is better In sample of 100 most frequent words in grade 1 material • m, j, b, d… pretty straight forward (always /m jh b d/) • Digraphs (2 letters -> 1 sound): -ng, -ck, (wh-)… • 2 letters -> 2 sounds (or): th -> /dh, th/ (thy, thigh) • 1 letter -> 2 sounds (and): x -> /k s/ (fox, box) • G: get, thing, laugh, large • E: because, get, been, they, great, often • O: go, do, from, for, how, good, people, got, one

14. Intro • Decoding • Comprehension • Discussion Decoding Assessment • Some words are more decodable than others • High frequency words tend to be less decodable • High frequency words tend to be ‘sight’ words (not ‘decoded’ but recognized instantly) (this is important) • Decoding skills are also important • If a word is well-known by a child, it may be treated as a sight word and NOT decoded (tyrannosaurus) • Therefore, unless we know which words are sight words for a child, it’s hard to assess decoding skills • Nonsense words must be decoded Decoding Assessment Measure So… DAM!

15. Intro • Decoding • Comprehension • Discussion Decoding Issues: Dialect, Idiolect and Assessment • If the child has trouble with the letter 'r' (common in English), and says 'cow' when shown 'car' is it a misreading? • If the child is from Boston and reads 'Carl' as 'Call' is it a misreading? What if the teacher has never heard anyone from Boston? • if the child sees the word 'ask' and says 'aks'? • if the child is a native speaker of another language, e.g., Spanish, and says 'seat' when presented the word 'sit', is it a misreading? • if the child is already learning to read, e.g., Spanish, and sees the word 'sore' and pronounces it 'sore-ay', is it a misreading? • SOLUTION: • Map the child's system, whatever it is • Augment the dictionary to diagnostic labels.

16. Intro • Decoding • Comprehension • Discussion Dialect Modeling in ASR Dialect is a system The system is not well modeled in ASR When the point is the Contrast system These are being merged Head sections adapted from Gunnar Fant’s Acoustic Theory of Speech Production

18. Intro • Decoding • Comprehension • Discussion Decoding Issues: Dialect, Idiolect and Assessment Grape g r ey p Grape g w ey p <r> Grape g r aa p ey <Hispanic> Bird b er d Bird b uh d <r> Bird b iy r d <Hispanic>… Rocks w aa k s <r>… Horse h ow s <r>… Three th w iy… Run w ah n…

19. Intro • Decoding • Comprehension • Discussion Decoding Results • nug: /n uw g/ - Sp. LTS, phonology, or short/long? • jore, tay, bap: /hhao r ey, t ay, b aa p/ - Sp. LTS • yan, vag, hine: /zhaa n, b aa g, ay n/ - Sp. Phon. • rin, bap, mot: /r iy n, b ey p, m ow t/ - short/long • jore: /zhow r ey/ - mixed Spanish phonology, LTS • jick: /hhih s k/ - Spanish LTS, misapplied English LTS

20. Intro • Decoding • Comprehension • Discussion Comprehension Assessment GRADE 2: …Kim has a dog. The dog's name is "Dot." Dot is a very white dog, and he has a black dot on his leg. Sam, Ben, and Kim like to play with Dot. They play ball with Dot all day long. Ben lost the ball in the mud, and Dot went into the mud to get the ball. Now, Dot is a brown dog from the mud! … Please describe Dot’s physical appearance? What does Dot look like? Like a dog A white dog A white dog with a black dot on his leg White… and a dot … on his leg Well he’s white and he has a brown dot on his leg Like a rat I don’t remember A Dalmatian Brown

21. Motivation/Desiderata ‘Equivalent’ measures for reading or listening (to compare) Separate understanding of probe from understanding of text (probe as picture adds interest and is not text or speech) Diagnose reading error patterns (by selection of alternate items) Easy to automate (compared to most comprehension tests) Items ascend in difficulty (several errors in a row stops the test) Many items (to allow retesting without repeating items) Equal number of plausible/implausible items (to mitigate guessing) PIAT-R (Peabody Individual Achievement Test – Revised) is best match, but… Not automated Reading only They did not respond to our request • Intro • Decoding • Comprehension • Discussion So… BARLA!

22. Intro • Decoding • Comprehension • Discussion Vocabulary Syntax Morphology Accent adjustment Prosody Find words Print adjustment Punctuation Decoding (LTS) In your first language, this might be easier In a new language, this might be easier

23. Intro • Decoding • Comprehension • Discussion BARLA Example Hear OR Read: • (pot and box reversed) • (dog inbox) • (box on boy) • (correct) The pot is on a box. The pot is on a box.

24. Intro • Decoding • Comprehension • Discussion BARLA Example • ‘tire’ (rhymes) • ‘fire’ (correct) • ‘fur’ (vowel differs) • ‘fairy’(shares, f, i, r) Hear or read: Here is a fire. Here is a fire.

25. Intro • Decoding • Comprehension • Discussion Decoding Opportunity • We sampled the space with 25 items • High Frequency Word: tell, nap • Low Frequency Word: fell, zap • Nonsense Word: zell, fap • However, desirable would be • Frequent retesting (perhaps in a game environment) • Diagnostic of which decoding skills need more work • Items need to be generated and coded for diagnosis

26. Intro • Decoding • Comprehension • Discussion An Opportunity! b b User Model Content Game Pedagogy p p d d Levels: Lower case simple to complex Upper case Digraphs… Move up if: 100% correct Move down if: 3 wrong in a row Letter sounds, English: a /ae/ b /b/ c /k/, /s/ …z /z/ By font, audio files Last played Sounds mastered Diagnosis Preferences… Engine

27. Intro • Decoding • Comprehension • Discussion An Opportunity! b b User Model Content Game Pedagogy t p a d Levels: Lower case simple to complex Upper case Digraphs… Move up if: 100% correct Move down if: 3 wrong in a row Last played Sounds mastered Diagnosis Preferences… Vocabulary, English: … bat … Engine

28. Intro • Decoding • Comprehension • Discussion A Comprehension Opportunity! Data collected for sentences analogous to PIAT-R But not analyzed Paradigm ready to hatch more uses For more information about the project: • http://nautilus.icsl.ucla.edu/tball/ For more information about the data: • iseli@ucla.edu

29. Intro • Decoding • Comprehension • Discussion An Opportunity! • Natural language processing and reading augmentation

30. Intro • Decoding • Comprehension • Discussion By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify Vocabulary • Simplify vocabulary • Simplify syntax • Simplify both 1. Select Text 2. Select View 3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. By "adding to people's brains" we mean adding to the way we deal with difficult things, to add to understanding according to our needs, and to figure out solutions to problems.

31. Intro • Decoding • Comprehension • Discussion By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. SimplifySyntax • Simplify vocabulary • Simplify syntax • Simplify both 1. Select Text 2. Select View 3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. By "augmenting human intellect" we mean increasing the capability of a person to approach a complex problem situation. We also mean how a person gains comprehension to suit his particular needs and derives solutions to problems.

32. Intro • Decoding • Comprehension • Discussion By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify Vocab& Syntax • Simplify vocabulary • Simplify syntax • Simplify both 1. Select Text 2. Select View 3. View By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems.

33. Intro • Decoding • Comprehension • Discussion Simplify Vocabulary & Syntax: Result By "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. By adding to what we can do, we mean a mix of several things. Faster and better understanding are examples. We also mean making it possible to get some understanding when before it was too difficult. And, we mean faster solutions, better solutions, and being able to find solutions that didn't seem possible before. By "difficult situations" we mean the problems people have in different kinds of jobs. The jobs can be in the government, running businesses, being scientists, lawyers, or designers. The problems can be there for twenty minutes or twenty years. We do not mean clever tricks that help in only some situations. We mean a way of life. In this new way of life many things are useful. Useful things include hunches, cut-and-try, things we can't touch, and our "feel for a things“. These things are more useful with great ideas, ways to talk about the ideas and ways to write them, intelligent ways of doing things, and powerful technologies to help us. The number of people and what they create are growing fairly quickly. But our problems are getting harder even more quickly. Also, the importance of finding solutions becomes greater because things are changing more quickly than they used to change. Things that before happened only in one place now are spreading to the whole world. This also makes it important to find solutions. If we could make our brains better in this way, any intelligent person would run as fast as possible to get it. To make them want to run, we need to show a way to do it. We also need to show what we might get out of it.

34. Original Vocabu-lary Syntax Both • Intro • Decoding • Comprehension • Discussion Words 239 275 249 315 Sentences 7 7 15 22 Word/sentence 34.1 39.2 16.6 14.3 Characters/ word 5.6 4.7 5.6 4.4 Reading Ease 10.7% 42.3% 28.2% 71.7% Flesch-Kincaid Grade Level 12.0 12.0 12.0 6.8 Summary of Views and Reading Ease

35. Intro • Decoding • Comprehension • Discussion Questions?