Scalable and Sustainable Technologies for Reading Instruction

1. Scalable and Sustainable Technologies for Reading Instruction An Interagency Education Research Initiative: NSF - IES - NIH Principal Investigators: Walter Kintsch, Donna Caccamise, Ron Cole, Lynn Snyder, & Richard Olson

2. The Push to Improve Literacy • Every child can learn to read, yet • Literacy is a national problem • We have good basic research, problem is • To develop practical implementations • To get them used widely in schools • IERI initiative

3. Colorado Literacy Tutor Tools for Teachers • K-6th grade: • emphasis on foundational reading skills: Interactive books for practice plus tutors for special problems • 6th-14th grade: • emphasis on comprehension and learning from text

4. Underlying Cognitive Science Research • Tutoring is an effective method of instruction: • Human tutors produce gains between .4 and 2.3 standard deviations over classroom teaching • Most human tutors have only modest domain knowledge and rarely use sophisticated tutoring strategies • Existing Artificial Intelligence tutors such as AUTOTUTOR (Graesser et a., 1999) and ATLAS (van Lehn et al., 2000) approach the effectiveness of human tutors (Graesser et al., 2001)

5. Underlying Cognitive Science Research • Creating the Next Generation of Intelligent Animated Conversational Agents: • Use of synthesized speech and speech recognition • Conversation with a talking head • Learners respond in English in addition to click-and-point

6. Underlying Cognitive Science Research • Latent Semantic Analysis: Theory, Technology and Practice • LSA as a map of meaning • High-dimensional semantic space • Text meaning as the sum of the words • Limitations: • Learns only from words • Neglect of order and syntax

7. A map is a 2-dimensional representation of distance and direction data:Data: Map:

8. A semantic space is a k-dimensional representation of word co-occurrence data (k<<<m,n):Data: Semantic Space:

9. High-dimensional spaces are capable of representing complex semantic relations: TREE DOG .70 BARK .06 .66

10. The meaning of a text is the sum of the words in LSA: • The accident victim was flown to the hospital • This sentence is more similar to (1) than (2), although it shares two content words with (2) and none with (1): • (1) A helicopter brought the man hurt in the crash to the emergency room cosine = .29 • (2) An accident occurred in the street where the victim could not get assistance cosine = .24

11. Interactive Books & Reading Tutors Ron Cole, Donna Caccamise, Lynn Snyder and the Colorado Literacy Tutor Team Center for the Study of Spoken Language University Of Colorado

12. Components of the Reading Tutor: • Foundational Skills Tutors • Teach underlying reading skills • Interactive Books • Teach fluent reading & comprehension • Managed Learning Environment • Enroll students, track and display progress, manage individual study plans, etc.

13. Beginning sounds

14. Letter sounds

15. Word reading

16. Silent “e”

17. Interactive Books • Teacher or child selects appropriate book • Child selects animated tutor or “coach” • Animated coach guides child through book • Coach can read words, sentences paragraphs or whole page • Coach asks comprehension, inference and summary questions using speech recognition to determine accuracy of responses • Each level of a book has the capability of invoking specific reading tutors if a child needs more foundational skill training

18. Read to me and read aloud

19. Click on image question interaction

20. Multiple choice question interaction

21. Summarization using Summary StreetTM

22. Comprehension and Learning • Walter Kintsch, Donna Caccamise, Lynn Snyder, and the LSA Research Group Institute of Cognitive Science University of Colorado • Tom Landauer Pearson Knowledge Technologies& University of Colorado

23. Comprehension and Learning • LSA computes how similar in meaning two texts are • The contents of a student’s essay can be compared with other essays and standards • Students receive feedback about the content of their writing - guidance for revising • A tool for self-assessment: feedback allows students to judge how well they are doing and what needs more work

24. Summary Street® • An existing, classroom-tested tested system that provides content-based feedback to middle-school students summarizing a text: • indicates what content is missing • what might be overemphasized • flags apparent problems sentences • helps with the organization of the material

29. The teacher keeps track of how much and how well the student did:

31. Provides hints about how the summary could be shortened: • Sentences are flagged that are very similar in meaning: • …...They also wrote books on paper. The books were made from bark paper that they folded together….. • Sentences that appear unrelated to the topic are questioned: • …..We also learned about the Incas…..

33. Anonymity of the computer It is not threatening to be corrected by a sub machine! The computer is sometimes wrong Students are never told what to do Problems are identified and hints are provided about how to solve them Students always make their own decisions What makes Summary Street effective?Student Interviews:

34. How effective is Summary Street? • David Wade-Stein & Eileen Kintsch (2004) Summary Street: Interactive computer support for writing. Discourse Processes, 22, 333-362. • 6th-graders write summaries, one with Summary Street, one on a word processor that provides only length and spelling feedback

35. Students working with Summary Street spend much more time on task:

36. Summaries written with Summary Street receive better teacher grades:

37. Of the 10 texts used in this study, the biggest improvements were observed for the most difficult texts • The best students did not need Summary Street; for the poorest students, Summary Street did not provide enough support; the middle group profited the most.

38. How effective is Summary Street? • Marita Franzke, Eileen Kintsch, Donna Caccamise, Nina Johnson, & Scott Dooley (submitted) Summary Street®:Computer Support for Comprehension and Writing. • 8th-graders practice summary writing for four weeks, with and without Summary Street; four classrooms, same teacher

39. Teacher grades assigned to summaries written with and without Summary Street over the 4-week study period; texts get progressively more difficult. Summary Street Control

40. Essentially identical results were observed for • Content scores • Inappropriate Details • Organization • Style • No improvements for • Mechanics • Plagiarism was infrequent in both groups and did not increase with practice

41. Performance on a test patterned after the Colorado Student Assessment Program (CSAP): • All students take the test under standard, paper-and-pencil conditions, so this is a test of distant transfer • Students trained with Summary Street outperform students who only used a word processor on questions requiring summarization; p < .05, effect size is .42 • No statistically reliable difference between conditions on other types of test items: • Inference • Vocabulary • Fact Finding • Other

42. Summary Street in Colorado schools: • 7 school districts - urban, suburban, rural • 85 teachers • 77 classrooms • 2292 students • about 140 texts

43. A Valid Test of Reading Comprehension • Current tests are not theoretically motivated • Items have been selected because they discriminate reliably between people • Levels of comprehension are not distinguished • Multiple-choice questions are used to score tests objectively and automatically • A valid test should • Distinguish between deep and shallow comprehension • Employ a free-response format

44. Beth Mulligan, Katherine Rawson, Praful Mangalath, & Walter Kintsch Designs for a comprehension test. • In two experiments 241 college students read 6 expository texts (147 - 322 words), recalled what they had written, and answered an extended inference question • Recall - shallow comprehension • Ability to reproduce the text • Inference - deep comprehension • Inference questions required a paragraph length response. Information from the text as well as general world knowledge had to be combined to obtain the answer • A variety of inference types were used, e.g., • what were the causes of an event described in the text? • what is the general theme of the examples given in the text?

45. Recall and Inference are separate components of comprehension Recall performances sets upper bound for inference performance

46. 21% of the students received a significantly higher grade on the recall test than on the inference test For 58 students who received a memory score of B or better, inference scores ranged from F to B+ A confirmatory factor analysis yielded two significant factors, a memory factor and an inference factor

47. Grading Essays with LSA: Classifying the vectors representing the students’ responses • Support-vector regression segments the semantic space into areas according to the grades human raters assign to essays • New essay is given a grade according to the area of the space it is in

48. LSA grading of protocols: • Memory • Human rater inter-reliability r = .83 • LSA-human correlation r = .80 • Inference • Human rater inter-reliability r = .80 • LSA-human correlation r = .68

49. Tools for Teachers • The success and reputation of many professions is based on the use of powerful tools • Teachers employ few tools, mostly because versatile, powerful tools were not available • Tools to empower teachers - not to substitute or replace them

50. Handouts are at: • http://www.reading.org/association/meetings/annual_handouts.html