Kathy McCoy

1. Kathy McCoy Artificial Intelligence Natural Language Processing Applications for People with Disabilities

2. Primary Research Areas • Natural Language Generation – problem of choice. • Deep Generation --- structure and content of coherent text • Surface Generation – particularly using TAG (multi-lingual generation and machine translation) • Discourse Processing • Second Language Acquisition • Applications for people with disabilities affecting their ability to communicate

3. Projects • Augmentative Communication • Word Prediction and Contextual Information (Keith Trnka) • Using prestored text (Jan Bedrosian+, Linda Hoag+, Tim Walsh) • ICICLE – CALL system for teaching English as a second language to ASL natives (Rashida Davis+) • Text Skimming– for someone who is blind to find an answer to a question (Debbie Yarrington) • Generating Textual Summaries of Graphs– (Sandee Carberry, Seniz Demir, Charlie Greenbacker, Peng Wu) • Summarizing Multi-Modal Documents – (Charlie Greenbacker, Sandee Carberry)

4. Developing Intelligent Communication Aids for People with Disabilities Kathleen F. McCoy Computer and Information Sciences & Center for Applied Science and Engineering in Rehabilitation University of Delaware

5. Augmentative Communication • Intervention that gives non-speaking person an alternative means to communicate User Population • May have severe motor impairments • Unable to speak • Unable to write • Cannot use sign language • Our focus here: adults with no cognitive impairments and very good literacy skills

6. Row-Column Scanning

7. Row-Column Scanning II

8. Can we be faster?

9. Language Representation: Words

10. Still Need to Spell!

11. Predicting Fringe Vocabulary • Word Prediction of Spelled Words (infrequent context-specific words) Methods • Statistical NLP Methods • Learning from the context of the individual • Other Contextual Clues • Geographic Location, Time of Day, Conversational Partner, Topic of Conversation, Style of the Document

12. Prediction Example

15. Trigram Model: P(w|h)=P(w|w-2 w-1)

16. Can we do better?? • Intuitively all possible words do not occur with equal likelyhood during a conversation. • The topic of the conversation affects the words that will occur. • E.g., when talking about baseball: ball, bases, pitcher, bat, triple…. • How often do these same words occur in your algorithms class?

17. Topic Modeling • Goal: Automatically identify the topic of the conversation and increase the probability of related words and decrease probability of unrelated words. • Questions • Topic Representation • Topic Identification • Topic Application • Topic Language Model Use

18. Topic Modeling Approach

19. Topic Identification

20. Topic Identification

21. Topic Application • How do we use those similarity scores? • Essentially weight the contribution of each topic by the amount of similarity that topic has with the current conversation.

23. Results Using Topics

24. Current Work: Graduation for Keith! • Other kinds of tuning to the user we can do: • Recency • Style (part of speech models) • Older work: Does keystroke savings translate into communication rate enhancement?

25. Using the Web to Create Semantic Relations for a Skimming System for Non-Visual Readers Debra Yarrington Dept. of Computer and Information Science University of Delaware {yarringt}@eecis.udel.edu

26. Introduction • This talk will discuss factors in designing a system for automatically skimming text documents in response to a question: • System: • Input: • Question • Potentially complex in nature • Text document • Output: • Web Page with links • Links to text related to the question • Links to text visual skimmers are likely to focus on

27. Goal • The goal of this system is to give nonvisual readers information similar to what visual readers get when skimming through a document in response to a question. Motivation • Working with college students who were blind and visually impaired • Students took significantly longer to find homework question answers within documents than their visual-reading counterparts

28. SubGoals: • Production of our skimming system will require the attainment of three major goals: • Achieving an understanding of what information in a document visual skimmers pay attention to when skimming in response to a question • Developing Natural Language Processing (NLP) techniques to automatically identify areas of text visual readers focus on as determined in 1. • Developing a user interface to be used in conjunction with screen reading software to deliver the visual skimming experience. This talk focuses on work done in 1. and 2.

29. Part 1: Visual Skimming Data Goal: To achieving an understanding of what information visual skimmers pay attention to when skimming through documents to answer questions Procedure: • Have visual readers skim through a document for a question answer while being tracked by an eye tracking system

30. Gathering Data • 14 complex questions and accompanying documents • 10 were 2-pages, 2 were 5-pages, and 2 were 8 pages or longer. • Documents were text documents • No images, few subtitles and lists • Examples of questions used: • “What effect does China’s rising oil prices have on other sectors of its economy?” • “According to Piaget, what techniques do children use to adjust to their environment?” • Individuals skimmed for question answer in a document while being tracked by an eye tracking system. • 43 subjects skimmed for answers to between 6-13 question, • Total of 513 question-answer skimming results • Subjects then answered multiple choice question

31. Results: • 423/510 questions answered correctly • Shows that even for complex questions, subjects were able to successfully answer the question • We wanted to show that the areas subjects paid most attention to when skimming had a connection to the question

32. Eye Tracker Data: Tobii Eye Tracker: • AOIs: • We could define areas of interest (AOI) in the text document ahead of time • We chose paragraphs, titles, subtitles, and the question as separate AOIs. • We then counted the number of gaze points (gazes of over 100 ms duration) in each AOI • HotSpot and Duration File: • The tracker gave us an image that showed “hot spots”, or locations and durations of where the eyes gazed • A file with locations and durations of gaze points

33. Skimming Data Results: • Individuals do focus on titles and subtitles • Subjects frequently focused on the first paragraph or paragraphs of a document • Most subjects did not focus on the first line of each paragraph • This is a technique available via screenreaders • Clearly this does not give screenreader users an experience similar to that of visual skimmers

34. Example of Technique 3:

35. Results Analysis: • We examined AOIs most frequently focused on that did not have physical attributes that would explain the attraction of people’s gazes • Assumption is that these areas were focused on because of their connection to the question.

36. Results Analysis: • Subjects did focus on areas of text containing the answer to the question • Even when answer is not straightforward. • Subjects are not matching words • Shows that subjects are making semantic connections between the question and the information they are skimming for

37. Subjects found question answer • Example: “How do people catch the West Nile Virus?” • The paragraph with the most gaze points for the most subjects was: “In the United States, wild birds, especially crows and jays, are the main reservoir of West Nile virus, but the virus is actually spread by certain species of mosquitoes. Transmission happens when a mosquito bites a bird infected with the West Nile virus and the virus enters the mosquito's bloodstream. It circulates for a few days before settling in the salivary glands. Then the infected mosquito bites an animal or a human and the virus enters the host's bloodstream, where it may cause serious illness. The virus then probably multiplies and moves on to the brain, crossing the blood-brain barrier. Once the virus crosses that barrier and infects the brain or its linings, the brain tissue becomes inflamed and symptoms arise.”

38. Subjects found question answer • Example: “How do people catch the West Nile Virus?” • The paragraph with the most gaze points for the most subjects was: “In the United States, wild birds, especially crows and jays, are the main reservoir of West Nile virus, but the virus is actually spread by certain species of mosquitoes. Transmission happens when a mosquito bites a bird infected with the West Nile virus and the virus enters the mosquito's bloodstream. It circulates for a few days before settling in the salivary glands. Then the infected mosquito bites an animal or a human and the virus enters the host's bloodstream, where it may cause serious illness. The virus then probably multiplies and moves on to the brain, crossing the blood-brain barrier. Once the virus crosses that barrier and infects the brain or its linings, the brain tissue becomes inflamed and symptoms arise.”

39. Subjects focused on areas that have a semantic relationship with the question • E.g., with the question, “Why was Monet’s work criticized by the public?” • the second most frequently focused on paragraph was: In 1874, Manet, Degas, Cezanne, Renoir, Pissarro, Sisley and Monet put together an exhibition, which resulted in a large financial loss for Monet and his friends and marked a return to financial insecurity for Monet. It was only through the help of Manet that Monet was able to remain in Argenteuil. In an attempt to recoup some of his losses, Monet tried to sell some of his paintings at the Hotel Drouot. This, too, was a failure. Despite the financial uncertainty, Monet’s paintings never became morose or even all that sombre. Instead, Monet immersed himself in the task of perfecting a style which still had not been accepted by the world at large. Monet’s compositions from this time were extremely loosely structured, with color applied in strong, distinct strokes as if no reworking of the pigment had been attempted. This technique was calculated to suggest that the artist had indeed captured a spontaneous impression of nature. • This Paragraph does not contain the answer

40. Subjects focused on areas that have a semantic relationship with the question • E.g., with the question, “Why was Monet’s work criticized by the public?” • the second most frequently focused on paragraph was: In 1874, Manet, Degas, Cezanne, Renoir, Pissarro, Sisley and Monet put together an exhibition, which resulted in a large financial loss for Monet and his friends and marked a return to financial insecurity for Monet. It was only through the help of Manet that Monet was able to remain in Argenteuil. In an attempt to recoup some of his losses, Monet tried to sell some of his paintings at the Hotel Drouot. This, too, was a failure. Despite the financial uncertainty, Monet’s paintings never became morose or even all that somber. Instead, Monet immersed himself in the task of perfecting a style which still had not been accepted by the world at large. Monet’s compositions from this time were extremely loosely structured, with color applied in strong, distinct strokes as if no reworking of the pigment had been attempted. This technique was calculated to suggest that the artist had indeed captured a spontaneous impression of nature. • This Paragraph does not contain the answer

41. Part 2: • Next Step: Developing Natural Language Processing (NLP) techniques to automatically identify areas of text visual readers focus on as determined in 1.

42. Process: • Generate keywords from question • Weight keywords based on inverse of # of paragraphs in which they occur in the document • Generate matching score for each paragraph • # of occurrences of each keyword x keyword’s weight • Rank paragraph’s likelihood of being related to the question based on matching score

43. Baseline Keyword Sets: • Set 1: All nonfunction words in question . E.g.,How does marijuana affect the brain • Poor results – poor correlation between question and areas of interest • Set 2: Synonyms, hypernyms and hyponyms of the nonfunction words (generated using WordNet) • Poor results – poor correlation between question and area of interest.

44. Set 3: Topically-Related Keywords • We must explore other ways of identifying text relevant to complex questions • Our solution: • use the World Wide Web to form clusters of topically-related words • Large, covers virtually all topics, constantly updated, constantly available • The topic is the question • The resulting word cluster words will be matched to paragraphs as described above for ranking relevant text.

45. Procedure: Cluster formation • Use content words from question as search engine (Google) query terms • Search returns ordered list of relevant URLs with accompanying snippets (we use top 60 “hits”) • Retrieve web page from URL • Locate snippet within web page (stripped of html) • Include 50 content words before snippet and 50 content words after snippet in cluster • Keep track of total count of each word in cluster

46. Results: • Semantic relationships are being identified • These semantic relationships more accurately identify relevant paragraphs of text

47. Results: • Question: How do people catch the West Nile Virus • Query Terms: how people catch west nile virus • Resulting Cluster:

48. Results: • Ranking of paragraph with answer to question using Web-based semantic relations in 2-page documents:

49. Future Work • Part 1: Analyzing Skimming Data • Look at smaller areas of interest • Users may have focused on one specific part of the paragraph • Look at area users focused on first before choosing to focus on a particular area

50. Future Work • Part 2: Developing clusters • Goal: finding best clustering to identify text identified as relevant by visual skimmers • Include IDF weighting for Web • “Monoamine” vs “trying” • Reordering query terms • How Marijuana Affect Brain vs Marijuana Brain Affect How • Explore varying number of URLs used to form clusters • Explore different window sizes (currently 100 content words) • Explore using phrases as query terms • Explore using synonyms, hypernyms, and hyponyms as query terms