Kathy McCoy (Debbie Yarrington) Dept. of Computer and Information Sciences University of Delaware

1. Replicating Semantic Connections Made by Visual Readers for a Scanning System for Nonvisual Readers Kathy McCoy (Debbie Yarrington) Dept. of Computer and Information Sciences University of Delaware & Consultant for National Institute on Disability and Rehabilitation Research (NIDRR) US Department of Education

2. Goal • The goal of this system is to give nonvisual readers information similar to what visual readers get when skimming through a document in response to a question. Motivation • Working with college students who were blind and visually impaired • Students took significantly longer to find homework question answers within documents than their visual-reading counterparts • Current screenreaders have limited search ability.

3. Approach Use eye-tracker to see what sighted people look at when they are skimming to answer a question Identify important paragraphs Develop Natural Language Processing Techniques to replicate the data Work with people who are blind to develop appropriate interfaces using the results from above.

4. Part 1: Visual Skimming Data Goal: To achieving an understanding of what information visual skimmers pay attention to when skimming through documents to answer questions Procedure: • Have visual readers skim through a document for a question answer while being tracked by an eye tracking system

5. Gathering Data • 14 complex questions and accompanying documents • 10 were 2-pages, 2 were 5-pages, and 2 were 8 pages or longer. • Documents were text documents • No images, few subtitles and lists

6. Example Questions Considered • “What effect does China’s rising oil prices have on other sectors of its economy?” • “According to Piaget, what techniques do children use to adjust to their environment?” • “How do people catch the West Nile Virus?”

7. Gathering Data • Individuals skimmed for question answer in a document while being tracked by an eye tracking system. • 43 subjects skimmed for answers to between 6-13 question, • Total of 513 question-answer skimming results • Subjects then answered multiple choice question

8. Eye Tracker Data: Tobii Eye Tracker: • AOIs: • We could define areas of interest (AOI) in the text document ahead of time • We chose paragraphs, titles, subtitles, and the question as separate AOIs. • We then counted the number of gaze points (gazes of over 100 ms duration) in each AOI • HotSpot and Duration File: • The tracker gave us an image that showed “hot spots”, or locations and durations of where the eyes gazed • A file with locations and durations of gaze points

9. Example of Eye Tracking Results

10. Results Analysis: • We examined AOIs most frequently focused on that did not have physical attributes that would explain the attraction of people’s gazes • Assumption is that these areas were focused on because of their connection to the question.

11. Subjects found question answer • Example: “How do people catch the West Nile Virus?” • The paragraph with the most gaze points for the most subjects was: “In the United States, wild birds, especially crows and jays, are the main reservoir of West Nile virus, but the virus is actually spread by certain species of mosquitoes. Transmissionhappens when a mosquito bites a bird infected with the West Nile virus and the virus enters the mosquito's bloodstream. It circulates for a few days before settling in the salivary glands. Then the infected mosquito bites an animal or a human and the virus enters the host's bloodstream, where it may cause serious illness. The virus then probably multiplies and moves on to the brain, crossing the blood-brain barrier. Once the virus crosses that barrier and infects the brain or its linings, the brain tissue becomes inflamed and symptoms arise.”

12. Subjects focused on areas that have a semantic relationship with the question

13. E.g., with the question, “Why was Monet’s work criticized by the public?” • the second most frequently focused on paragraph was: In 1874, Manet, Degas, Cezanne, Renoir, Pissarro, Sisley and Monet put together an exhibition, which resulted in a large financial loss for Monet and his friends and marked a return to financial insecurity for Monet. It was only through the help of Manet that Monet was able to remain in Argenteuil. In an attempt to recoup some of his losses, Monet tried to sell some of his paintings at the Hotel Drouot. This, too, was afailure. Despite the financial uncertainty, Monet’s paintings never became moroseor even all that somber. Instead, Monet immersed himself in the task of perfecting a style which still hadnot been accepted by the world at large. Monet’s compositions from this time were extremely loosely structured, with color applied in strong, distinct strokes as if no reworking of the pigment had been attempted. This technique was calculated to suggest that the artist had indeed captured a spontaneous impression of nature. • This Paragraph does not contain the answer

14. Part 2: • Next Step: Developing Natural Language Processing (NLP) techniques to automatically identify areas of text visual readers focus on as determined in 1.

15. Process: • Generate keywords from question • Weight keywords based on inverse of # of paragraphs in which they occur in the document • Generate matching score for each paragraph • # of occurrences of each keyword x keyword’s weight • Rank paragraph’s likelihood of being related to the question based on matching score

16. What is it that we match? Keyword Sets: • Directly using the words from the query did not work well; using words similar to the words in the query also did not work. • We needed to find a way to match the “loose semantic connections” found in the eye-tracking data.

17. Topically-Related Keywords • Our solution: • use the World Wide Web to form clusters of topically-related words • Intuition – to find loosely related words, we want to find words that are discussed “with” the words in the question • Use a google search to identify places on the web that the question words are discussed – take words from those areas.

18. Procedure: Cluster formation Use content words from question as search engine (Google) query terms Search returns ordered list of relevant URLs with accompanying snippets Retrieve web page from URL Locate snippet within web page (stripped of html) Include 50 content words before snippet and 50 content words after snippet and call that a snippet phrase

19. Procedure: Cluster formation II Take the top 50 snippet phrases containing the most search terms Generate a word cluster with those phrases Add a global meaning weight so as to eliminate words that are very common (Global Indirect Document Frequency seeded from a large list of words) Take top 25% of cluster – and use it to rank sentences Rank paragraphs by the sentences

20. Results:

21. Example Important Sentences: How do People catch the West Nile Virus? west nile virus it is spread by mosquitoes transmission happens when a mosquito bites a bird infected with the west nile virus and the virus enters the mosquito bloodstream most people infected with the west nile virus have no signs or symptoms most people recover from west nile virus without treatment

22. to help control west nile virus eliminate standing water in your yard about 20 percent of people develop a mild infection called west nile fever some laboratory workers involved in west nile research have contracted the disease from infected animals mosquitoes breed in pools of standing water in rare cases it is possible for west nile virus to spread through other routes including watch for sick or dying birds and report them to your local health department

23. west nile virus is common in areas such as africa west asia and the middle east in the united states wild birds especially crows and jays are the main reservoir of west nile virus but the virus is actually spread by certain species of mosquitoes your best bet for preventing the virus and other mosquito borne illnesses is to avoid exposure to mosquitoes and eliminate mosquito breeding sites your overall risk of contracting west nile virus depends on these factors time of year then the infected mosquito bites an animal or a human and the virus enters the host bloodstream where it may cause serious illness even if you are infected your risk of developing a serious west nile virus related illness is extremely small

24. Results: Discounted Cumulative Gain with EyeTracking Results

25. Current Work on Skimming • Incorporating Physical Attributes in assessment of paragraphs • Developing a user interface in conjunction with potential users • Important that it provide access to information like what a visual reader gets • Read important sentences with indication of paragraph? • Read word clusters with indication of paragraph? • Allow user to stop and start skimming with a keypress

26. Thank You! Questions?