1 / 11

AQUAINT Dialogue Experiment

AQUAINT Dialogue Experiment. Jean Scholtz Information Access Division National Institute of Standards and Technology jean.scholtz@nist.gov. Purpose of the Experiment. To investigate “dialogue” between a system and an analyst to learn what types of dialogue analysts expect to engage in

eamon
Download Presentation

AQUAINT Dialogue Experiment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AQUAINT Dialogue Experiment Jean Scholtz Information Access Division National Institute of Standards and Technology jean.scholtz@nist.gov AQUAINT PI meeting Dec. 3-6, 2002

  2. Purpose of the Experiment • To investigate “dialogue” between a system and an analyst • to learn what types of dialogue analysts expect to engage in • to learn how analysts react to different types of system responses AQUAINT PI meeting Dec. 3-6, 2002

  3. Wizard of Oz Experiment • Used a web-based text chat to eliminate any possible confounds with usability of user interfaces • Experiment design • 5 systems participated • 2 analysts used each system • 10 scenarios were used; order was randomized for each system • analysts were given 15 minutes to explore each scenario • Data collection • logs of dialogues • rating questionnaires filled out by the analyst after each scenario • observation notes AQUAINT PI meeting Dec. 3-6, 2002

  4. Results- Systems • Overall the systems were rated reasonably high by the analysts AQUAINT PI meeting Dec. 3-6, 2002

  5. Results - Systems • Analysts also judged most scenarios as “successful” AQUAINT PI meeting Dec. 3-6, 2002

  6. Results - Analysts • Initial queries and dialogues were extremely varied. • Most initial queries were phrased as questions but they also used statements such as • “I need”; “please provide information on”; “looking for background information on….” • Analysts at times provided context in the initial query • Analyst: "Subject is effect of pollution on black sea fishing industry, What are sources of pollution, trends in reducing pollution, and international cooperation in reducing pollution?” • Analysts did not always take turns. They asked questions as they occurred to them. • Analysts posed multipart questions. If the system did not understand, they broke these down into separate parts. • Analysts posed general questions. If the system did not understand, they asked more specific questions. AQUAINT PI meeting Dec. 3-6, 2002

  7. Results - Analysts • Analysts expect the system to remember context • Example: • analyst, “good info. pls describe the how question." • Example: • Wizard, "the answer is 90 billion dollars" • analyst ,"The same for 2000, please." • Example: • Wizard,"I have no further information for the year 1998." • analyst,"OK on your anwser for 1998 can you do the same for 2000 and skip 1999" AQUAINT PI meeting Dec. 3-6, 2002

  8. Results - Analysts • Self-clarification – Analysts interrupt to clarify their own questions or even to cancel them. • Example: • analyst, “iwould should read i would" • Example: • analyst,"Can you tell me which of these was the first joint venture? • Analyst,"forget that last question" • Analysts don’t just say no, or yes for that matter. When asked questions by the system that can be answered as yes or no, analysts often add constraints. • Example: • Wizard,"Would you like more figures?" • analyst, "Keep going but would like timeline on figures if possible." • analyst, “also do u have any projections?" • Example: • wizard,"Thank you. Are you interested in seeing information about The International Waters Assessment Team as it is related to Black Sea pollution?" • analyst, “Yes particularly about fishing if possible.” AQUAINT PI meeting Dec. 3-6, 2002

  9. Results- Analysts • Responding to system questions is not always straight forward • Example: • Wizard, “We still have more, would you like to see?" • analyst,"can you give me info on when they were formed and the year they were formed” • Example: • Wizard, "There is information about inflation rate, GDP, exchange rate, and per capita income. Which indicator would you like?" • analyst, “i would like to see all indicators” • Example: • analyst, “give me allinformation concerning the economic situation in indonesia since the the year 1998" • Wizard,"I did not understand the word "allinformation". Please try using a different word" • analyst, “actually two words namely all information" • Wizard,"I did not understand the word "namely". Please try using a different word. " • analyst,” i would like all available information you have concerning the economic situation in indonesia" AQUAINT PI meeting Dec. 3-6, 2002

  10. Results- Analysts • Analysts have high expectations – ask about content • Example • Wizard, "The cultivated opium production for 2001 is 7606. • Analyst, “Is this metric tons?" • Wizard,"The cultivated opium production for 2001 is 7606 hectares • Example: • wizard, “However, both the da Vinci and ZEUS systems must receive governmental approval for each procedure that a surgeon plans to use it for.'" • wizard, “On July 11, 2000, the U.S. Food and Drug Administration (FDA) approved the da Vinci Surgical System, making it the first robotic system allowed to be used in American operating rooms.'" • analyst, “When you say governmental approval do you mean federal or something else?" • Wizard," i think this refers to US federal approval" • analyst, "for each surgery?" AQUAINT PI meeting Dec. 3-6, 2002

  11. Next Steps • Issues to be addressed for next experiment • Scenarios • need longer, more involved scenarios • Database • need a common database • User Interface • need to move to system supplied user interfaces • Metrics • handling of different classifications of dialogues/responses AQUAINT PI meeting Dec. 3-6, 2002

More Related