HITIQA: High-Quality Interactive Question Answering

1. HITIQA: High-Quality Interactive Question Answering 6-Month Review University at Albany, SUNY Rutgers University AQUAINT 6-Month Workshop

2. HITIQA Team • SUNY Albany: • Prof. Tomek Strzalkowski, PI/PM • Prof. Rong Tang • Prof. Boris Yamrom, consultant • Ms. Sharon Small, Research Scientist • Mr. Ting Liu, Graduate Student • Mr. Tom Palen, summer intern • Mr. Peter LaMonica, summer intern/AFRL • Rutgers: • Prof. Paul Kantor, co-PI • Prof. K.B. Ng • Mr. Robert Rittman, Graduate Student • Ms. Ying Sun, Graduate Student • Mr. Vladimir Menkov, Consultant/Programmer • Mr. Peng Song, summer student AQUAINT 6-Month Workshop

3. USER PROFILE; TASK CONTEXT SEARCH & CATEGORIZE Question: What recent disasters occurred in tunnels used for transportation? KB QUESTION NL PROCESSING • Semantics: • What the question • “means”: • to the system • to the user SEMANTIC PROC Clarification Dialogue: S: Are you interested in train accidents, automobile accidents or others? U: Any that involved lost life or a major disruption in communication. Must identify loses. Answer & Justification ANSWER GENER. other auto Vehicle type FUSE & SUMMARIZE TEMPLATE SELECTION location train Focused Information Need QUALITY ASSESSMENT Losses/Cost Possible Category Axes Seen HITIQA System AQUAINT 6-Month Workshop

4. Key Research Issues • Question Semantics • how the system “understands” user requests • Human-Computer Dialogue • how the user and the system negotiate this understanding • Information Quality Metrics • how some information is better than other • Information Fusion • how to assemble the answer that fits user needs. AQUAINT 6-Month Workshop

5. Extracting Question Semantics What are the laws dealing with the quality and processing of food or drugs? ? Answer Cluster food drug food labeling recall FDA life threaten food drug ground terms Alternative Cluster law quality un/assigned attributes drug drug trafficking enforcement FBI AQUAINT 6-Month Workshop

6. Possible Answer Clusters • Each cluster represents: • Complementary answer pieces • With attribute grounding • Specific concept/variable instantiation • Alternative interpretations of analyst’s question • Within a cluster: • Instances of some concepts/variables • Complementary descriptions • Redundancy increases confidence AQUAINT 6-Month Workshop

7. Inducing answer “frames” • Collect references to the topic of interest • high-precision query • cluster to separate nuggets from noise • extract verb patterns, n-grams • Form a ‘naïve’ rule to find more examples • low recall is expected • high precision is desired • identify ‘signature’ features – initial rules • Bootstrap the rules to find new signature patterns in new examples AQUAINT 6-Month Workshop

8. What does the question mean to the user? The speech act The focus User’s task/intention/goal User’s background knowledge What does the question mean to the system? Available information Information that can be retrieved The dimensions of the retrieved information Data-Driven Interaction • Shared Understanding • Semantic gaps drive the dialogue: • to negotiate between user’s meaning and system’s meaning • to fill the gaps in the expected answer • to resolve ambiguities in the data • to reduce dimensionality of the answer space AQUAINT 6-Month Workshop

9. Dialogue Motivators • Dialogue arises from: • System’s need to clarify before proceeding • Analyst’s need to clarify to keep system on target • What is returned from database: • Alternative interpretations: need to select • differentiate candidate answers from others • Off-target interpretations: need to re-target • reformulate the question • Partial answers • follow through linked questions • A dialogue is unique to each analyst-data pair AQUAINT 6-Month Workshop

10. What kind of Dialogue? Good afternoon, how can help? Hello I wanted to notify you of a change of address Yes certainly, can I take your name? Yeh, its Miss Danielle Lansley And your old post code please? SS6 9GD Oh I have a different post code to that What for my old address? Errr well for the address that’s on file Oh what address have you got? What address are you at now? I’m at 1A Willop Call Oh we’ve actually got that address Oh right ohhhh AQUAINT 6-Month Workshop

11. Quality Criteria CONTENT Accuracy and Objectivity Completeness; uniqueness Importance; Verifiability AUTHORITY Reliability; credibility PRESENTATION Clarity and Un-ambiguity Style and Gravitas Orientation and Level Readability and Usability TIMELINESS Recency Currency Measurable Quality Indicators IN/OUT-DEGREE MEASURE Number of cites or links to/from Credibility of these cites/links DOCUMENT SIZE STYLISTIC FEATURES Typical sentence length Use of pronouns, punctuations LINGUISTIC FEATURES Sentence forms, verbs References to names, amounts STRUCTURAL FEATURES Organization of sections Use of section titles, etc. COLLECTION FEATURES Information Quality AQUAINT 6-Month Workshop

12. Information Quality Assessments Initial Framework System for Judgment Experiments Focus Group Studies Pretest Judgment Experiments Quality Metrics AQUAINT 6-Month Workshop

13. Focus Group Studies • Identify quality aspects salient to analysts’ work • Participants: • Journalists, editors (newspaper, TV & Radio) • Faculty of Journalism & Communication • Design: 90 min discussion and/or task oriented • Sessions completed • March 8 (Times Union Albany) • April 9 (SUNY Albany) • Future Sessions: Rutgers, NBC News AQUAINT 6-Month Workshop

14. Quality experiments design • Developed simple, practical GUI • Supports document manipulation (TREC, Web) • Supports quality assessment • Supports gathering material for answer • Subjects perform sessions • Gather material on a given topic • E.g., Laws governing food and drug production • Assess the quality of each retained text • In phase 2, actually compose the answer AQUAINT 6-Month Workshop

15. Quality Assessment GUI AQUAINT 6-Month Workshop

16. Quality Annotation status • Expert Sessions (phase I) • 10 experts, 10 documents each • Approx. 2 hours per session • Student sessions (phase I) • 40 students, 1000 documents • Train & test on expert judged material • Compute student-expert differential AQUAINT 6-Month Workshop

17. Evidence about relevance of a document statistical information analyst judgments links to other documents internal linguistic evidence named entity evidence Evidence about “confidence” of a document source validity evidence grammatical evidence linguistic evidence Search techniques discrete - different methods linear and Support Vector models for specific methods non-linear optimization for “elliptical models” Boolean rule learning Other training data available TREC filtering track Info fusion: what and how? AQUAINT 6-Month Workshop

18. Off-line experimentation scores of items on multiple scales are known value judgments by evaluators (during design) and users (during adaptive usage) are known search the space of fusion formulas to find the version that produces best results for training data. Parameter variation. On-line experimentation users drag related points together system works in background to find which features must have been salient to make these similar user can do a “what if” update of the display Evidence Fusion Experiments AQUAINT 6-Month Workshop

19. Lemur as a Tool – batch fusion • Linux box – datafusion.rutgers.edu • 11 Ranking algorithms • (Rel, x_1, …., x_11) • Pattern-finding, Machine Learning • Can we predict the relevance from x • Rel=f(x) • Use f(x) as a ranking score AQUAINT 6-Month Workshop

20. Recent Relevant Representative Low Focused Fusion (Elliptic) Representative Attribute_2 Relevant Fuzzy Logic (AND) Recent High High Low Attribute_1 Evidence Fusion Displays DISPLAY SPACE ABSTRACT FUSION SPACE The user can select which characteristic is displayed • By observing many users the system learns: • patterns of fusion which are most effective • choices that produce a useful display • preferences of individual users and user types AQUAINT 6-Month Workshop

21. Information Visualization • Supports Evidence Fusion • Dimensional displays • Supports Information Quality Decisions • User interfaces • Supports Clarification Dialogue • Multi-media dialogue: “picture = Kilo-word” • Navigation through information space • Multiple views and orientation AQUAINT 6-Month Workshop

22. Clusters in gravitational space Points represent individual texts Color-coded clusters. Each cluster has different shape and density showing how well formed it is. Verify if the cluster solution makes sense AQUAINT 6-Month Workshop

23. Current Status Summary • Initial HITIQA architecture completed • End-to-end prototype running • Most coded in Java • Data: TREC • A subset of TREC queries converted into “analytical” questions • User studies underway • Focus groups done • Information fusion work underway AQUAINT 6-Month Workshop

24. Plans for the next 6 months • Mapping topical clusters onto Frames • Frames represent “transactions”: events, situations, etc • Build initial Dialogue Manager • Data driven, mixed initiative • Complete Phase I of user experiments • Collect information quality assessments • Interactive Visualization prototype • Start information fusion experiments AQUAINT 6-Month Workshop