AQUAINT Phase II 6-month Workshop Tampa, October 6-8, 2004

1. HITIQA-2Intelligence Analyst’s Assistant in High-Quality, Interactive Question AnsweringResearch Progress Report AQUAINT Phase II 6-month Workshop Tampa, October 6-8, 2004 AQUAINT 6-month Mtg

2. HITIQA Research Team • SUNY Albany: • Prof. Tomek Strzalkowski, PI/PM • Prof. Boris Yamrom, co-PI • Ms. Sharon Small, Research Scientist • Ms. Hilda Hardy, Research Scientist • Mr. Sean Ryan, Research Assistant • Graduate students • Rutgers: • Prof. Paul Kantor, co-PI • Prof. K.B. Ng • Prof. Nina Wacholder • Graduate students AQUAINT 6-month Mtg

3. HITIQA Research Objectives • HITIQA is an Analytical QA System • “Scenario” QA: look for facts and events in context • Not a factoid system – but factoids are complementary • Semantics is central: data-driven, knowledge-based • QA is Dialogue with Information • Analytical task: topic + context (time, recipient, purpose) • Evolving analytical strategy: line of questions & actions • Detect, follow, anticipate, negotiate strategy turns and shifts • HITIQA Approach • Phase I: Create basic end-to-end capabilities + validate • Phase II: Build-up knowledge + sustain productive dialogue • Phase III+: Augment the analytical process through active assistance AQUAINT 6-month Mtg

4. HITIQA Deployment • HITIQA has been deployed at • MITRE: AQUAINT test-bed • RDEC/SAIC: 2 large servers, local/VPN access • Used in tomorrow’s exercise • PNNL: local – Metrics Challenge Workshop • Albany: local & on-line • HITIQA on-line • Currently accessible from unsecured internet locations (Albany) • Also within VPN (SAIC) • Unlimited access over firewalls – in progress AQUAINT 6-month Mtg

5. Tryouts and Evaluations • On-site workshops with USNR and other analysts • Two workshops conducted in Phase 1 with USNR • ARDA Metrics Challenge summer workshop: 2.5 weeks/8 analysts • Future workshops (Spring 2005): SAIC, Albany • On-line evaluations with USNR • Monthly weekend drills for 6 months • Longitudinal studies: extended drill scenarios • Status: Firewall problems, working around these • Formal Evaluations • Dry run based on the results of the Metrics Challenge? • Testing Facilities • PNNL and MITRE installs • SAIC large-scale installation • AFRL Rome AQUAINT 6-month Mtg

6. HITIQA-2 Task Structure Task 8: Tryouts & Evaluations User Contact Task 2: Dialogue Task 1: QA Task 7: Visual Interface Task 3: Knowledge Acquisition Task 4: System Adaptation Task 6: Answer Generation Task 5: Qualities and Aspects internals AQUAINT 6-month Mtg

7. Extended QA Capabilities • Expanded Scenario Support • Persistent memory of interactions • Questions + clarifications and offers • Questions and answers within a scenario • Follow-through questions • Handling of drill-down questions • Handling of variant questions • Composite Answer Space model • Expandable Answer for the entire scenario • New information found anywhere during scenario updates previous answers AQUAINT 6-month Mtg

8. Exploring Answer Space via Dialogue POSSIBLE DISCARD NEAR MISSES, ALTERNATIVE INTERPRETATIONS EXACT QUESTION MATCH Anticipating related information AQUAINT 6-month Mtg

9. Scenario Structure • Scenario = analytical problem • A series of questions asked by analyst • What is the history of the nuclear arms program between Russia and Iraq? • Who has helped financed the nuclear arms program in Iraq? • (Composite) Question • A question posed by analyst + • Any follow-on by either HITIQA or analyst • A: How has al-Qaida conducted its efforts to acquire weapons of mass destruction? • H: We have this information referring to bin Laden but no mention of al-Qaida. Are you interested? AQUAINT 6-month Mtg

10. Scenario-level answer structure Q0: sarin’s impact on community? Q0: What is sarin’s potency? Q1: sarin development? potency • Botulin 100K times more toxic than sarin • Persists for 30 minutes in clothes Q2: nerve agents? Develop(X,sarin) nerve agents AQUAINT 6-month Mtg

11. Composite Question Structure Original question posed by analyst Clarification/offer by HITIQA Visual panel action by analyst Q = Q0 + Q1 + Q2 + …+ + AQUAINT 6-month Mtg

12. Events and Relationships • Events are basic information units in HITIQA • Generic events • Typed events • Domain-grounded events • Represented internally as frames: • Event type: e.g., transfer, attack, … • Attributes: e.g., people, locations, … • Roles: e.g., agent, target, destination, … • Frames are grouped into topics & “swarms” • Attribute & keyword overlap → topical clusters • Shared frame types & roles → event clusters • Effect, affect, sequence, … → event “swarms” AQUAINT 6-month Mtg

13. Event Frames EXTRACT … Iraq possesses a few working centrifuges and the blueprints to build them. Iraq imported centrifuge materials from Nukem of the FRG and from other sources. One decade ago, Iraq imported 27 pounds of weapons-grade uranium from France, for Osirak nuclear research center. In 1981, Israel destroyed the Osirak nuclear reactor. In November 1990, the IAEA inspected Iraq and found all material accounted for. Peter Clausen, director of research at the Union of Concerned Scientists, said scientists are divided on whether one nuclear bomb can be made by Iraq from the 27 pounds of weapons-grade uranium. Marvin Miller, senior nuclear scientist at MIT in the US, said a crude Iraqi nuclear bomb couldn't fit on a missile, but could be carried in a large aircraft. Generic Frame ASSIGN ROLES & SPECIALIZE Typed Frame FRAME TYPE: TRANSFERWMDTransfer TRANSFER TYPE (TOPIC): imported TRANSFER DEST (LOCATION): Iraq TRANSFER SOURCE (LOCATION): France TRANSFER OBJECT(WEAPON): uranium AQUAINT 6-month Mtg

14. Event Clusters and Swarms • Groups of closely related events → clusters • Shared types & roles → event clusters (imports of uranium) • Attributes & text similarity → topical clusters (nerve agents) • Events may be parts of larger topics or stories • One event makes another event likely → swarming links • If missile exports by North Korea is of interest then likely missile developments status in NK may be relevant also. • Provide guidance for dialogue & exploration • May facilitate hypothesis formation (by analyst) AQUAINT 6-month Mtg

15. Multiple Views of Answer Space Topical cluster: e.g. nerve agents Event cluster: e.g., import of sarin GEN DEV TRF DEV TRF target target GEN TRF TRF target Swarming Links target DEV DEV AQUAINT 6-month Mtg

16. HITIQA Dialogue capabilities • System actions: offers, suggestions, clarifications • Offering closely related and/or swarming events: • Imports from Korea? Missile developments there? • Suggesting additional events in a cluster with relevant events • Sarin production in Iran? In Pakistan? • Analyst’s actions affecting answer space • Accept, partially accept, or reject offers • Strategy changes, diversions • Support hypothesis formation • HITIQA can affect analysts actions through: • Explicit suggestions and offers • Indirectly through: direction of dialogue, answer presentation • Effects can be direct/immediate or indirect/delayed • What information to save (direct) • What question to ask next (indirect) AQUAINT 6-month Mtg

17. Example (from a HITIQA drill) Analyst:What is the history of the nuclear arms program between Russia and Iraq? HITIQA:Do you want to see material on cooperation on development of weapons between Iraq and Russia? Analyst:Yes …Who has helped finance the nuclear arms program in Iraq? HITIQA:Would you be interested in information on weapons in Iraq? Analyst:Yes … What type of nuclear weapons does Iraq possess? HITIQA: Would you be interested in information on dollars in Russia relative to your question? Analyst:Yes … What is the finance debt that Iraq owes Russia? AQUAINT 6-month Mtg

18. Knowledge Acquisition Process • Template development – frame mining • Prop-Bank/Verb bank/Time Bank, etc. • Corpus mining of frequently occurring event types • Roles and relationships • Develop frames for selected event types: • Determine entities in the frame & role for each entity • Determine seed rules for frame/role extraction • Classify modal and other relationships • e.g., denials, threats, allegations, … Now AQUAINT 6-month Mtg

19. Knowledge Acquisition Process • Bootstrapping over text corpora • Develop feature set (context elements) • Bootstrapping exploits duality of lexical and pattern space • Expand from seed rules to high recall extraction • Frame acquisition from structured data • Statistical structure-to-text alignment • Use CNS/WMD database • Extract seed rules from aligned corpora AQUAINT 6-month Mtg

20. Preliminary event frames AQUAINT 6-month Mtg

21. Preliminary event frames, cont’d • Modal Attributes: • Polarity: positive, negative, actual, probable, future, … • Manner: say, claim, threaten, allege, advise, refute, … • Source, if known AQUAINT 6-month Mtg

22. Answer generation • Current Frame-based summaries • Frames in the answer space rendered into headlines • Passages sorted by “threads”: target, time, location, etc. • Build more coherence into the answer • Use elements Discourse Structure Theory • Applied to passages from multiple documents • Passages are output to form a more logical flow • Maximize lucidity of the answer • Use dialogue history to structure the answer → folders • Compute rhetorical relations between answer elements • justification, elaboration, evidence, contradiction, etc. • Answer Summaries • Summarize answer passages using XDoX Summarizer AQUAINT 6-month Mtg

23. Answer Organization Approach Semantic relations →→→→→ Rhetorical relations Shared attributes mapped onto between text passages Swarming links →→→→→ Frames Passages AQUAINT 6-month Mtg

24. Answer Structuring Options We also believe that Bin Ladin was seeking to acquire or develop a nuclear device. Al-Qa'ida may be pursuing a radioactive dispersal device what some call a dirty bomb. Frame Type: Transfer Type: acquire Source: Destin: Bin Laden, Al-Qaida Cargo: nuc dev., dirty bomb Specifically, more Israeli military intelligence sources reported that Bin Laden paid over 2 mil pounds sterling to a middle-man in Kazakhstan, who promised to deliver a dirty bomb to Bin Laden within two years. Frame Type: Transfer Type: deliver Source: mid-man in KZ Destin: Bin Laden Cargo: dirty bomb effect In fact, Frame Type: Transfer Type: obtain Source: Destin: Bin Laden Cargo: nuclear weapons The Saudi-owned, London-based Arabic newspaper, Al-Hayat, declared that Bin Laden had obtained nuclear weapons. However, negation Osama bin Laden probably does not have a nuclear weapon, but likely has chemical or biological weapons, Defense Secretary Donald H. Rumsfeld said. Frame Type: ~Capable Type: possess Agent: Bin Laden Instr: nuclear weapons AQUAINT 6-month Mtg

25. Integrated Visual/Language Interface • Visual navigational context for dialogue • Visual representation for event frames, answer spaces, and links between answer spaces. • Multi-level views: scenario, question, frame • Visual interactions integrated with QA process. • Integrated Visual/QA interface • Questions/Answer actions immediately reflected on visual • Folders reflecting user/system dialogue focus • Visual alerts for system updates AQUAINT 6-month Mtg

26. Integrated Visual interface AQUAINT 6-month Mtg

27. Dialogue focus tracking • Animation used to: • Center clusters and folders in focus • Form new folders when system’s offers accepted • Break up and reorganize old folders • Change colors when relevance decisions are made AQUAINT 6-month Mtg

28. Folder manipulation via dialogue HITIQA: Would you be interested in concealment activities in Iraq? AQUAINT 6-month Mtg

29. Visual Interface: frame view AQUAINT 6-month Mtg

30. Representing Content Aspects • Properties of information “orthogonal” to content • Type of topic: e.g., political, scientific, military, … • Type of content: e.g., historical, biographical, • Type of communication: e.g., human characteristics • “Metadata” in answer and dialogue • To help organize the answer space • To extend dialogue beyond content only • Acquired through machine learning • Learn over large framed corpora • When trained insert into HITIQA interface AQUAINT 6-month Mtg

31. CA research progress • Identify reliable textual & linguistic indicators • Lists of words; named entities • Information from frames (events, roles) • Textual information (number of words, vocab. size) • Anaphora resolution • Incorporate into HITIQA • Frames • Interface • Evaluate • Reliability of indicators • Usefulness of selected CAs for analysts • Impact on Hitiqa system –increment in finding useful information and on end-to-end performance AQUAINT 6-month Mtg

32. Some preliminary results • 100 words in List T as indicators • 5 sample sets, each consisting of 240 documents for training and 60 for testing. • Accuracy = correctly classified documents / total documents • Results are extremely good! Better than 90%. Sometimes much better. AQUAINT 6-month Mtg

33. Intelligence Value of Information • What makes high-quality information? • Accuracy, reliability, significance, depth, etc. • Detail level, bias, opinion/viewpoint, objectivity, … • How to recognize high-quality information? • Textual and contextual indicators • We can compute some qualities (e.g. depth, viewpoints) • Individualized quality models for users • while other qualities appear highly personalized • What qualities matter to analysts? • Why some information is better than other? • Possible panel discussion: Thursday 6-7:30pm AQUAINT 6-month Mtg