Scenario Exercise Observations Scenario #1: Multiple Perspectives

1. Scenario Exercise ObservationsScenario #1: Multiple Perspectives Dr. John D. Prange AQUAINT Program Manager jprange@nsa.gov 301-688-7092 http://www.ic-arda.org

2. Scenario #1 – Group 1Point Loma 1

3. Scenario #1 – Group 2Point Loma 2

4. Scenario #1 – Group 3Point Loma 1

5. SCENARIO #1: Multiple Perspectives/Viewpoints The Office of the Secretary of State has requested information on a variety of issues concerning the Israeli Roadmap. The Secretary of State will be briefing allies on the current status of the Roadmap in 3 days. You have been asked to prepare input to your office’s senior analyst who will be pre-briefing the Secretary’s staff late in the day, day after tomorrow. In particular, your pre-briefing input should summarize the Roadmap and the opposing views and positions on it internationally. You have general knowledge about Roadmap but your background has not been focused on the Israeli/Palestinian conflict so much as on other parts of the Middle East. You have been asked to include in your pre-brief the evidence and arguments that are used by each side of the debate; the names of well-known advocates of each side; the likelihood that terrorist groups will undermine a peace process; and the impact of the US role in the process

6. Observations on Analytic Strategy • Analysts felt that they needed more information on the nature of the presentation to Secretary of State’s Staff; exactly who are the “allies” • Up front time spent making contacts with other more experienced / knowledgeable analysts • Analytic Strategy • All Teams: Obtain general background on area as first step • Team 1: Background-> viewpoints > political conditions • Sources: listed which sources would be looked at first – example: may want to look initially only at specific newspapers and specific bylines within those newspapers • Came up later during clarification on question: wanted specific not originally specified

7. Observations on Analytic Strategy • Analytic Strategy (Continued) • Team 2: Background-> terrorist groups-> reaction from European Allies • Also wanted specified sources, in particular wanted official docs, not news • Team 3: Background -> who responsible for bombings-> what did each group expect • Initially wanted to see full documents as lay of land, then more specific

8. Observations on Questions • General Observations: • Asynchronous response of system to question warped interaction; Choke point of exercise were the Oracles – Just couldn’t respond fast enough • All analysts were able to frame a large number of questions at the start of exercise – One recorder “complained” about “writer’s fatigue; Unfortunately the Oracles were not able to keep up with rapid flow of questions • Researchers felt that most questions posed were in fact suitable for QA systems • Frequently Analyst and Researcher had conversation that resulted in a considerable refinement of the final question(s)

9. Observations on Questions • Team Specific: • After a series of factoids, analyst wanted to pose questions like “what was the result of the XXX summit”. These questions were difficult to map that into a series of subquestions • One Team: Reported disagreements between researcher and oracle over expected capabilities of future systems • One Team: Research attempted to generalize a series of questions concerning “Viewpoints” of different principals into a significant data collection task – collect / process / structure data first and then develop answers to specific questions or provide the final total result to analyst in a manner suitable for browsing • Similar but Alternative Approach: Formulate a set/series of questions as a multiple dimensional matrix, that the system would fill in, instead of responding to each individual questions.

10. Observations on Questions • Team Specific: • Answers did not always come from text; wanted to see maps, photos , etc. • One Team expressed concern about getting too much information for a report due in a day and a half • Summarization across multiple documents would have been effective way to get a set of answers

11. Observations on Answers • Oracles were overwhelmed; simply could not respond fast enough • Should consider include actual current systems in the mix with the Oracle • Analyst’s trust broken in system when it did not return expected known answer; changed strategy of what questions to ask because of concerns of system’s ability; analyst needed the system to have the ability to justify itself (also related to sources) • Interface might not be simple text: leads to clumsy questions

12. General Observations • Researcher surprised that analyst dismissed open-source; strongly preferred to use classified sources • Teams thought that analyst asked questions that would be expected • Several observers remarked that they saw differences between the “Analyst’s” strategy / task decomposition and their own but that eventually most questions were asked. Turned out to be more of an order in which the questions were asked. • Analyst disappointed that some questions that seemed very natural to him were turned into much more stilted formulation by researcher

13. General Observations • System that would return precisely answer was more a hindrance in early stages when more general, extensive background info was wanted • Analyst did ask open-ended questions • Need to have control of system in analyst’s hand since need for justification pervasive • Oracle did bring to analyst’s attention a facet that analyst had previously ignored in an earlier question • Oracle often wanted to go back and add additional information to answers of earlier questions based upon newly found information • Exercise was excellent for getting a view into cognitive process of analyst

14. General Observations • Should we do it again? • How is this different from dialogue evaluation? (if live system is used instead of oracle) • One Response: Level of participation much greater; not pressure of evaluation • Has it given an analyst a better idea of QA systems? • Orient future exercises toward better understand and explore the new / different analytic strategy that Advanced QA is thought to offer. Can we use the exercise to determine the efficiency / effectiveness of this new Scenario-based Advanced QA strategy

15. General Observations • Suggestions for change: • Only one oracle (and a pre-prepared oracle) • Analyst needs to see data • Definitely need to reduce lag time • Got less insight into analyst process since always going back to current questions (had opposing view on this) • Duplicate oracle with real system (also serves purpose to see if current system technology up to task) • Use trained intermediaries for oracles? • (Group 2) got good insight into analyst process during downtime waiting for oracle • One Researcher want write-up / analysis of questions because she was too busy doing role to really understand what was going on • Observed that we were “simulating tomorrow’s system” (Advanced QA) with yesterday’s technology” (Google)

16. Questions / Comments