1 / 31

Capturing User Intent for Analytical Process

Capturing User Intent for Analytical Process. Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College. Hien Nguyen Jacob Russell Adam Olson University of Wisconsin - Whitewater. Presentation at AITP meeting, February 2 nd 2010. Outline. Problem Challenges

vaughan
Download Presentation

Capturing User Intent for Analytical Process

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Capturing User Intent for Analytical Process Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College Hien Nguyen Jacob Russell Adam Olson University of Wisconsin - Whitewater Presentation at AITP meeting, February 2nd 2010.

  2. Outline • Problem • Challenges • Our User Intent Model • Preliminary evaluation • Future work Presentation at AITP meeting, February 2nd 2010.

  3. Outline • Problem • Challenges • Our User Intent Model • Preliminary evaluation • Future work Presentation at AITP meeting, February 2nd 2010. .

  4. Problem APEXB, Senior Analyst Iran’s desk Expertise: Iran, Iraq, Middle East Nuclear weapon programs APEXL, Advisory Analyst Iran’s desk Expertise: Iraq, iranian political policies APEXF, Junior Analyst Iran’s desk Expertise: assessment of Iranian clerical support. Presentation at AITP meeting, February 2nd 2010.

  5. Problem APEXB, Senior Analyst Does not like to talk to anyone APEXL, Advisory Analyst Rushes through everything APEXF, Junior Analyst Face book, Second life Presentation at AITP meeting, February 2nd 2010.

  6. Problem Modeling an analyst’s intent to improve the effectiveness of collaboration among intelligence analysts • Capture the process instead of topical interests • May help improve the diversity of collaboration Presentation at AITP meeting, February 2nd 2010.

  7. Research Challenges • Partiality and incompleteness of captured knowledge • Dynamic information spaces, user foci, and preferences • Uncertainty in analyst’s actions Presentation at AITP meeting, February 2nd 2010.

  8. Research Questions • What is user intent and how do we capture it • How do we evaluate the effectiveness of the intent model Presentation at AITP meeting, February 2nd 2010.

  9. Outline • Problem • Challenges • Our User Intent Model • Preliminary evaluation • Future work Presentation at AITP meeting, February 2nd 2010.

  10. Our Intent Model • Analyst intent = Goals + Actions + Commitment • Our user model consists of 3 components that are designed to capture intent: • Foci: What is the working space of the analyst and what he is concentrating on? (Goal & Commitment) • Rationale: Whydoes the analyst have these foci? (Goals) • Action: How are the analyst’s goals accomplished? (Actions) Presentation at AITP meeting, February 2nd 2010.

  11. Our Intent Model Search APEXF Retain Event database Foci Rationale Action Presentation at AITP meeting, February 2nd 2010.

  12. Example - Rationale Search:Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program? G0 APEXF 12/11/2007 at 11:00 Ayatollah_ khamenei Iranian_cleric_community Nuclear_ program Related_to Related_to Related_to isa isa Related_to isa Ahmadinejad iranian_ community Cleric_ community Program Iranian … isa isa community Presentation at AITP meeting, February 2nd 2010.

  13. Example - Foci Search:Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program? APEXF 12/11/2007 at 11:00 G0 iranian_cleric_community nuclear_program ayatollah_khamenei … Commitment level: 1.0 Presentation at AITP meeting, February 2nd 2010.

  14. Example - Action Search:Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program? APEXF 12/11/2007 at 11:00 Searching for Evidence Bayesian network HMM .82 .06 .06 2 .01 .17 1 3 .06 Going through Documents Examining Evidence .88 .91 .03 Presentation at AITP meeting, February 2nd 2010.

  15. Example Search:Is iranian clerical community supporting Iran's nuclear program? 12/11/2007 at 11:10 Search:Are there clerics who oppose Iran's nuclear program? 12/11/2007 at 11:05 APEXF Presentation at AITP meeting, February 2nd 2010.

  16. Example - Rationale G3 Iran_nuclear_porogram isa APEXF G0 G1 G2 Ayatollah_ khamenei Iranian_cleric_community Nuclear_ program Related_to Related_to Related_to isa isa Related_to isa Ahmadinejad iranian_ community …. Cleric_ community Program Iranian isa isa community Presentation at AITP meeting, February 2nd 2010.

  17. Example - Foci G3 APEXF … G2 G1 G0 cleric Iran_nuclear_program Iran … Commitment level: 0.66 iranian_cleric_community nuclear_program ayatollah_khamenei … Commitment level: 0.33 cleric Iran_nuclear_program Iran … Commitment level: 0.5 Presentation at AITP meeting, February 2nd 2010.

  18. Example - Action APEXF Searching for Evidence .82 .06 .06 2 .01 .17 1 3 .06 Going through Documents Examining Evidence .91 .03 .88 Bayesian networks HMM Presentation at AITP meeting, February 2nd 2010.

  19. Example - Intent • Goal: • Iranian clerics and nuclear weapons program • Actions: • Searches • Commitment: • F(0.33, 0.5, 0.66) APEXF Presentation at AITP meeting, February 2nd 2010.

  20. Outline • Problem • Motivation • Our User Intent Model • Preliminary evaluation • Future work Presentation at AITP meeting, February 2nd 2010. .

  21. Evaluation • Do we capture user intent more precisely compared to a simple interest list approach? • Do we capture a user’s intent earlier in the analytic process compared to the interest-based approach? Presentation at AITP meeting, February 2nd 2010.

  22. Evaluation • Testbed: APEX 07, 8 intelligence analysts • Tasks: • Where does the Iranian clerical community stand on Ayatollah Khamenei? • President Ahmadinejad'spolicies with regards to Iran’s civilian and military nuclear program? • Database: contain 5613 events, 8 types of actions. Presentation at AITP meeting, February 2nd 2010.

  23. Evaluation Presentation at AITP meeting, February 2nd 2010.

  24. Do we capture user intent more precisely? • Analysts: APEXL- APEXC, APEXE-APEXH APEXL- APEXK, APEXF-APEXB. • Procedure: Create a user model for each user using 25 consecutive events on December 11, 2007. Precision = • Cgoals: a goal node that is found in both intent models. Presentation at AITP meeting, February 2nd 2010.

  25. Do we capture user intent more precisely? P-value=0.0396<0.05 Presentation at AITP meeting, February 2nd 2010.

  26. Whether we capture a user’s intent earlier? • Analysts: APEXF – APEXB • APEXB: nuclear weapon program and Iran. • APEXF: grand Ayatollah Intent (t=5) Interest (t=8) Presentation at AITP meeting, February 2nd 2010. .

  27. Future work • Research and implementation: • Generation of goal descriptions • Integrating between content-based and action-based information • Link between common intent and collaboration effectiveness • Evaluation: • Prediction of action sequences • Define measures of collaboration effectiveness Presentation at AITP meeting, February 2nd 2010.

  28. Acknowledgement • This research is funded by CASE program (IARPA) grant #FA8750-07-2-0219 DOD-AFRL (2007-2008). Thanks to the CASE technical advisory team for making the data available, giving constructive feedback and supporting this research. Presentation at AITP meeting, February 2nd 2010.

  29. Related work • Eugene Santos Jr. and Hien Nguyen. “Modeling Users for Adaptive Information Retrieval by Capturing User Intent”. Book Chapter. In M. Chevalier, C. Julien, C. Souledupuy (Eds.). Collaborative and Social Information Retrieval and Access: Techniques for Improved User Modeling. 2009. • Hien Nguyen and Eugene Santos Jr.(a) “An Evaluation of the Accuracy of Capturing User Intent for Information Retrieval”. In Proceedings of the 2007 International Conference on Artificial Intelligence. Las Vegas, NV. June 2007. • Hien Nguyen and Eugene Santos Jr.(b) “Effects of prior knowledge on the effectiveness of a hybrid user model for information retrieval”. In Proceedings of the Homeland Security and Homeland Defense VI conference. Orlando, FL. March 2007. Vol. 6538. • Hien Nguyen, Eugene Santos Jr., Aaron Schuet, and Nathan Smith. “Hybrid User Model for Information Retrieval”. In Technical Report of Modeling Others from Observations workshop at AAAI-2006 conference. • Hien Nguyen, Eugene Santos Jr., Qunhua Zhao and Hua Wang. “Capturing User Intent for Information Retrieval”. In Proceedings of the 48th Annual Meeting for the Human Factors and Ergonomics Society (HFES-04), New Orleans, LA. 2004. Pages 371-375 Presentation at AITP meeting, February 2nd 2010.

  30. Past undergraduate research project • - Hybrid Approach to Word Sense Disambiguation. Jacob Russell, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI. (nominated by UWW Undergraduate Research Committee to be presented at CUR on the Hill). • Implementation of intent-based user model for business domain. Adam Olson, Nathan Lutterman, LQ McDonald III, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI. • Web-based user model services for analytic process. 2008. Nathan Lutterman, Adam Olson, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Maryland, MD. • Simulations of network defects. 2007. Russell Jacob, Nathan Smith, Nicholas Orcust, Hien Nguyen, Jim Harris, Benette Harris. Poster presented at Undergraduate research day, April 4, 2007 and National Conference for Undergraduate research (NCUR 2007), April 12-14. CA. • Evaluation of Hybrid User Model for Information Retrieval. 2006. Aaron Schuett, Nathan Smith, and Hien Nguyen. Poster presented at National Conference of Undergraduate Research 2006, Raleigh, NC. Presentation at AITP meeting, February 2nd 2010.

  31. Current undergraduate research project • Roulette game – Application of Modeling under Uncertainty. Richard Lilja, Hien Nguyen • Intent model for supporting medical trauma treatment. LQ McDonald, Adam Olson, Hien Nguyen Presentation at AITP meeting, February 2nd 2010.

More Related