1 / 10

Ferret in the ARDA Challenge Metrics Workshop

Ferret in the ARDA Challenge Metrics Workshop. Language Computer Corporation Sanda Harabagiu, PI John Lehmann, John Williams, Finley Lacatusu, Andrew Hickl, Robert Hawes, Paul Aarseth, Luke Nezda, Jeremy Bensley,Patrick Wang, Seth Cleveland and Ricardo Ruiz. Thanks. NIST: Emile Morse

jerome
Download Presentation

Ferret in the ARDA Challenge Metrics Workshop

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ferret in the ARDA Challenge Metrics Workshop Language Computer Corporation Sanda Harabagiu, PI John Lehmann, John Williams, Finley Lacatusu, Andrew Hickl, Robert Hawes, Paul Aarseth, Luke Nezda, Jeremy Bensley,Patrick Wang, Seth Cleveland and Ricardo Ruiz

  2. Thanks • NIST: Emile Morse • PNNL: Antonio Sanfilipo • ARDA: Lynn Franklin • AQUAINT: The ARDA Challenge Metrics Workshop • Participants: the interest and patience.

  3. Thoughts • The ARDA Challenge Metrics Workshop was a wonderful experience: • We benefited from: • Evaluations with analysts • Massive data to analyze • Learn better how our system works and how it could be enhanced • It was a challenge: • Technically: • It had to work for many hours w/o crashing/freezing/memory leaks/communications • It had to produce good answers • It had to be FAST

  4. QA systems • There are two different QA systems developed at LCC: • KIWI (uses justifications based on theorem proving) • PALENTIR (takes advantages of the IE technology for identifying and extracting answers) • Represents: • Topics • Contexts • Indexes the collection based on: • Semantic Information (100+ name classes) • Keywords • Significant relations (in progress)

  5. The premise • We do not evaluate the systems • We look for measures of evaluating systems • The interest: • By implementing a new architecture, one learns a lot • It is very different from doing research; • Participate in TREC;

  6. Palentir • The architecture had to be changed continually for an extensive amount of time • Lots of decisions had to be made to optimize its functionality • E.g.: • Number of answers that are ranked and returned • Question interpretation had to be enhanced • Keyword selection • Context management

  7. Ferret • A proof of concept of the adoption of QUABs • The interface: • Google-like feel • No component was “off-the-shelf” • We welcome feedback • There was a development time issue: • Only 1 month to prepare the system and present it to analysts • Many problems to be addressed, researched, implemented, tested

More Related