1 / 14

NRRC Summer ‘02 Mini-Workshop Report Reuse

NRRC Summer ‘02 Mini-Workshop Report Reuse. Abraham Ittycheriah, Andy Latto, Marc Light, Nancy McCracken. June 6th, 2002. MITRE. Reuse: what is it?.

nathan-serrano
Download Presentation

NRRC Summer ‘02 Mini-Workshop Report Reuse

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NRRC Summer ‘02Mini-Workshop ReportReuse Abraham Ittycheriah, Andy Latto, Marc Light, Nancy McCracken June 6th, 2002 MITRE

  2. Reuse: what is it? An advanced QA system should be able to accumulate questions, answers, and other auxiliary information. This information could then be “reused” to enable the system to better answer future questions. In this way, a system could duplicate a human’s ability to gain knowledge and proficiency in an area as she or he answers questions. • More than just “same question, different wording” (TREC-9) • What dates are the Sydney Olympics on? vs. • What are the dates of the Summer Olympics in Sydney? • More than just “context processing” (TREC-10 Context) • What type of vessel was the Varyag?, In what facility was it constructed?, In what country is this facility located?

  3. Examples • When did the Brady bill come into effect?(February 1994) • What president passed the bill?(Bill Clinton) • How much calcium should an adult woman get in her diet? • How much Vitamin D do you need to absorb the recommended amount of calcium? • What’s the name of the Nike golf ball Tiger Woods uses? • What golf ball did Tiger Woods use in the US open? • Who did Billy Jean King beat in battle of the sexes? When tennis champion Billie Jean King accepted the half-humorous challenge Bobby Riggs threw down in 1973, ... • How many titles did King win? • Golfer Betsy King or tennis player King ...

  4. What We Proposed To Do • Explicate: find questions and answers such as the previous examples from a small number of domains (e.g., epidemiology, terrorism, nuclear proliferation, etc.) • Categorize: find and classify different types of possible re-use • Describe: write short document discussing the categories, their characteristics, and their distributions • Recommend: discuss possible future work in the form of a full workshop

  5. The Reuse Corpus • Single-user Question-Answer sets (sequences) • Researchers and grad students generated questions • Instructions: research a topic of interest and emulate a QA system by using a web search engine • 221 questions, 29 topics, one sequence each, 8 q’s ave. • Multi-user sets • Web queries from summer of 2000, grepped out 1 Million questions (John Henderson is responsible for corpus) • 2130 questions, 13 topics, 154 on average (large variance) • Annotations • Researchers annotated 100 reuse possibilities during two day workshop at MITRE • Reuse Corpus will be openly available but only a fraction of the multi-user questions will be “derived” so that they can be distributed

  6. Topics • IBM: Challenger Space Craft, Choosing between a macintosh and a pc, Columbine High School shooting, Dave Koresh and Waco, Death Penalty, Ebay business model, End of Apartheid in 1993, Global warming, Human cloning, Martha Stewart Corporation, NAFTA, Pathfinder lands on Mars, Terrorism in the 21st Century, the Brady bill, The collapse of the Soviet Union, The fall of Enron, The Israeli-Palestinian conflict, Timothy McVeigh, Who are the Taliban, World Trade Center - Structural integrity, World Wide Web • Syracuse: heel pain, calcium, ice cream, poetry, Saskatoon, special education, Washington DC • MITRE: all of Syracuse’s + Tiger Woods, Tupac Shakur, The Oscars, the Stanley Cup, the 2000 Summer Olympics, Digital cameras

  7. Reuse Categories

  8. Examples sameQuestions • Which team won the Stanley Cup in 1994? • Who won the Stanley Cup playoffs in 1994? • When does the summer 2000 Olympics start? • When are the Summer 2000 Olympics? • How much calcium should an adult woman get in her diet? • How much Vitamin D do you need to absorb the recommended amount of calcium? • How many awards did “Gone with the Wind” win? • Who starred in gone with the wind? • Who poses the biggest threat to the United States? • What countries are developing weapons of mass destruction? subSuperQuestions embeddedQs questionFacts questionClarifications

  9. More Examples sameDocuments • What is achilles tendonitis? • What are the causes of achilles tendonitis? • How can achilles tendonitis be treated? • What irons does tiger woods use? (Titleist) • What loft one wood does tiger woods use? (Titleist) • What putter does Tiger Woods use? (Titleist) • When did the Brady bill come into effect? (February 1994) • What president passed the bill? (Bill Clinton) InductiveAnswers answerLimits

  10. Even More Examples sameFacts • What medicines cure anthrax? (Cipro,and other antibiotics) • What does Cipro cure? (anthrax) • Who did Billy Jean King beat in battle of the sexes? When tennis champion Billie Jean King accepted the half-humorous challenge Bobby Riggs threw down in 1973, ... • How many titles did King win? • Golfer Betsy King or tennis player King ... • Is Tupac still alive? • How Tupac really died, where can I find a picture of tupacs death, what happened between tupac and suge knight, when did tupac die?, where can I find the newspaper entitled is tupac alive, who killed tupac shackur topicCharacterizations evidences

  11. Please, No More Examples! • How much calcium should an adult woman get in her diet? Adult women need 1000-1500mg per day ... • What are the factors that increase calcium excretion? High protein diet, caffeine, alcohol, sodium, low exercise. Note that the minimum daily requirements seem to assume some of these, so a lifestyle without these factors may require fewer mgs of calcium per day. answerRevisions

  12. Strategies for Implementing Reuse • Keep a Cache of previous questions and answers • Build topic detection and tracking for question sets • Build a classifiers that take the current question and a previous question on the same topic and assigns a reuse categories to the pair (likely the NULL category) • For example, a classifier could be built to identitify sameQuestions • Use the User to help • Assess the reuse capabilities by evaluating QA system with and without them turned on

  13. Recommendations for Full Workshop • Collect and annotate a larger Reuse Corpus • Extend a functioning QA system (maybe even work with 2 systems that differ in approach) • Build QA TDT system • Build question-answer set classifiers for reuse categories • Mine the user for information relevant to identifying reuse categories and for vetting information to be reused • Evaluate the systems with and without the reuse modules

  14. Conclusion • Introduce and demarcate the topic of reuse • Collected Reuse Corpus and started annotating • Developed Categories of reuse • Discussed strategies for implementing reuse • Wrote 18 page report The basis for a solid proposal for next year’s workshop

More Related