Answering List Questions using Co-occurrence and Clustering

1. AnsweringList Questions using Co-occurrence and Clustering Majid Razmara and Leila Kosseim Concordia University m_razma@cs.concordia.ca

2. Introduction • Question Answering • TREC QA track • Question Series • Corpora Target: American Girl dolls • FACTOID: In what year were American Girl dolls first introduced? • LIST: Name the historical dolls. • LIST: Which American Girl dolls have had TV movies made about them? • FACTOID: How much does an American Girl doll cost? • FACTOID: How many American Girl dolls have been sold? • FACTOID: What is the name of the American Girl store in New York? • FACTOID: What corporation owns the American Girl company? • OTHER: Other

3. Hypothesis • Answer Instances • Have the same semantic entity class • Co-occur within sentences, or • Occur in different sentences sharing similar context • Based on Distributional Hypothesis: “Words occurring in the same contexts tend to have similar meanings” [Harris, 1954].

4. Target 232: "Dulles Airport“ Question 232.6: "Which airlines use Dulles” Ltw_Eng_20050712.0032 (AQUAINT-2) United, which operates a hub at Dulles, has six luggage screening machines in its basement and several upstairs in the ticket counter area. Delta, Northwest, American, British Airways and KLM share four screening machines in the basement. Ltw_Eng_20060102.0106 (AQUAINT-2) Independence said its last flight Thursday will leave White Plains, N.Y., bound for DullesAirport. Flyi suffered from rising jet fuel costs and the aggressive response of competitors, led by United and US Airways. New York Times (Web) Continental Airlines sued United Airlines and the committee that oversees operations at Washington Dulles International Airport yesterday, contending that recently installed baggage-sizing templates inhibited competition. Wikipedia (Web) At its peak of 600 flights daily, Independence, combined with service from JetBlue and AirTran, briefly made Dulles the largest low-cost hub in the United States. 4

5. Our Approach • Create an initial candidate list • Answer Type Recognition • Document Retrieval • Candidate Answer Extraction • It may also be imported from an external source (e.g. Factoid QA) • Extract co-occurrence information • Cluster candidates based on their co-occurrence

6. Answer Type Recognition • 9 Types: • Person, Country, Organization, Job, Movie, Nationality, City, State, and Other • Lexical Patterns • ^ (Name | List | What | Which) (persons | people | men | women | players | contestants | artists | opponents | students) PERSON • ^ (Name | List | What | Which) (countries | nations) COUNTRY • Syntagmatic Patterns for Other types • ^ (WDT | WP | VB | NN) (DT | JJ)* (NNS | NNP | NN | JJ | )* (NNS | NNP | NN | NNPS)(VBN | VBD | VBZ | WP | $) • ^ (WDT | WP | VB | NN) (VBD | VBP) (DT | JJ | JJR | PRP$ | IN)* (NNS | NNP | NN | )* (NNS | NNP | NN) • Type Resolution

7. Type Resolution • Resolves the answer subtype to one of the main types • List previous conductors of the Boston Pops. • Type: OTHER Sub Type: Conductor  PERSON • WordNet's Hypernym Hierarchy

8. Source Domain Document Retrieval • Document Collection • Source Document Collection • Few documents • To extract candidates • Domain Document Collection • Many documents • To extract co-occurrence information • Query Generation • Google Query on Web • Lucene Query on Corpora

9. numHits(“SubType Term” OR “Term SubType”) numHits(“Term”) Candidate Answer Extraction • Term Extraction • Extract all terms that conform to the expected answer type • Person, Organization, Job • Intersection of several NE taggers: LingPipe, Stanford tagger & Gate NE • To get a better precision • Country, State, City, Nationality • Gazetteer • To get a better precision • Movie, Other • Capitalized and quoted terms • Verification of Movie • Verification of Other numHits(GoogleQuery intitle:Term site:www.imdb.com)

10. Co-occurrence Information Extraction • Domain Collection Documents are split into sentences • Each sentence is checked as to whether it contains candidate answers

11. Hierarchical Agglomerative Clustering • Steps: • Put each candidate term ti in a separate cluster Ci • Compute the similarity between each pair of clusters • Average Linkage • Merge two clusters with highest inter-cluster similarity • Update all relations between this new cluster and other clusters • Go to step 3 until • There are only N clusters, or • The similarity is less than a threshold

12. The Similarity Measure • Similarity between each pair of candidates • Based on co-occurrence within sentences • Using chi-square (2) • Shortcoming

13. Pinpointing the Right Cluster • Question and target keywords are used as “spies” • Spies are: • Inserted into the list of candidate answers • Are treated as candidate answers, hence • their similarity to one another and similarity to candidate answers are computed • they are clustered along with candidate answers • The cluster with the most number of spies is returned • The spies are removed • Other approaches

14. Cluster-2 Cluster-9 spain, bangladesh, japan, germany, haiti, nepal, china, sweden, iran, mexico, vietnam, belgium, lebanon, iraq, russia, turkey Target269:Pakistan earthquakes of October 2005 Question 269.2:What countries were affected by this earthquake? Cluster-31 oman pakistan, 2005, afghanistan, octob, u.s, india, affect, earthquak pakistan, 2005, afghanistan, octob, u.s, india, affect, earthquak pakistan, 2005, afghanistan, octob, u.s, india, affect, earthquak Recall = 2/3 Precision = 2/3 F-score = 2/3 14

15. Results in TREC 2007 F=14.5

16. Evaluation of Clustering • Baseline • List of candidate answers prior to clustering • Our Approach • List of candidate answers filtered by the clustering • Theoretical Maximum • The best possible output of clustering based on the initial list

17. Evaluation of each Question Type

18. Future Work • Developing a module that verifies whether each candidate is a member of the answer type • In case of Movie and Other types • Using co-occurrence at the paragraph level rather than the sentence level • Anaphora Resolution can be used • Another method for similarity measure • 2 does not work well with sparse data • for example, using Yates correction for continuity (Yates’ 2) • Using different clustering approaches • Using different similarity measures • Mutual Information

19. Questions?