290 likes | 298 Views
This text explores the challenges of syntax-driven semantics and presents alternative approaches such as semantic grammars, information retrieval, and information extraction. It covers topics like semantic structures, user input prediction, priming, drawbacks of semantic grammars, and more.
E N D
Robust Semantics, Information Extraction, and Information Retrieval CS 4705
Problems with Syntax-Driven Semantics • Syntactic structures often don’t fit semantic structures very well • Important semantic elements often distributed very differently in trees for sentences that mean ‘the same’ I like soup. Soup is what I like. • Parse trees contain many structural elements not clearly important to making semantic distinctions • Syntax driven semantic representations are sometimes pretty verbose V --> serves
Semantic Grammars • Alternative to modifying syntactic grammars to deal with semantics too • Define grammars specifically in terms of the semantic information we want to extract • Domain specific: Rules correspond directly to entities and activities in the domain I want to go from Boston to Baltimore on Thursday, September 24th • Greeting --> {Hello|Hi|Um…} • TripRequest Need-spec travel-verb from City to City on Date
Predicting User Input • Rely on knowledge of task and (sometimes) constraints on what the user can do • Can handle very sophisticated phenomena I want to go to Boston on Thursday. I want to leave from there on Friday for Baltimore. TripRequest Need-spec travel-verb from City on Date for City Dialogue postulate maps filler for ‘from-city’ to pre-specified to-city
Priming User Input • Users will tend to use the vocabulary they hear from the system: lexicalentrainment (Clark & Brennan ’96) • Reference to objects: the scarey M&M man • Re-use of system prompt vocabulary/syntax: Please tell me where you would like to leave/depart from. Where would you like to leave/depart from? • Explicit training vs. implicit training • Training the user vs. retraining the system
Drawbacks of Semantic Grammars • Lack of generality • A new one for each application • Large cost in development time • Can be very large, depending on how much coverage you want them to have • If users go outside the grammar, things may break disastrously I want to leave from my house at 10 a.m. I want to talk to a person.
Information Retrieval • How related to NLP? • Operates on language (speech or text) • Does it use linguistic information? • Stemming • Bag-of-words approach • Very simple analyses • Does it make use of document formatting? • Headlines, punctuation, captions • Collection: a set of documents • Term: a word or phrase • Query: a set of terms
But…what is a term? • Stop list • Stemming • Homonymy, polysemy, synonymy
Vector Space Model • Simple versions represent documents and queries as feature vectors, one binary feature for each term in collection • Is a term t in this document or in this query or not? D = (t1,t2,…,tn) Q = (t1,t2,…,tn) • Similarity metric:how many terms does a query share with each candidate document? • Weighted terms: term-by-document matrix D = (wt1,wt2,…,wtn) Q = (wt1,wt2,…,wtn)
How do we compare the vectors? • Normalize each term weight by the number of terms in the document: how important is each t in D? • Compute dot product between vectors to see how similar they are • Cosine of angle: 1 = identity; 0 = no common terms • How do we get the weights? • Term frequency (tf): how often does i occur inDoc j? • Inverse document frequency (idf): # docs/ # docs term i occurs in • tf . idf weighting: weight of term i for doc j is product of frequency of i in j with log of idf in collection
Evaluating IR Performance • Precision: #relevant docs returned/total #docs returned -- how often are you right when you say this document is relevant? • Recall: #relevant docs returned/#relevant docs in collection -- how many of the relevant documents do you find? • F-measure combines P and R • Are P and R equally important?
Improving Queries • Relevance feedback: users rate retrieved docs • Query expansion: many techniques • add top N docs retrieved to query and resubmit expanded query • WordNet • Term clustering: cluster rows of terms in term-by-document matrix to produce synonyms and add to query
IR Tasks • Ad hoc retrieval: ‘normal’ IR • Routing/categorization: assign new doc to one of predefined set of categories • Clustering: divide a collection into N clusters • Segmentation: segment text into coherent chunks • Summarization: compress a text by extracting summary items or eliminating less relevant items • Question-answering: find a span of text (within some window) containing the answer to a question
Information Extraction • Another ‘robust’ alternative • Idea: ‘extract’ particular types of information from arbitrary text or transcribed speech • Examples: • Named entities: people, places, organizations, times, dates • <Organization> MIPS</Organization> Vice President <Person>John Hime</Person> • MUC evaluations • Domains: Medical texts, broadcast news (terrorist reports), …
Appropriate where Semantic Grammars and Syntactic Parsers are not • Appropriate where information needs very specific and specifiable in advance • Question answering systems, gisting of news or mail… • Job ads, financial information, terrorist attacks • Input too complex and far-ranging to build semantic grammars • But full-blown syntactic parsers are impractical • Too much ambiguity for arbitrary text • 50 parses or none at all • Too slow for real-time applications
Information Extraction Techniques • Often use a set of simple templates or frames with slots to be filled in from input text • Ignore everything else • My number is 212-555-1212. • The inventor of the wiggleswort was Capt. John T. Hart. • The king died in March of 1932. • Context (neighboring words, capitalization, punctuation) provides cues to help fill in the appropriate slots • How to do better than everyone else?
The IE Process • Given a corpus and a target set of items to be extracted: • Clean up the corpus • Tokenize it • Do some hand labeling of target items • Extract some simple features • POS tags • Phrase Chunks … • Do some machine learning to associate features with target items or derive this associate by intuition • Use e.g. FSTs, simple or cascaded to iteratively annotate the input, eventually identifying the slot fillers
Domain-Specific IE from the Web (Patwardhan & Riloff ’06) • The Problem: • IE systems typically domain-specific – a new extraction procedure for every task • Supervised learning depends on hand annotation for training • Goals: • Acquire domain specific texts automatically on the Web • Identify domain-specific IE patterns automatically • Approach:
Start with a set of seed IE patterns learned from a hand-labeled corpus • Use these to identify relevant documents on the web • Find new seed patterns in the retrieved documents
MUC04 IE Task • Corpus: • 1700 news stories about terrorist events in Latin America • Answer keys about information that should be extracted • Problems: • All upper case • 50% of texts irrelevant • Stories may describe multiple events • Best results: • 50-70% precision and recall with hand-built components • 41-44% recall and 49-51% precision with automatically generated templates
Procedure • Apply pre-defined syntactic patterns to a training corpus of documents for which relevant/irrelevant judgments known • Count how often partial lexicalizations of each (e.g. <subj> was killed) appear in relevant vs. irrelevant documents
Rank patterns based on association with domain (frequency in domain documents vs. non-domain documents) • Manually review patterns and assign thematic roles to those deemed useful • From 40K+ patterns 291 • Now find similar web documents
Domain Corpus Creation • Create IR queries by crossing names of 5 terrorist organizations (e.g. Al Qaeda, IRA) with 16 terrorist activities (e.g assinated, bombed, hijacked, wounded) 80 queries • Restricted to CNN, English documents • Eliminated TV transcripts • Yield from 2 runs: 6,182 documents • Cleaned corpus: 5,618 documents
Learning Domain-Specific Patterns • Hypothesis: new extraction patterns co-occurring with seed patterns from training corpus will also be associated with terrorism • Generate all extraction patterns in CNN corpus (147,712) • Compute correlation of each with seed patterns based on frequency of co-occurrence in same sentence – keep those occurring more often that chance with some seed • Rank new patterns by their seed correlations
Highly Ranked Patterns • Filter: Measure semantic affinity: how often does this pattern extract an entity of a particular category (e.g. victim, target)? • Compute semantic affinity for each extraction pattern wrt 6 categories: target, victim plus distractors: perpetrator, organization, weapon, other • E.g. Frequency of extracting target/frequency of extracting any of 6 categories weighted by log probability of target
Remove patterns not strongly associated with desired classes: • Evaluate on MUC-4 • Baseline: • Recall 64%/Precision 43% on targets • Recall 50%/Precision 52% on victims
Results for Web-Learned Patterns • Use 396 terrorism extraction patterns learned from MUC training set as seeds • Produce ranked list of new patterns from web using semantic affinity of 3.0 threshold • Chose top N (50-300) patterns to add to seed set • Performance:
Combining IR and IE for QA • Information extraction: AQUA
Summary • Many approaches to ‘robust’ semantic analysis • Semantic grammars targeting particular domains Utterance --> Yes/No Reply Yes/No Reply --> Yes-Reply | No-Reply Yes-Reply --> {yes,yeah, right, ok,”you bet”,…} • Information extraction techniques targeting specific tasks • Extracting information about terrorist events from news • Information retrieval techniques --> more like NLP