1 / 23

ADGEN: Advanced Generation for Question Answering

ADGEN: Advanced Generation for Question Answering. Kevin Knight and Daniel Marcu (co-PIs) USC/Information Sciences Institute December 4-6. ADGEN Research Focus. Of the myriad variations of a text that the machine might produce for an analyst, only a fraction are coherent.

king
Download Presentation

ADGEN: Advanced Generation for Question Answering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ADGEN: Advanced Generation for Question Answering Kevin Knight and Daniel Marcu (co-PIs) USC/Information Sciences Institute December 4-6

  2. ADGEN Research Focus • Of the myriad variations of a text that the machine might produce for an analyst, only a fraction are coherent. • What makes a text coherent? • New Approach: • We have millions of examples of coherent texts • We can validate ideas empirically, develop models • We can train models automatically

  3. ADGEN in AQUAINT • 1. Answer generation • Input: collection of text fragments (including phrases and paragraphs) • Fuse phrases into sentences, order sentences to form millions of possible texts • Rank and select most coherent presentation • 2. Text improvement • Input: existing text • Apply probabilistic rewriting operations • Select rewrite that most improves coherence without sacrificing any of the basic material

  4. Text-Level Language Models • Given an unordered bag of answers/clauses/sentences, assign an order that yields a coherent text. • Typical discourse study: “if we scramble sentences in an English document, the result is not coherent, so text has structure…” • Let’s do something about it!

  5. Models have multiple applications • Word-level ordering Machine Translation Meaning-to-Text Generation Text-level ordering Essay Grading Multi-document Summarization ?

  6. Article Ordering Experiments • Test set of 51 scrambled Wall Street articles • - re-ordered by human • 50% perfectly match original • 90% only one off • Initial modeling ideas • - a good article usually contains reasonable adjacent sentence pairs • - adjacent sentences A and B are often related in interesting ways • - if A includes sell, B often includes sale • - asymmetric -- reverse is not true nearly as often

  7. Article Ordering Experiments • Initial generative story for P(B | A) • - Given sentence A = a1 … an: • decide on length m for B (in # of words) • for i = 1 to m • pick a “source word” position j from A • pick a “target word” bi for B, according to P(bi | aj) • print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. A B

  8. Article Ordering Experiments • Initial generative story for P(B | A) • - Given sentence A = a1 … an: • decide on length m for B (in # of words) • for i = 1 to m • pick a “source word” position j from A • pick a “target word” bi for B, according to P(bi | aj) • print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. A B

  9. Article Ordering Experiments • Initial generative story for P(B | A) • - Given sentence A = a1 … an: • decide on length m for B (in # of words) • for i = 1 to m • pick a “source word” position j from A • pick a “target word” bi for B, according to P(bi | aj) • print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. Terms A B

  10. Article Ordering Experiments • Initial generative story for P(B | A) • - Given sentence A = a1 … an: • decide on length m for B (in # of words) • for i = 1 to m • pick a “source word” position j from A • pick a “target word” bi for B, according to P(bi | aj) • print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. Terms weren't disclosed, but industry sources said the price was about $2.5 million. A B

  11. S2 S1 S4 S3 text boundary Article Ordering Experiments • Train P(bi | aj) on large, “adjacent-sentence” database from 1987 WSJ • - using standard machine-translation package GIZA developed at JHU WS’99 • Portion of learned words predicted by judge: • Deployment on 51 test articles • - for each article, compute P(B | A) costs for every sentence pair in an article • - select sentence order that minimizes total cost, using standard TSP algorithm 0.2191190 judge 0.0674230 court 0.0601497 he 0.0385838 the 0.0329773 that 0.0310724 his 0.0240846 to 0.0204204 " 0.0198243 case 0.0148957 law 0.0138114 a 0.0115714 of 0.0112826 ruling 0.0109133 in 0.0108175 against 0.0091600 lawyers 0.0087317 by 0.0083675 appeals 0.0082462 said Best tour: <tb> S4 S3 S1 S2 <tb>

  12. Article Ordering Results

  13. Disassembling Bags of Sentences • Another evaluation: • - Given mixed bag of sentences drawn from two articles • X1X4 Y3 X2 Y1 Y2 X3 • - Assign an order to them • Y2 Y3 Y1 X1 X2 X3 X4 • - Does the assigned order separate the two stories? • Answer: 86% correct on 49 article pairs.

  14. Article Ordering Results

  15. Building from the Baseline:What’s Wrong With Machine Articles? • Plenty. • Is it possible to distinguish original human articles from machine-ordered articles? • (If not, we are done!) • Classification task: Believed to be human articles 80+% Believed to be machine articles

  16. What’s Wrong With Machine Articles? • Plenty. • Is it possible to distinguish original human articles from machine-ordered articles? • (If not, we are done!) • Classification task: Believed to be human articles 61% Believed to be machine articles

  17. Exploiting What’s in the Classifier • Classifier can’t actually order new articles for us, because it only looks for certain things. • But, its knowledge can be put back into the ordering algorithm • - large-scale feature-based training • - interesting technical challenges • Idea: next iteration, the classifier is back to 50% and we start looking for new features. • People are good at writing classifiers.

  18. Why is this important for QA? • Outputs in the QA-definitions task: • Today: • Who is Gunter Blobel? • A1: molecular biologist • A2: cellular and molecular biologist • A3: won the 1999 Nobel Prize in medicine • What we want: • Dr. Gunter Blobel is a cellular and molecular biologist • who won the 1999 Nobel Prize in medicine.

  19. Why is this important for QA? • Outputs of causal questions: • Today: • sets of sentences. • What we want: • coherent text. • In order to move beyond lists of sentences and sentence fragments, we need general purpose NLG algorithms.

  20. Next Steps • Deal with redundancy • Terms weren't disclosed, but industry sources said the price was about $2.5 million. • The sale includes the rights to Germaine Monteil in North and South America. • Terms were not disclosed by either party. • Deal with contradiction • Terms weren't disclosed, but industry sources said the price was about $2.5 million. • Revlon said it paid $2.2 million for Germaine Monteil. • Close the loop on generating/exploiting more discourse features • Create useful generation capability

  21. Thank You

  22. Sample Problem • 1. Terms weren't disclosed, but industry sources said the price was about $2.5 million. • 2. Revlon is a cosmetics concern, and Beecham is a pharmaceutical concern. • 3. Revlon Group Inc. said it completed the acquisition of the U.S. cosmetics business of Germaine Monteil Cosmetiques Corp., a unit of London-based Beecham Group PLC. • 4. The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products.

  23. Sample Problem • 1. Terms weren't disclosed, but industry sources said the price was about $2.5 million. • 2. Revlon is a cosmetics concern, and Beecham is a pharmaceutical concern. • 3. Revlon Group Inc. said it completed the acquisition of the U.S. cosmetics business of Germaine Monteil Cosmetiques Corp., a unit of London-based Beecham Group PLC. • 4. The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. Correct order: 3, 1, 4, 2

More Related