1 / 10

Linguistic Resources for the 2013 TAC KBP Sentiment SF Evaluation

This resource provides linguistic data and tools for the evaluation of sentiment analysis tasks, including query selection, annotation, assessment, and justification.

cundiff
Download Presentation

Linguistic Resources for the 2013 TAC KBP Sentiment SF Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Linguistic Resources for the 2013 TAC KBP Sentiment SF Evaluation Joe Ellis (presenter), Jeremy Getman, Jonathan Wright, Stephanie Strassel Linguistic Data Consortium University of Pennsylvania, USA

  2. 2013 Source Corpus TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  3. Query Selection • 1 - Select queries and reference docs • Four slots used in Sentiment Slot Filling • positive-towards, positive-from • negative-towards, negative-from • Sentiment Slot Filling queries comprised of • Entity – Slot • Rich queries (at least 2-3 instances of sentiment in source corpora) • Sentiment defined as a positive or negative emotion, evaluation, or judgment. • 2 - Link namestrings to KB or mark as NIL TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  4. Query Selection • 1 - Select queries and reference docs • Four slots used in Sentiment Slot Filling • positive-towards, positive-from • negative-towards, negative-from • Sentiment Slot Filling queries comprised of • Entity – Slot • Rich queries (at least 2-3 instances of sentiment in source corpora) • Sentiment defined as a positive or negative emotion, evaluation, or judgment. • 2 - Link namestrings to KB or mark as NIL TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  5. Annotation • For each query annotator spends up to 2 hours searching corpus for instances of sentiment of the correct directionality and polarity Ronnie James Dio neg-from: InfraBlue cavfancier TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  6. Assessment • Assess validity of fillers &justification from humans & systems • Filler • Correct – meets the slot requirements and supported in document • Wrong – doesn’t meet slot requirements and/or not supported in doc • Inexact – otherwise correct, but is incomplete, includes extraneous text, or is not the most informative string in the document • Predicate • Correct, Wrong, Inexact-Short, Inexact-Long • Subject/Object • Correct, Wrong, Inexact • Ignore TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  7. New in 2013: Justification • Justification is the string(s) of text that show a relation is true • Predicate: Includes all three pieces of information necessary to justify the entity/slot/filler relation • Subject: proves the entity’s involvement in the relation • Object: proves the filler’s involvement in the relation • Each part can be comprised of up to two, discontiguous strings • <Ronnie James Dio – neg-from – Westboro Baptist Church> • Predicate 1: Westboro Baptist Church said they oppose anyone they believe worships Satan. • Predicate 2: The fundamentalist church said that includes Ronnie James Dio, who died Sunday. Ronnie James Dio neg-from cavfancier TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  8. 2013 SSF Discoveries • The reclassification of top-level governments of GPEs as GPEs themselves proved particularly beneficial in Sentiment SF • Examples like (1) are much more prevalent than examples like (2) • (1) The Palestinian government has denounced what it calls the Israeli army's 'current practice of shoot now and ask questions later.‘ • (2) We're kinda like David Hasselhoff; where we're big in Germany, but nobody else cares. • Especially useful since actions as indicators of sentiment were invalid • e.g. - Israel launched an air strike against Syria. • -Towards slots significantly less productive than-From slots • Most annotatable sentiment is in discussion forum • But post authors can only be fillers, not query entities • Neither NIL or non-NIL – UNKNOWN! TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  9. 2013 SSF Discoveries Discussion Forums • <post> and <quote> tags seemed to be difficult to parse • <post author="Monketey Ghost" datetime="2010-12-21T07:54:00" id="p13"> <quote orig_author="beachnut"> Found what I think is some backwards speech on the fade-out at the very end of Ronnie James Dio's Holy Diver album. Anyone? </quote> Just listened and can't tell. Great album, though. Love that guy. </post> • Seemingly difficult to extract succinct justification from DF docs • Predicate strings consisting of entire posts: • <post author=“Lakhota” datetime=“2011-10-06T17:29:00” id=“p8”> Elizabeth Warren's a senator from Massachusetts. She's a great American patriot. No one fights harder for the middle class. </post> • More ‘Ignore’ assessments in SSF than in other tasks Ronnie James Dio pos-from Monketey Ghost ? beachnut ? TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  10. Delivered 2013 Resources TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

More Related