1 / 11

Linguistic Resources for the 2013 TAC KBP Cold Start Evaluation

Linguistic Resources for the 2013 TAC KBP Cold Start Evaluation. Joe Ellis (presenter ), Jeremy Getman, Jonathan Wright, Stephanie Strassel. Linguistic Data Consortium University of Pennsylvania, USA. Query Selection.

tad
Download Presentation

Linguistic Resources for the 2013 TAC KBP Cold Start Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Linguistic Resources for the 2013 TAC KBP Cold Start Evaluation Joe Ellis (presenter), Jeremy Getman, Jonathan Wright, Stephanie Strassel Linguistic Data Consortium University of Pennsylvania, USA

  2. Query Selection • Annotators search and annotate chains of entities connected by KBP slots • Cold Start queries comprised of • Entity Appleton Museum of Art org:top_members_employees John Lofgren per:title director TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  3. Query Selection • Annotators search and annotate chains of entities connected by KBP slots • Cold Start queries comprised of • Entity – Slot 0 Appleton Museum of Art org:top_members_employees John Lofgren per:title director TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  4. Query Selection • Annotators search and annotate chains of entities connected by KBP slots • Cold Start queries comprised of • Entity – Slot 0 – Slot 1 • Inverse slots to increase connectivity • e.g. per:cities_of_residence– gpe:residents_of_city Appleton Museum of Art org:top_members_employees John Lofgren per:title director TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  5. Query Selection • Annotators search and annotate chains of entities connected by KBP slots • Cold Start queries comprised of • Entity – Slot 0 – Slot 1 • Inverse slots to increase connectivity • e.g. org:founded_by– {per,org,gpe}:organizations_founded Appleton Museum of Art org:top_members_employees John Lofgren per:title director TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  6. Query Selection • Annotators search and annotate chains of entities connected by KBP slots • Cold Start queries comprised of • Entity – Slot 0 – Slot 1 • Inverse slots to increase connectivity • e.g. org:top_members_employees –per:top_member_employee_of • Cold Start corpus • KBA output • Comprised of web documents from Ocala, FL; Kentucky; Guyana Appleton Museum of Art org:top_members_employees John Lofgren per:title director TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  7. Annotation • Unlike other SF tasks, Cold Start annotation is performed concurrently with query development • Multiple fillers at each “hop” level, all of which must be annotated and correctly connected to one another • London – gpe:residents_of_city – per:charges • Lance Barrett • first-degree attempted burglary • theft of a firearm • carrying a concealed weapon • Lesa Bailey • criminal conspiracy to make meth • unlawful possession of meth precursors • possession of a controlled substance TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  8. Assessment • Assess validity of fillers &justification from humans & systems • Filler • Correct – meets the slot requirements and supported in document • Wrong – doesn’t meet slot requirements and/or not supported in doc • Inexact – otherwise correct, but is incomplete, includes extraneous text, or is not the most informative string in the document • Predicate • Correct, Wrong, Inexact-Short, Inexact-Long • Subject/Object • Correct, Wrong, Inexact • Ignore TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  9. New in 2013: Justification • Justification is the string(s) of text that show a relation is true • Predicate: Includes all three pieces of information necessary to justify the entity/slot/filler relation • Subject: proves the entity’s involvement in the relation • Object: proves the filler’s involvement in the relation • Each part can be comprised of up to two, discontiguous strings • <Harkat-ul-Mujahideen - org:country_of_headquarters - Pakistan> • Predicate 1: the Islamabad headquarters of Harkat-ul-Mujahideen • Predicate 2: Islamabad, the capital city of Pakistan Ronnie James Dio -per:date_of_death: Sunday [2010-05-16] TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  10. 2013 Discoveries • New justification scheme used in unexpected, creative ways • Additional predicate strings used to disambiguate entities • <VitalyGinzburg - per:cause_of_death - cardiac arrest> • Predicate 1: Ginzburg died late Sunday of cardiac arrest. • Predicate 2: VitalyGinzburg, a Nobel Prize-winning Russian physicist and one of the fathers of the Soviet hydrogen bomb TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

  11. Delivered 2013 Resources TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

More Related