iCLEF-2003 at Maryland Headline Generation and Interactive Query Formulation

1. iCLEF-2003 at MarylandHeadline Generation and Interactive Query Formulation Douglas W. Oard and the Maryland/BBN iCLEF team CLEF 2003

2. Outline • Document selection • First 40 translated words • Automatically generated translated “headlines” • Application: news stories from speech recognition • Iterative refinement • User-assisted query translation • First 40 words + access to full translation

3. Berne, 25 May (sda) the Upper House of Parliament supports the objectives that national campaign against racism. It is ready, all possibilities to examine, like the Council of Europe the action of the working group that Youth Upper house of Parliament supports objectives that national campaign against racism Eight months before popular vote has itself in Berne Swiss federal committee presented Authorities of Pakistan have that on Monday Indian consulate general closed Swiss federal committee to anti-racism law presented Upper house of parliament endorses active participation that Switzerland at European youth campaign UN human right commission has on for first time resolution as well as anti-semitism Boy LDU does not want scientologists in rows Since aunt died in Zurich George Hug lead and two brothers and sisters worldwide fight Of members pc. Gallen have that Party exclusion Galler physician and opponent walter Fischbacher requests Black humans must legitimize themselves thus again once In the small urban villa at the Museum in Amsterdam took 11 1987 one of the largest German tax affairs their beginning. Here **time-out** hid the company Sunpark sport B.V. Steffis income before that German treasury. Nut/mother of Islamabad, 26th Dec (sda/afp) the authorities of Pakistan have that on Monday Indian Consulate General in Karachi closed. To the reason that said a minister of foreign affairs of Pakistan Najmuddin Sheikh, India is into unrests in Berne, 31 January (sda) eight months before the popular vote has itself on Monday in Berne the Swiss federal committee “to the anti racism law” presented. Nearly all parties are, likewise involved employers and Trade unions as well as Berne, 7 September (sda) the Upper House of Parliament endorses an active participation that Switzerland at the European youth campaign against racism and intolerance. It is ready, so rapidly as possible the necessary decisions to Geneva, 9 March (sda) the UN human right commission has on Wednesday for the first time a resolution against the racism generally as well as the antisemitism, the racism against Arabs and against black ones discharges. Before one Zurich, 26 October (sda) the boy LDU does not want Scientologist in its Rows. Executive committee and presidency have one in the statutes incompatability rule imported, like JLDU vice-president Roland Tschaeppeler on Wednesday Since their aunt died 1990 in Zurich, George Hug lead and its two brothers and sisters world-wide a fight against physician, those medical allowance with financial interests mix. “gradually”, Barbara Hug reports, well-known St. Gallen, 26 August (sda) of members of FDP of the city pc, Gallen have that Party exclusion of the pc, Galler of physician and opponent of the Anti racism of law walter Fischbacher requests. Like it in one Report of Friday is called. Black humans must legitimize themselves thus again once. The psychological and social effects S.14 of racism however, like among other things chance inequality, bad education and poverty, “incomprehensible-prove” too such IQ Headline Generation

4. Types of Headlines • “Eye catcher”: ‘Under God’ Under Fire • Indicative: Pledge of Allegiance; Constitutionality • Informative: U.S. Court Decides Pledge of Allegiance Unconstitutional

5. HMM Hedge Form headlines by selecting words from story, in order, allowing morphological variation After months of debate following the Sept. 11 terrorist hijackings, the Transportation Department has decided that airline pilots will not be allowed to have guns in the cockpits. After months of debate following the Sept. 11 terrorist hijackings, the Transportation Department has decided that airline pilots will not be allowed to have gunsin the cockpits. After months of debate following the Sept. 11 terrorist hijackings, the Transportation Department has decides that airline pilots will not be allowed to have guns in the cockpits. Humans were able to do this on all prose stories

6. Human-Generated Headlines Sentence Number

7. Hedge Trimmer Form headlines by removing constituents from a parse of the first sentence. Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields. S 28 words NP VP S NP VP V NNP VP V PP Time Adv NNP VP NP AP PP

8. Hedge Trimmer Form headlines by removing constituents from a parse of the first sentence. Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields. S 24 words NP VP S NP VP V NNP VP V PP Time Adv NNP VP NP AP PP

9. Hedge Trimmer Form headlines by removing constituents from a parse of the first sentence. Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields. S 20 words NP VP S NP VP V NNP VP V PP Time Adv NNP VP NP AP PP

10. Hedge Trimmer Form headlines by removing constituents from a parse of the first sentence. Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields. S 18 words NP VP S NP VP V NNP VP V PP Time Adv NNP VP NP AP PP

11. Hedge Trimmer Form headlines by removing constituents from a parse of the first sentence. Kurdish guerilla forces moving with lightning speed poured into Kirkuk today immediately after Iraqi troops, fleeing relentless U.S. airstrikes, abandoned the hub of Iraq’s rich northern oil fields. S 14 words NP VP S NP VP V NNP VP V PP Time Adv NNP VP NP AP PP

12. Hedge Trimmer Form headlines by removing constituents from a parse of the first sentence. Kurdish guerilla forcesmoving with lightningspeedpoured into Kirkuktodayimmediatelyafter Iraqi troops,fleeing relentless U.S.airstrikes,abandonedthe hub of Iraq’s richnorthernoil fields. S 12 words NP VP S NP VP V NNP VP V PP Time Adv NNP VP NP AP PP

13. Human-Generated Headlines

14. Named Entity Preservation • Syntactic structure alone is not sufficient for deciding which constituents to remove • Examples: • Original: According to local authorities the death toll is expected to reach 1,500 with 7000 wounded in Algeria’s earthquake. (18) • without NEP: According to local authorities the death toll is expected to reach 1,500 with 7000 wounded in Algeria’s earthquake.(10) • with NEP: According to local authorities the death toll is expected to reach 1,500 with 7000 wounded in Algeria’s earthquake. (10)

15. Intrinsic Evaluation: BLEU BBN/UMD Trim

16. Intrinsic Evaluation: RED BBN/UMD Trim

17. Making Translated Headlines Document Systran: Translate First Sentence Hedge Trimmer “Headline”

18. Manual Intrinsic Evaluation Q0: Can you tell what this story is about? Q1: Were you (was it) correct?

19. iCLEF Results • Search effectiveness • First40: Fα = 0.473 • Trimmer: Fα = 0.379 • Search speed • First40: 16.75 seconds/doc • Trimmer: 12.35 seconds/doc • Significant with p<0.007

20. Topic Questionnaires Q1: Were you familiar with this topic before the search? Q2: Was it easy for you to guess what the document was about based on the surrogate? Q3: Was it easy to make relevance judgments? Q4: Do you have confidence in your judgments?

21. Monolingual Searcher Choose Document-Language Terms Generate Possible Translations Select Document-Language Terms Query Focus: Query Formulation Cross-Language Searcher Author Choose Document-Language Terms Choose Query-Language Terms Query-Document Matching Document

22. 2003 End-to-End Experiments • 64 trials (8 participants) • Half with automatic query translation • Half with user-assisted query translation • Spanish documents • Systran translations into English • Experienced searchers • 6 of 8 were librarians, average age 33 • 7 of 8 had little or no experience with MT • 8 of 8 reported poor or no Spanish skills

24. Bilingual Term List translation aligned with Sentence aligned Corpus Examples from Parallel Text English Word te Spanish Word tg Passage containing te Passage containing tg

25. English Text + Translation Lexicon • Create a pool of candidate examples • Find possible synonyms w/round-trip translation • Select all translations with any “back translation” • Find terms that characterize the pool • Rank non-stopwords by sentence-based tf*idf • Select top 10 terms as a query • Choose one example of usage • Rank candidate pool sentences using the query • Select the top-ranked sentence

26. 2002 Results Fα=0.8 20 minutes per topic

27. 2003 Results Fα=0.8 10 minutes per topic