1 / 12

Overview of QAST 2008 - Question Answering on Speech Transcriptions -

Overview of QAST 2008 - Question Answering on Speech Transcriptions -. J. Turmo, P. Comas (1), L. Lamel, S. Rosset (2) , N. Moreau, D. Mostefa (3) (1) UPC, Spain (2) LIMSI, France (3) ELDA, France QAST Website : http://www.lsi.upc.edu/~qast/. Objectives Description of the tasks

elia
Download Presentation

Overview of QAST 2008 - Question Answering on Speech Transcriptions -

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of QAST 2008 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1), L. Lamel, S. Rosset (2) , N. Moreau, D. Mostefa (3) (1) UPC, Spain (2) LIMSI, France (3) ELDA, France QAST Website : http://www.lsi.upc.edu/~qast/

  2. Objectives Description of the tasks Participants Results Future work Outline

  3. Objectives of QAST 2008 • Development of robust QA for speech transcripts • Measure loss due to ASR inaccuracies manual transcriptions, automatic transcriptions • Measure loss atdifferent ASR word error rates • Test with different kinds of speech spontaneous speech,prepared speech • Development of QA for languages other thanEnglish English,French, Spanish

  4. QAST 2008 Organization Task jointly organized by : • UPC, Spain (Coordinator) J. Turmo, P. Comas • ELDA, France N. Moreau, D. Mostefa • LIMSI-CNRS, France S. Rosset, L. Lamel

  5. Evaluation Data

  6. Factual questions: ~75% Expected answers = named entities (10 types: person, location, organization, language, system, measure, time, color, shape, material) Definition questions: ~25% 4 types of answers: person, organization, object, other ‘NIL’ questions: ~10% Questions

  7. Participants could submit up to: 2 submissions per task (and per WER) 5 answers per question Answers for ‘manual transcriptions’ tasks: Answer_string + Doc_ID Answers for ‘automatic transcriptions’ tasks: Answer_string + Doc_ID + Time_start + Time_end Submissions

  8. Four possible judgments (as in QA@CLEF): Correct / Incorrect / Inexact / Unsupported ‘Manual transcriptions’ tasks: Manual assessment with the QASTLE interface ‘Automatic’ transcriptions tasks Automatic assessment (script) + manual check 2 metrics: Mean Reciprocal Rank (MRR) measures how well right answers are ranked on average Accuracy fraction of correct answers ranked in the first position Assessments

  9. 49 submissions from 5 participants: Participants

  10. Best results for manual transcriptions

  11. Best results for ASR transcriptions

  12. 5 participants (as in 2007) 4 different countries (vs. 5 in 2007) Germany, Spain, France, Mexico 49 submitted runs (vs. 28 runs in 2007) Loss in accuracy with ASR transcribed speech (performance falls when WER rises) QAST 2009: Written & Oral Questions... Conclusion

More Related