40 likes | 150 Views
Evaluation of Multi-Media Data QA Systems AQUAINT Breakout Session – June 2002. Howard Wactlar, Carnegie Mellon Yiming Yang, Carnegie Mellon Herb Gish, BBN. Extracting Answers From …. Web-based charts and graphs – Yiming Yang Continuous speech – Herb Gish Video – Howard Wactlar.
E N D
Evaluation of Multi-Media Data QA SystemsAQUAINT Breakout Session – June 2002 Howard Wactlar, Carnegie Mellon Yiming Yang, Carnegie Mellon Herb Gish, BBN
Extracting Answers From … • Web-based charts and graphs – Yiming Yang • Continuous speech – Herb Gish • Video – Howard Wactlar
Multimedia Data Extraction • Long term goal: Ability to respond to same challenge and metrics as current and forthcoming TREC-QA • But with different source data • Multiple channels of information (with temporal and spatial dimensions) simultaneously • Problem: Errors inherent in extraction and interpretation • …but parallel sources of data may corroborate or refute
Evaluation Task Proposal: Multi-modal Biography Generation • Task Description: • Given a name, generate a multi-modal summary of key characteristics and events of person’s life in a specified time frame. • Response: • Natural language passages/sentences of key events • Recent/historical pictures of individual • video/radio segments (e.g., speech,interview) augmenting key events • Org chart, map of movements • Data: • Web images and text • Video and radio from news sources • Eval: • Subjective (analyst / other user) assessment • Objective: precision, recall, recency, coherence Goal for next meeting: Define application-based task and metrics