1 / 9

SDC: The Spoken Dialog Challenge

SDC: The Spoken Dialog Challenge. Maxine Eskenazi Alan Black. SDC: the Spoken Dialog Challenge. Organized by the Dialog Research Center (NSF-sponsored) Guided by Community input Uses a common platform to compare research techniques to compare evaluation techniques

jaime-britt
Download Presentation

SDC: The Spoken Dialog Challenge

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SDC: The Spoken Dialog Challenge Maxine Eskenazi Alan Black

  2. SDC: the Spoken Dialog Challenge • Organized by the Dialog Research Center (NSF-sponsored) • Guided by Community input • Uses a common platform • to compare research techniques • to compare evaluation techniques • to compare simulated users • Share: • Data • System (optional) • Access to real users

  3. SDC 2010 • October 2009: Training webcasts • January 2010: Data distributed • January 2010 - April 2010: Development period • May 2010: Control tests • July - August 2010: Live tests • December 2010: Reporting: Special Session SLT

  4. Getting started • Task: bus information using the Let’s Go system with real users • Dialog data distributed: • 700GB (1 year of data, ~14,000 dialogs) • Labeled (crowdsourcing and some expert) • Detailed logs and audio • Base system distributed: • Olympus, training sessions, but participants could use their own systems

  5. What happened? • Participants:Academia and industry, internationally • Four systems in control test • Three systems in live test • Three evaluation schemes • One simulated user system • Scheme to transfer calls to other countries (thanks to ATT!) • Want to know the results? Come to SLT!

  6. SDC 2011 • Same application • Allows those who did not participate this year to join • “first pancake” this year • Will allow comparison over two years • Looking for new applications with real users for SDC 2012 • Aiming at publications and a teaching tool – two tracks

  7. SDC 2011 Schedule • Track 1 SDC 2011 Track 2SDC 2011 Primary TrackSecondary track • 2010 Nov Development2010 Dec   Development2011 Jan   Development2011 Feb   Development2011 Mar   Control Tests2011 Apr   Control TestsDevelopment           Dry Runs 2011 May   CT labeling       Development2011 Jun   CT results        DevelopmentLive Test/Sim2011 Jul   Live Test/SimDevelopment2011 Aug   Live Test/SimControl Tests2011 Sep   LT labelling      Control Tests                             Dry runs 2011 Oct   LT results        CT labeling EvaluationSDC2012Stream 12011 Nov   Evaluation        CT Results          DevelopmentLive Test/SimDevelopment2011 Dec   EvaluationLive Test/SimDevelopment2012 Jan                     Live Test/SimDevelopment2012 Feb                     LT labeling         Control Tests2012 Mar                     LT results          Control TestsEvaluation          ... 2012 Apr                     Evaluation2012 May                     Evaluation

  8. SDC • Questions?

  9. max@cmu.edu • awb@cs.cmu.edu

More Related