Language & Speech Technology

1. Language & Speech Technology Arjan van Hessen+* Franciska de Jong* Roeland Ordelman* * Computer science, University Twente + Speech & Language group, TeleCats

2. Document Retrieval Using Intelligent Disclosure

3. DRUID “Developing Tools for the Indexing & Retrieval of Multi Media Content” • time-coded indexing with DUTCH speech recogniser • television news broadcast • benchmark international SDR research • parallel sources available (teletext, auto cues)

4. Druid: what • Extract information from non-textual content • Classify and index the information • Give access to the information via linked time codes

5. Druid: how • Speech recognition • Large vocabulary, speaker independent • Recognition of visual objects • Story detection • Linking to related information

6. Indexing & Retrieval Large vocabulary recognition

7. Druid Speech recogniser • ABBOT speech recogniser (Cambridge, Sheffield) • Feature extraction • Phone classification (NN) • Word recognition (HMM)

8. Broadcast news • Pro’s • Easy available • Often high quality, undisturbed speech • Availability of related sources • (auto-cues, news papers) • Contra’s • Mixed languages • Different quality of speech (wide & narrow band), mixed together

9. Development • British English  Dutch • TNO-NRC corpus: 10h read speech (newspaper data) • Additional phoneme training • Groningen corpus: 20h read speech • Speech Styles corpus: 16h spontaneous speech • Final training • Broadcast corpus: 50 x “8 o’clock news” broadcasts (10h speech) • Corpus Spoken Dutch: 1000h spontaneous speech (to be done in 2002)

10. Language modelling • Acoustic recognition stops at a certain level • Recognition can only improve with: • Statistical language models (large vocabulary recognition) • Finite state grammars(small vocabulary recognition)

11. Large vocabulary recognition • Recognition is directed by • Acoustic features • Word frequency (= 65K most used words) • Bi-grams (65K2 combinations) • Tri-grams (65K3 combinations)

12. Large vocabulary recognition • Building reliable acoustic feature requires 100 hours of speech • Building reliable LM requires 10.000 hours of text • Different context models (sport, finance, politics etc.)

13. Language modelling Standard LM procedure • text normalisation Dutch diseases: • spelling reform 90’s • compounding • foreign words • increase of English

14. Text collection • Nederlandse Persdata bank • Electronic version of 4 major Dutch newspapers (1994-2002) • NOS Auto cues • Daily Auto-cues of the 8 o’clock news and the news for children (1999-2002) • TeleText • Daily recording of the teletext of the news, discussion & sport programs (1998-2002) • WWW • Daily downloading of news providers & papers (2000-2002)

15. Text collection Number of words of the newspaper collection after normalisation

16. Phonetic transcriptions • Phonetic dictionaries • Celex (300k, SAMPA) • VLIS database (1300k, Van Dale Data Format) • Rule-based decompounded-compounded dictionary (600k, SAMPA) • G2P tool • Machine learning algorithm (vd Bosch) • 95% correct (without syllable & stress information)

17. Text normalisation I • Cleaning of punctuation marks • Expansion • Numbers, abbreviations • Statistical capital letter reduction • Rotterdam, rotterdam, ROTTERDAM  Rotterdam • KOK, Kok, kok  kok • Spelling correction • Reduction of “doubles” caused by the spelling reform of the nineties (pannekoek  pannenkoek) • Removal, correction, or adding of accentuation marks • cafe, café , cafeé, cafë etc.  café • hét, hèt  het

18. Text normalisation II • German and Dutch are “compound” languages • Increased number of words • Relative high number of “new” words • (Eclipsbril = Eclipse glasses) • Lowe lexical coverage  High OOV • LC = #word/(#distinct words) • OOV = 1- LC

19. Text normalisation III drugsbaas drugsbanden drugsbaron drugsbaronnen drugsbazen drugsbedrijf drugsbeleid drugsbende drugsbendes drugsbestaan drugsbestellingen drugsbestrijdend drugsbestrijder drugsbestrijders drugsbestrijding drugsbezit drugsbezitters drugsboef drugsboeven drugsbonzen drugsbrigade drugsbrigades drugsbron drugsbuisje drugsbureau drugsbusiness drugsbuurt drugscafé drugscafés drugscampagnes drugscare drugscircuit drugsclans drugsclip drugscocktail drugscocktails drugsconferentie drugsconflict drugsconnecties drugsconsument drugsconsumptie drugscontainers drugscontrole drugscontroles drugsconventie drugscriminaliteit drugscrimineel drugscriminelen drugsdaglicht drugsdeal drugsdealen drugsdealend drugsdealende drugsdealer drugsdealers drugsdeals drugsdebat drugsdelict drugsdelicten drugsdeskundige drugsdiscussie drugsdode drugsdoden drugsdollars drugsdominee drugsdood drugsdossier drugsdossiers drugsdraaiboek drugseconomie drugseenheid drugsellende drugsexcessen drugsexperiment drugsexpert drugsexperts drugsexport drugsfabricage drugsfabrikanten drugsfamilie drugsfunctionaris drugsgebied drugsgebruik drugsgebruiker drugsgebruikers drugsgebruikster drugsgeld drugsgelden drugsgelieerde drugsgeschiedenis drugsgeschillen drugsgewoonte drugsgoeroe drugsgroeperingen drugsgrondstoffen drugshaarden drugshandel drugshandelaar drugshandelaars drugshandelaarster drugshandelaren drugshandlangers drugshel drugshoertje drugshol drugshond drugshonden drugshoofdstad drugshuizen drugshulpverleners drugshulpverlening drugsimago drugsimport drugsindustrie drugsinkomsten drugsinstelling drugsinval drugsinvoer drugsjacht drugsjagende drugsjager drugsjaren drugsjongens drugskartel drugskartels drugbeleid drugbestrijding drugbezit drugdealer drugdealers drugdeals drugdelict drugdistributeur druggebruik druggebruiker druggebruikers drughandel drugkartels drugmisbruik drugrunner drugsaanpak drugsactie drugsacties drugsactiviteiten drugsadviseur drugsafdeling drugsaffaire drugsaffaires drugsafrekeningen drugsattributen drugsavonturen drugsavontuur

20. Text normalisation VI • Decompounding • Low frequency compounds are decompounded if decompounding improves the Lexical Coverage • 50% of the unique words that were not in one of the phonetic dictionaries could be successfully decompounded although some error were made: • zeeroverschatten  zeerover + schatten  zeerovers + chatten

21. TOP 10 de 5532695 van 2763280 het 2535365 en 2210685 een 2146813 in 1994480 dat 1129136 is 1080972 op 957296 te 897219 Most / least frequent words • DOWN 10 • milko 39 • miljardenovername 39 • mifune's 39 • middeninkomen 39 • michelingids 39 • mexx 39 • metaalnijverheid 39 • metaaldetectoren 39 • mesquita 39 • mervyn 39

22. Language modelling

23. Language modelling Effect on the ratio after decompounding

24. Different language models First use the general LM to detect the sub-category Use the politic LM to improve recognition results

25. Segmentation I • Full news broadcasts are too long (20 min.) • Retrieved items may start and/or stop in the middle of phrases • different LM has to be assigned to different “stories”

26. Segmentation II • Segmentation in phrases, sentences, and paragraphs • Prosodic information • F0 • Pauses • Different LM assigning

27. Results

28. Results WER extra Read speech 30% (OOV = 2.5%) 15 hrs training material Broadcast news 36.9% (OOV = 14%) 5 hrs training material Historical archives 90% (OOV = 20%) 1933 Historical archives 60% (OOV = 10%) 1940 Historical archives 43% (OOV = 14%) 1960

29. “de Israëlische premier Chevron houdt vanavond en televisie toespraak zullen ingaan op de crisis die is ontstaan na de bloedige aanslagen van het weekend in Jeruzalem en hij vaak zo'n kwam vanochtend vroeg terug uit Amerika heeft gesproken met president Bush het ene op het vliegveld van Tel Aviv pasje om met ministers pers en ben een Jezus met weinig gevoel voor huizen vanavond is het kabinet kabinet beraadt geweld gaat ook vanochtend door op de westelijke Jordaan oever bijen is z'n vijven dertig jarige Palestijn door Israëlische militairen gedood die bij controle proberen te vluchten of stonden Shivaheeft pech” “de Israëlische premier Sharon houdt vanavond ‘n televisie toespraak. Hij zal dan ingaan op de crisis die is ontstaan na de bloedige aanslagen van dit weekend in Jeruzalem en Haifa. Sharon kwam vanochtend vervroegd terug uit Amerika; daar heeft hij gesproken met president Bush. Meteen al op het vliegveld van TelAviv sprak Sharon met de ministers Peres en Ben Illiëzer en met veiligheidsfunctionarissen. Vanavond is het kabinet kabinetsberaadt. ‘t geweld gaat ook vanochtend door, op de westelijke Jordaanoever bij Jinien is 'n vijfendertig jarige Palestijn door Israëlische militairen gedood toen ie bij controle probeerden te vluchten. Correspondent: Shivra Hertzberg” DRUID 7.2 3 December 2001 12:14

30. OOV problems 20% (14k) of the 65k most frequent words (MFW) are not in the phonetic dictionary 86% of these 14k words starts with a capital letter 50% of these 14k words are names (family, geographic, companies) that are not in the phonetic dictionary and are difficult to transcribe by G2P because they often do not follow Dutch transcription rules

31. Demo 8 o’clock TV news Daily radio news Adjust

32. DRUID • Evaluation • A time consuming, boring, but necessary process!!

33. Questions ?