1 / 64

The MEI Team August 23, 2000

Mandarin-English Information (MEI): Investigating Translingual Speech Retrieval Johns Hopkins University Center of Language and Speech Processing Summer Workshop 2000. The MEI Team August 23, 2000. MEI Team. Senior Members Students. Helen Meng Chinese University of Hong Kong

fifi
Download Presentation

The MEI Team August 23, 2000

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mandarin-English Information (MEI):Investigating Translingual Speech RetrievalJohns Hopkins University Center of Language and Speech ProcessingSummer Workshop 2000 The MEI Team August 23, 2000

  2. MEI Team • Senior Members • Students Helen Meng Chinese University of Hong Kong Erika Grams Advanced Analytic Tools Sanjeev Khudanpur Johns Hopkins University Gina-Anne Levow University of Maryland Douglas Oard University of Maryland Patrick Schone US Department of Defense Hsin-Min Wang Academia Sinica, Taiwan Berlin Chen National Taiwan University Wai-Kit Lo Chinese University of Hong Kong Karen Tang Princeton University Jianqiang Wang University of Maryland

  3. Outline • Motivation • Background • The Multi-scale Paradigm • multi-scale query processing • multi-scale document indexing • multi-scale retrieval • The Perfect Retrieval Myth • Experiments and Findings • Conclusions and Future Work

  4. Motivation • Monolingual speech retrieval applications are emerging, e.g. • http://speechbot.research.compaq.com Internet-accessible Radio and Television Stations source: www.real.com, Feb 2000

  5. Motivation (cont): Internet User Population 2000 2005 English English Chinese Source: Global Reach

  6. English Text Query (Exemplar) Mandarin Audio News Broadcasts English-to-Chinese Translation Mandarin Audio Indexing (ASR) Retrieval Engine Ranked List of Mandarin Spoken Documents Interactive Refinement Speech-to-Speech Translation English Spoken Documents MEI: The Big Picture

  7. Concept Demo Karen, Erika

  8. Two Prevailing Problems in CL-SDR • Translation problem • out-of-vocabulary (OOV) in translation • too many translations • Recognition problem • OOV in recognition • acoustic confusions • Solution: subword units may help • transliteration, e.g. Northern Ireland /bei3 ai4 er3 lan2/ (in query) • recognition of subword units, e.g. Iraq --> a rock (in document)

  9. Background for Mandarin Speech Recognition • 400 syllables • full phonological coverage in Mandarin Chinese • 6,800 characters • full textual coverage in written Chinese (GB-coded) • each character pronounced as a syllable • Unknown number of Chinese words • one to several characters per word • character combinations create different meanings • ambiguity in word tokenization

  10. OOV and Acoustic Confusions in Mandarin SDR Query: …Iraq...

  11. Subwords for Retrieval Pros • Character n-grams • robust to word-level mismatches due to different tokenization • Syllable n-grams • robust to word/character-level mismatches due to homophones • Partial matches possible Con • Subwords contain reduced lexical knowledge c.f. words

  12. The MEI Investigation • Use of a multi-scale representation for crosslingual spoken document retrieval (CL-SDR) • Words and subwords Research Challenges • Multi-scale query translation • Multi-scale audio indexing • Multi-scale retrieval

  13. President Bill Clinton and Chinese President Jiang Zemin engaged in a spirited, televised debate Saturday over human rights and the Tiananmen Square crackdown, and announced a string of agreements on arms control, energy and environmental matters. There were no announced breakthroughs on American human rights concerns, including Tibet, but both leaders accentuated the positive … Query by Example English Newswire Exemplars Mandarin Audio Stories 美国总统克林顿的助手赞扬中国官员允许电视现场直播克林顿和江泽民在首脑会晤后举行 的联合记者招待会。。特别是一九八九镇压民主运动的决定。他表示镇压天安门民主运动 是错误的,他还批评了中国对西藏精神领袖达国家安全事务助理伯格表示,这次直播让中国 人第一次在种公开的论坛上听到围绕敏感的人权问题的讨论。在记者招待会上 …

  14. Evaluation Collection Development Collection: TDT-2 Evaluation Collection: TDT-3 Jan 98 Jun 98 Oct 98 Dec 98 17 topics, variable number of exemplars 56 topics, variable number of exemplars English text topic exemplars: Associated Press New York Times 2265 manually segmented stories 3371 manually segmented stories Mandarin audio broadcast news: Voice of America Mar 98 Exhaustive relevance assessment based on event overlap

  15. Abstract Task Model American English Text Exemplar Mandarin Chinese Broadcast News Cross-Language Speech Retrieval Ranked List of News Stories

  16. Evaluation of Ranked Lists Relevance Judgments VOA 0427.22 Relevant VOA 0521.14 Not VOA 0604.39 Not VOA 0419.12 Relevant VOA 0513.17 Relevant VOA 0527.13 Not … …

  17. Recall-Precision Graph

  18. Variation Across Exemplars

  19. 0.353 Average Across Exemplars

  20. Variation Across Topics 1.0 0.8 0.6 Mean Uninterpolated Average Precision 0.4 0.2 0.0 Topic

  21. Comparing Two Systems Topic

  22. Significance Testing • Statistical significance • Null hypothesis: mean average precision across topics is drawn from same distribution • Paired 2-tailed t-test, significant if p<0.05 • For System A vs. System B, p=0.94 • Meaningful differences • Rule of thumb: 5-10% relative • For System A vs. System B, relative difference is <1%

  23. Translingual and Multi-ScaleQuery Processing

  24. Bilingual Term List Relevance Judgments English Exemplar LDC 000100010000010100 President Bill Clinton and… LDC Named Entity Tagging Term Selection Term Translation Query Construction BBN Ranked List Mandarin IR System U Mass Evaluation Mandarin Audio Speech Recognition Document Construction Cornell Mean Uninterpolated Average Precision LDC Dragon Story Boundaries LDC

  25. Multi-Scale Query Translation • Words and Phrases (Gina, Sanjeev) • Subwords (Helen, Wai-Kit, Berlin, Karen)

  26. Bilingual Term List • Combination of • LDC English-Chinese bilingual term list • Chinese-English Translation Assistance File (CETA) [inverted] Total English Terms Total Translation Pairs Phrasal Terms Phrasal Translation Pairs 199,444 395,216 81,127 105,750 Term human right(s) human rights # translations 7 30 1

  27. Query Term Selection • Tagged named entities (BBN Identifinder) • Person: partners of Goldman, Sachs, & Co. • Organization: UN Security Council • Dictionary-based “phrases” • translatable multi-word units, e.g • “Wall Street”, “best interests”, “guiding principles”, “human rights” • automatic tagging: greedy, left-to-right, max match • Chi-squared filtering • Compared to English background model

  28. Query Term Translation • Named entities • if absent from dictionary, translate individual terms • e.g. “Security Council” versus “First Bank of Siam” • Numeric Expressions • special processing for digits • e.g. “12:30 pm, June 15, 1969” • Remaining terms • Consult bilingual term list, lemmatize if necessary • e.g. “televised” translates as “television”

  29. Query Construction • Unbalanced queries • Use all plausible translations for each term • Balanced queries • Pseudo-term weight: average of translations’ weights • Structured queries • Recompute pseudo-term weight from translations’ term frequency and document frequency

  30. Strategies in Query Translation • Phrase based translation is significantly better • Named entities and numeral translations are (barely) helpful • Balanced translation matches Structured queries • also extends easily to subword units

  31. Terms total OOV # (by token) 87,004 3,028 # (by type) 12,402 1,122 Untranslatable Terms suharto 97 (# of occurrences) netanyahu 88 starr 62 arafat 50 bjp 45 vajpayee 44 estrada 44 …. hsu 19 zemin 7

  32. Subword Transliteration • Kosovo (/ke1-suo3-wo4/, /ke1-suo3-fo2/, • /ke1-suo3-fu1, /ke1-suo3-fu2/) Mandarin Audio Document English Query Exemplar ……..Kosovo…... …../ke-suo-fo/…. Sound alike --> match in phonetic space?

  33. Named Entities Syllables, e.g. wang jian qiang wang xin min PinYin / WadeGiles Spellings e.g. Wang Jianqiang, Wang Hsinmin Trans. Error-Driven Learning [Brill 1994] PRONLEX, 85K(train), 4.5K (test) 82%(phoneme), 45% (word) • Acquire English Pronunciation • PRONLEX Lookup • Spelling-to-Pron Generation • e.g. christopher English Phones, e.g. /kk rr ih ss tt aa ff er/ Subword Transliteration Procedure (1)

  34. /kk rr ih ss tt aa ff er/ • Cross-lingual Phonological Rules • Syllable nuclei insertion Handle consonant clusters Word-final consonants, etc. /kk ax rr ih ss ax tt aa ff er/ Cross-lingual Phonetic Mapping English phones to Chinese “phones” • Trans. Error-Driven Learning • 4800 words (train) • [Chen H. H., NTU; WWW] • FST aligns Eng / Chin phones • /k e l i s i t uo f u/ Chinese phone lattice generation Syllable bigram language model N-best syllable sequence hyp N=1 (one-best hypothesis) /ji li si te fu/ (hyp) /ke li si tuo fu/ (ref) Subword Transliteration Procedure (2)

  35. Cross Lingual Phonetic Matching • Documents are indexed with syllable bigrams (in addition to words and character bigrams if necessary) • Query terms are translated as words where possible, phonetically where necessary

  36. Multi-Scale Query Construction Helen

  37. Multi-Scale Query Construction:Objectives Bag of English query terms (selected) Multi-scale query representation in Chinese Query Construction • Multi-scale representation integrates: • translated phrases, named entities, numeric expressions, translated terms • transliterated syllables • words, characters and syllable n-grams

  38. English Bag of Terms Israeli<Ph>Prime Minister</Ph><NE>Benjamin Netanyahu</NE> Chinese Translations and Transliteration ne-tan tan-ya ya-hu Character bigrams and Transliterations ne-tan tan-ya ya-hu Multi-Scale Query ConstructionProcedures words + syl bigrams char + syl bigrams Syllable bigrams and Transliterations yi-se se-lieshou-xiangben-jie jie-mingne-tan tan-ya ya-hu syl bigrams

  39. Multi-Scale Audio Document Indexing Hsin-min, Helen, Berlin, and Wai-kit

  40. Previous Chinese Example

  41. Audio Document IndexingObjectives • Augment words with subword-based indexing • Dragon word recognition outputs are provided • Character-based indexing • Characters derived from Dragon’s recognized words • Syllable-based indexing • Syllables derived by pronunciation lookup using Dragon’s recognized words • Address Dragon’s ASR errors • Augment with alternative (word/char/syl) hypotheses e.g. syllable lattice [Chen & Wang, ICASSP-2000]

  42. Dragon’s syl Alternative syl • Develop a syllable recognizer to produce • lattice representation Syllable Lattice Development • Dragon’s recognition accuracies • Evaluated against anchor scripts • 82.0%(word) 87.9%(char) 92.1%(syl) • Syllable substitution errors (5.2%) • MEI’s syllable recognition accuracy • Trained on Hub4 Mandarin (VOA, 11 hours, 1997) • 70.2% (syl)  !!!

  43. Strategy • Improve MEI’s syllable recognizer • Design a structure for document indexing which incorporates • Dragon’s word / character / syllable hypotheses • MEI’s syllable hypotheses (hopefully complementary to Dragon’s syllables)

  44. MEI Syllable Recognizer:Improve Acoustic Models • Forced alignment with Dragon’s output for each document • Blind speaker adaptation with Dragon’s syllables • MEI syllable accuracy: 70.2%(original)87.7% !!! VOA Audio for Doc i Forced Alignment Speaker Adaptation Dragon Outputs for Doc i Speaker-Adapted Acoustic Models Baseline Acoustic Models Syllable Recognition MEI Syllables for Doc i

  45. VOA Audio for Doc i Forced Alignment Speaker Adaptation Dragon Outputs for Doc i Speaker-Adapted Acoustic Models Baseline Acoustic Models Syllable Recognition 1998 Xinhua Language Models MEI Syllables for Doc i • Syllable trigram language model • MEI syllable accuracy: 70.2%87.7%90.0% !!! MEI Syllable Recognizer:Incorporate Language Model

  46. Dragon’s syl MEI’s syl Two separate recognition outputs Dragon’s syl MEI’s syl Audio Document Indexing withMultiple Syllable Recognition Outputs The revised syllable lattice

  47. Dragon’s syl MEI’s syl Dragon’s chr Dragon’s word Multi-scale Audio Document Indexing

  48. Fusion of Words and Subwordsin Multi-Scale Retrieval Wai-Kit Lo, Pat Schone

  49. Loose Coupling • Merging ranked lists from separate runs • For each query and document pair, the score is recalculated as • wk are the weights for different retrieval runs • K denotes a retrieval run at some scale (word, characters, syllables, combinations) • Sk (Qi, Dj) is a rank-based score between query i and document j in retrieval run k

  50. fusion Loose Coupling Word Word Word Char2 Char2 Syl2

More Related