Language and Speech Processing at Johns Hopkins University

1. Language and Speech Processing at Johns Hopkins University March 5, 2010 Center for Language and Speech Processing The Johns Hopkins University

2. The JHU Center for Language and Speech Processing CLSP was established in 1992 with outside support to promote research and education in the science and technology of speech and language. Electrical and Computer Engineering Cognitive Science Computer Science CLSP Biomedical Engineering Applied Math & Statistics Human Language Tech.Center of Excellence

3. Electrical & Computer Eng Andreas Andreou Mounya Elhilali Hynek Hermansky Frederick Jelinek (Director) Damianos Karakos Sanjeev Khudanpur Computer Science Chris Callison-Burch Jason Eisner David Yarowsky Applied Math & Statistics Carey Priebe Cognitive Science / Psychology Justin Halberda Geraldine Legendre Kyle Rawlins Paul Smolensky (Asst Dir) Colin Wilson Biomedical Engineering Eric Young Applied Physics Laboratory James Mayfield Christine Piatko HLT Center of Excellence Kenneth Church Mark Dredze` Speech and Language Faculty at JHU(does not list senior research staff, postdocs, students, …) • Aren Jansen • Ben Van Durmé

4. CLSP Vision Statement • Understand how human language is used to communicate ideas/thoughts/information. • Develop technology for machine analysis, translation, and transformation of multilingual speech and text.

5. CLSP Mission Statement • Research • Advance state of the art in our interdisciplinary field • Focus on developing key algorithms and statistical models • Focus on strategic languages, including low-resource languages • Education • Attract the best students and train them to be leaders • Offer full spectrum of courses • Conduct annual international summer school at JHU • Outreach • Be responsive to government and industry problems • Serve as a “hub” for the HLT community • Organize international summer research workshop at JHU • Welcome short- and long-term visitors

6. Speech Recognition Acoustic processing Acoustic-phonetic modeling Pronunciation modeling Language modeling Speech Applications Keyword spotting Spoken term detection Speaker verification Language identification Speech Science Auditory physiology Neuromorphic signal processing Natural Language Processing Morphological analysis Syntactic analysis (parsing) Information extraction Co-reference resolution Machine Translation Low-resource languages Arabic and Chinese Knowledge-Base Population Automatic content extraction Inference and learning Machine Learning Small-sample learning Structured prediction Minimally supervised learning Research: Primary Areas

7. Sponsored Research in Speech & Language in WSE is ≈ $2.5M/year

8. CLSP Mission Statement • Research • Advance state of the art in our interdisciplinary field • Focus on developing key algorithms and statistical models • Focus on strategic languages, including low-resource languages • Education • Attract the best students and train them to be leaders • Offer full spectrum of courses • Conduct annual international summer school at JHU • Outreach • Be responsive to government and industry problems • Serve as a “hub” for the HLT community • Organize international summer research workshop at JHU • Welcome short- and long-term visitors

9. Education: Interdisciplinary Environment • Who and where • PhD, MSE, and BS students from multiple depts. • Shared interdisciplinary offices in CSEB • Shared technical perspective and computing infrastructure • Coursework • Interdisciplinary core curriculum (extends dept. requirements) • Variety of other relevant courses (growing list, new plans) • International 2-week summer school • Research • Students do research from the start • Students work with faculty from multiple departments and HLTCOE • Other learning • Distinguished outside speaker every week • Student speaker and town meeting every week • Reading groups and conference travel

10. Sample Courses for an MSE in Human Language Technology

11. WSE has the 2nd largest university group in the U.S. working on Human Language Technology 38 PhDs awarded, many more MSEs CLSP PhDs presently hold research/faculty positions at Carnegie Mellon University U. of Massachusetts, Amherst Swarthmore College Michigan State University Hong Kong Polytechnic Univ. Bogazici University (Turkey) U. of Karlsruhe (Germany) Saarland University (Germany) CLSP PhDs presently hold senior technical/research positions at Apptek BBN Convergys e-Scription Fair Isaac Google (several) Microsoft (several) MITRE IBM (several) NSA (several) Nuance (several) SRI International Education: Track Record

12. CLSP Mission Statement • Research • Advance state of the art in our interdisciplinary field • Focus on developing key algorithms and statistical models • Focus on strategic languages, including low-resource languages • Education • Attract the best students and train them to be leaders • Offer full spectrum of courses • Conduct annual international summer school at JHU • Outreach • Serve as a “hub” for the HLT community • Be responsive to government and industry problems • Organize international summer research workshop at JHU • Welcome short- and long-term visitors

13. JHU Summer Workshops in HLT:Integrating Research and Education • Organized by JHU on behalf of the Human Language Technology field • 3 teams per summer (since 1995) • selected & refined from  25 proposals by “interactive peer review” • each team comes to JHU for 8 weeks of intense collaborative research • Mixed teams of senior and student researchers • Team ≈ 3 academics, 1 industry, 1 govt, 2-3 grad students, 2 undergrads • 30+ participants  8 weeks  15 years • More than 160 star students trained in HLT research (1998—2007) • Outcomes • Numerous research breakthroughs • New, long-term collaborations, tangible knowledge transfer • Diverse expertise, research infrastructure, data resources

14. A Few of ManyWorkshop Accomplishments • A small sample of research results and their wider impact • Statistical Machine Translation (1999) • GIZA++ is extensively used to build SMT systems even today • MEAD Multilingual Multi-document Summarization (2001) • 100s of worldwide users, active developers in the community • SuperSID: High-level information for Speaker-ID (2002) • Major breakthrough in speaker recognition technology • Factored Language Models (2002) • Improved ASR technology for conversational Arabic • Moses Machine Translation Repository (2006) • The de facto standard in statistical machine translation • More than 100 refereed publications • Detailed technical reports also available on CLSP web-site

15. Human Language TechnologyCenter of Excellence at JHU • Long-term research mission: Automatically analyze a wide range of speech, text, and document images in multiple languages. • Founded with government support in 2007 • Has brought many new researchers and research challenges into the CLSP community • Aggressively hiring the top new Ph.D.s nationally

16. Human Language TechnologyCenter of Excellence at JHU Whiting School of Engineering Sponsor RD Leadership JHU Provost Executive Director Sponsor Technical Board Administrative Staff Security Staff Director of Research Researchers Sponsor Researchers Center for Language and Speech Processing

17. Prof. Andreas G. Andreou: Sensory Information Processing in Natural and Synthetic Systems Applications Algorithms for robust ASR • Robust acoustic feature representation and dimensionality reduction • Algorithms and architecture optimization for Chip Multi Processors (CMP) in Exascale systems Multimodal scene analysis • Active and passive processing for scene analysis (visual & auditory) • Acoustic and EM micro-Doppler imaging Bio-inspired systems • Energy efficient microsystems for processing what and where in natural environments. • Research • Principles of sensory information processing in biology. • Sensory communication. • Algorithms and processor architecture design for energy efficient acoustic, speech and vision processing. • Physics of sensing and computation.

18. Prof. Chris Callison-Burch: Statistical Machine Translation Research Statistical machine translation Syntactic translation models Low resource languages Data-driven paraphrasing Evaluation measures, creation of shared data resources

19. Prof. Kenneth Church:Human Language Technology (HLT) at Scale Applications Web search Cloud computing Language modeling Text analysis Spelling correction Word-sense disambiguation Terminology Translation Lexicography Compression Speech recognition and synthesis OCR Research Speech Processing at Scale Language Processing at Scale Web Search at Scale Mining Speech/Language with Zero Linguistic Resources

20. Prof. Mark Dredze: Applications of Machine Learning to Real-World Text Processing Applications Domain adaptation Extending NLP models to new datasets Cross-domain learning Applying NLP techniques to languages with few resources Knowledge base population Building large high precision knowledge bases from text Intelligent email Improved email clients by aiding the user with artificial intelligence Research Adaptation of machine learning algorithms between text domains Large scale information processing and learning Intelligent user interfaces for information management

21. Prof. Jason Eisner: Algorithms and Models for Language Processing Applications Parsing sentence structure Faster and more accurate algorithms Unsupervised or cross-lingual learning Machine translation Model syntax, structure, word order Combinatorial methods for translation and for training models Morphology / phonology Word spelling and pronunciation Variant word forms (conjugation, transliteration, misspelling, …) Information integration Truth maintenance Deductive databases Reasoning from facts in text Research Novel algorithms for NLP Bayesian statistical models of linguistic structure Machine learning (structured prediction, novel training objectives) Declarative formalisms for grammars and algorithms

22. Prof. Mounya Elhilali: Reverse Engineering the Neurobiology of Speech and Audio Processing • Applications • Speech intelligibility in noise and distortions • Auditory scene analysis and speaker segregation • Speech enhancement • Hearing prostheses • Adaptive audio systems • Robotics and autonomous systems • Object tracking in sensor networks • Communication channels • Microphone Design • Research Goals • Information representation and computational strategies employed by the brain • Sound perception in distorted or complex acoustic environments

23. Prof. Hynek Hermansky: Robust Acoustic Speech Processing Applications Speech recognition • what has been said? Speaker identification • who is speaking? Speaker verification • is the talker the one claimed to be? Language identification • which language is being used? Speech and audio coding • how to store/transmit the signal efficiently? Enhancement of degraded speech • how to make noise or reverberated speech easier listening to? Technology • Proprietary techniques based on temporal cues in the signal and on artificial neural net post-processing • Emulations of auditory processing in biology

24. Prof. Aren Jansen:Knowledge-based Approaches to Speech Processing Applications Noise-Robust Speech Recognition Invariance and efficiency through sparsity Low-Resource Speech Recognition What can be done with little or no transcribed training data? Spoken Term Detection and Discovery “Google” for speech documents Query-by-example vs. text queries Large-Scale Speech Processing Scaling speech technology to massive problem sizes Research Pursuit of more invariant representations of speech Unsupervised/semi-supervised learning of speech units Sparse representations and models Computational models of human speech perception

25. Prof. Frederick Jelinek:Statistical Speech Recognition and Machine Translation Research Statistical aspects of Automatic Speech Recognition (ASR) Language Modeling • Predicting next word given the past Reconstruction of ASR output • Create a grammatical sentence preserving the speaker’s intended meaning Rescoring of ASR output alternatives Search algorithms for ASR and Machine Translation Interests • Statistical grammar and parsing • Signals and systems • ASR treatment of out-of-vocabulary words and phrases • Machine translation

26. Prof. Damianos Karakos: Statistical Aspects of Speech and Language Applications Speech recognition Adaptation to the speech topic Error corrective techniques Machine translation System combination Language modeling Document categorization Automatic clustering into meaningful categories Detection of topics of interest Technology Data fusion and dimensionality reduction for improved inference in text classification. Novel language modeling techniques for speech recognition.

27. Prof. Sanjeev Khudanpur:Statistical Modeling for Information Processing Applications Automatic speech recognition • Domain and genre adaptation • Pronunciation variability modeling Machine translation (text & speech) • Output language word ordering • Context dependent translation Multimedia search and retrieval • Searching large speech archives • Content-based image/video search Robotic minimally invasive surgery • Automated skill assessment • Automated surgical training Basic Research • Stochastic Modeling of Signals and Systems • Parameter Estimation • Model Structure Estimation • Information Theory and Statistics

28. Prof. Benjamin Van Durme: Computational Semantics and Large-Scale Text Processing Applications Knowledge Acquisition Enable “everyday” reasoning Formal interpretation of generic sentences (e.g., dictionary definitions) “Deep” Information Extraction Infer implicit relations Semantic language modeling Recognize higher order modification of factoids Organizing Social Media Dynamic clustering of authors, documents, feeds Research Application of theoretical semantics to problems in language technology Streaming algorithms for efficient processing of large text collections

29. Prof. David Yarowsky:Minimally Supervised Learning for Low-Resource Languages Applications Machine Translation • Translation discovery without aligned bilingual text • Exploiting language universals and language family relationships Natural Language Processing • Word sense disambiguation • Inflectional and derivational morphology Information Extraction • Biographic fact extraction • Characterizing communicants • Informal genres Basic Research • Cross-language information projection • Cross-domain knowledge transfer • Co-training • Active learning and human computation • Creative bootstrapping from multiple knowledge sources

30. Linguistics and Human Language Processing Prof. Paul Smolensky: Architecture of Universal Grammar Prof. Colin Wilson: Theoretical, Experimental, & Computational Phonology Prof. Geraldine Legendre: Syntax, Morphology, Acquisition Prof. Kyle Rawlins: Formal & Computational Semantics Prof. Justin Halberda: Word Learning in Children + a new professor …Human Sentence Processing

31. Lots of Great Ph.D. Students:The Next Big Things! Welcome!