HUMAN AND SYSTEMS ENGINEERING:

1. HUMAN AND SYSTEMS ENGINEERING: Bridging the Gap in Human and Machine Performance Joseph Picone, PhD Human and Systems Engineering Professor, Electrical and Computer Engineering URL: http://www.isip.msstate.edu/publications/seminars/external/2004/dod/

2. 10 years at MS State • Public Domain Speech Recognition • Jumpstarted in 1997 by a DoD grant • Center for Advanced Vehicular Systems • State funded to support Nissan • Three Complementary Thrusts • Extension center colocated with Nissan in Canton, Mississippi • Statewide economic development • Assist first-tier suppliers • Evolution For Better Infrastructure Page 1 of 7 Introduction to Human and Systems Engineering

3. A Virtual Tour of CAVS at Mississippi State University Page 2 of 7 Introduction to Human and Systems Engineering

4. Intelligent Electronic Systems At A Glance • Computer Networking: • Wireless Communications • Intelligent Sensors • Collaborative Vehicles • Intelligent Systems: • Speech Processing • Machine Learning • Dialog Systems • Human Factors and Ergonomics • Integrative Activities: • Challenge X • Capstone Design Experiences Page 3 of 7 Introduction to Human and Systems Engineering

5. Instrument the campus bus system to collect real-time data • Modular architecture to support a variety of sensors and high speed data communications • Phase I Testbed: Campus Bus Networking Page 4 of 7 Introduction to Human and Systems Engineering

6. In-vehicle dialog systems improve information access. • Advanced user interfaces enhance workforce training and increase manufacturing efficiency. • Noise robustness in both environments to improve recognition performance • Advanced statistical models and machine learning technology • Dialog Systems Applications in Automotive Page 5 of 7 Introduction to Human and Systems Engineering

7. Speaker Verification Via Metadata Extraction • Recognition of emotion, stress, fatigue, and other voice qualities are possible from enhanced descriptions of the speech signal • Fundamentally the same statistical modeling problem as other speech applications • Fatigue analysis from voice under development under an SBIR (from Shriberg, et al., IEEE Spectrum, April 2003) Page 6 of 7 Introduction to Human and Systems Engineering

8. The Challenge X Program • • Competition created by automotive industry, government, and academic partners • Challenges university-level engineering students to decrease total energy consumption and emissions • Maintain or exceed vehicle utility and performance • Cooperative venture between industry and universities • Faculty Advisor:G. Marshall Molen Page 7 of 7 Introduction to Human and Systems Engineering

9. APPLICATIONS OF RISK MINIMIZATIONTO SPEECH RECOGNITION Jon Hamaker, Aravind Ganapathiraju and Joseph Picone Intelligent Electronic Systems Human and Systems Engineering URL: http://www.isip.msstate.edu/publications/seminars/external/2004/dod/

10. Abstract and Biography ABSTRACT: Statistical techniques based on Hidden Markov models (HMMs) with Gaussian emission densities have dominated the signal processing and pattern recognition literature for the past 20 years. However, HMMs suffer from an inability to learn discriminative information and are prone to overfitting and over‑parameterization. In this presentation, we will review our attempts to apply notions of risk minimization into pattern recognition problems such as speech recognition. New approaches based on probabilistic Bayesian learning are shown to provide an order of magnitude reduction in complexity over comparable approaches based on HMMs and Support Vector Machines. BIOGRAPHY: Joseph Picone is currently a Professor in the Department of Electrical and Computer Engineering at Mississippi State University and an Academic Thrust Leader at the Center for Advanced Vehicular Systems. For the past 15 years he has been promoting open source speech technology. He has previously been employed by Texas Instruments and AT&T Bell Laboratories. Dr. Picone received his Ph.D. in Electrical Engineering from Illinois Institute of Technology in 1983. He is a Senior Member of the IEEE and a registered Professional Engineer. Page 1 of 10 Applications of Risk Minimization

11. Optimal decision surface is obviously a line • Introduce two more data points • Optimal decision surface changes abruptly • Generalization and Risk • How much can we trust isolated data points? • Optimal decision surface is still a line (good generalizaton) • Can we integrate prior knowledge about data, confidence, or willingness to take risk? Page 2 of 10 Applications of Risk Minimization

12. Deterding Vowel Data: 11 vowels spoken in “h*d” context; 10 log area parameters; 528 train, 462 SI test • Static Pattern Classification With SVMs Page 3 of 10 Applications of Risk Minimization

13. Applications of SVMs to Conversational Speech • Notes: • SVMs not exposed to alternative segmentations during training (closed-loop) • SVM performance is high when there is no mismatch between the training and evaluation conditions • Complexity (parameter count) approaches HMMs Page 4 of 10 Applications of Risk Minimization

14. A kernel-based learning machine • Incorporates an automatic relevance determination (ARD) prior over each weight (MacKay) • A flat (non-informative) prior over a completes the Bayesian specification • Relevance Vector Machines Page 5 of 10 Applications of Risk Minimization

15. The goal in training becomes finding: • Estimation of the “sparsity” parameters is inherent in the optimization – no need for a held-out set! • A closed-form solution to this maximization problem is not available. Iteratively reestimate • Iterative Reestimation of Hyperparameters Page 6 of 10 Applications of Risk Minimization

16. Deterding Vowel Data: 11 vowels spoken in “h*d” context; 10 log area parameters; 528 train, 462 SI test • RVM and SVM Comparison — Static Patterns Page 7 of 10 Applications of Risk Minimization

17. RVMs yield a large reduction in the parameter count while attaining superior performance • Computational costs mainly in training for RVMs but is still prohibitive for larger sets – O(N3) vs. O(N2) for SVMs and O(N) for HMMs • RVM and SVM Comparison — Alphadigits Page 8 of 10 Applications of Risk Minimization

18. Preliminary Results on Learning • Data increased to 10,000 training vectors • Reduction method has been trained up to 100k vectors (on toy task). Not possible for Constructive method Page 9 of 10 Applications of Risk Minimization

19. Summary — Practical Risk Minimization? • Reduction of complexity at the same level of performance is interesting: • Results hold across tasks • RVMs have been trained on 100,000 vectors • Results suggest integrated training is critical • Risk minimization provides a family of solutions: • Is there a better solution than minimum risk? • What is the impact on complexity and robustness? • Applications to other problems? • Speech/Non-speech classification? • Speaker adaptation? • Language modeling? Page 10 of 10 Applications of Risk Minimization

20. Brief Bibliography • Influential work: • M. Tipping, “Sparse Bayesian Learning and the Relevance Vector Machine,” Journal of Machine Learning, vol. 1, pp. 211-244, June 2001. • D. J. C. MacKay, “Probable networks and plausible predictions --- a review of practical Bayesian methods for supervised neural networks,” Network: Computation in Neural Systems, 6, pp. 469-505, 1995. • D. J. C. MacKay, Bayesian Methods for Adaptive Models, Ph. D. thesis, California Institute of Technology, Pasadena, California, USA, 1991. • E. T. Jaynes, “Bayesian Methods: General Background,” Maximum Entropy and Bayesian Methods in Applied Statistics, J. H. Justice, ed., pp. 1-25, Cambridge Univ. Press, Cambridge, UK, 1986. • V.N. Vapnik, Statistical Learning Theory, John Wiley, New York, NY, USA, 1998. • V.N. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, New York, NY, USA, 1995. • C.J.C. Burges, “A Tutorial on Support Vector Machines for Pattern Recognition,” AT&T Bell Laboratories, November 1999. • Applications to Speech Recognition: • J. Hamaker and J. Picone, “Advances in Speech Recognition Using Sparse Bayesian Methods,” submitted to the IEEE Transactions on Speech and Audio Processing, January 2003 (in revision). • A. Ganapathiraju, J. Hamaker and J. Picone, “Applications of Risk Minimization to Speech Recognition,” to appear in the IEEE Transactions on Signal Processing, August 2004. • J. Hamaker, J. Picone, and A. Ganapathiraju, “A Sparse Modeling Approach to Speech Recognition Based on Relevance Vector Machines,” Proceedings of the International Conference of Spoken Language Processing, vol. 2, pp. 1001-1004, Denver, Colorado, USA, September 2002. • J. Hamaker, Sparse Bayesian Methods for Continuous Speech Recognition, Ph.D. Dissertation, Department of Electrical and Computer Engineering, Mississippi State University, December 2003. • A. Ganapathiraju, Support Vector Machines for Speech Recognition, Ph.D. Dissertation, Department of Electrical and Computer Engineering, Mississippi State University, January 2002. Applications of Risk Minimization

21. Effects of Transcriptions Errors on Supervised Learning in Speech Recognition Ram Sundaram Joseph Picone BBN Technologies Mississippi State University Cambridge, Massachusetts Mississippi State, Mississippi URL: http://www.isip.msstate.edu/publications/seminars/external/2004/dod/

22. Abstract and Motivation ABSTRACT:Hidden Markov model-based speech recognition systems use supervised learning to train acoustic models. On difficult tasks such as conversational speech there has been concern over the impact erroneous transcriptions have on the parameter estimation process. This work analyzes the effects of mislabeled data on recognition accuracy. Training is performed using manually corrupted transcriptions, and results are presented on three tasks: TIDigits, Alphadigits and Switchboard. For Alphadigits, with 16% of the training data mislabeled, the performance of the system degrades by 12% relative to the baseline. On Switchboard, at 16% mislabeled training data, the performance of the system degrades by 8.5% relative to the baseline. An analysis of these results revealed that the Gaussian mixture model contributes significantly to the robustness of the supervised learning training process. MOTIVATION: Recover an investment of three and a half long years spent retranscribing and resegmenting Switchboard  Page 1 of 5 Transcription Errors

23. Robustness to Transcription Errors — TIDigits • Introduced random transcription word errors in a controlled fashion on TIDigits • Observed no significant degradation in performance until the TER was artificially high (16%). • What makes an HMM-based speech recognition system so robust to such errors? Page 2 of 5 Transcription Errors

24. Robustness to Transcription Errors — Comparison • No significant degradation with transcription errors (including Switchboard!) • Context-dependent phone models are more robust than word models Page 3 of 5 Transcription Errors

25. Study maximum likelihood estimates of the mean and variance for a Gaussian estimator • Analyze how much does an incorrect model learn from the erroneous data by examining state occupancies • Analyze how much the correct model is influenced by the erroneous transcriptions Analyze State Occupancies Through Training Page 4 of 5 Transcription Errors

26. Summary • Transcription errors do not corrupt the acoustic models significantly • Alphadigits — at 16% TER, WER degrades only by 12% • SWB — at 16% TER, WER degrades only by 8.5% • Robustness to erroneous data mainly due to Gaussian distribution • State-tying helps in decreasing the TER during the context-dependent modeling stage • Mixture training adds more robustness by modeling other variations in the correct portion of the data Page 5 of 5 Transcription Errors

27. Brief Bibliography • R. Sundaram and J. Picone, “Effects of Transcription Errors on Supervised Learning in Speech Recognition,” submitted to the International Conference on Acoustics, Speech, and Signal Processing, Montreal, Quebec, Canada, May 2004. • R. Sundaram, Effects of Transcription Errors on Supervised Learning in Speech Recognition, M.S. Thesis, Department of Electrical and Computer Engineering, Mississippi State University, August 2003. • R. Sundaram and J. Picone, “The Effects of Transcription Errors,” Proceedings of the Speech Transcription Workshop, Linthicum Heights, Maryland, USA, May 2001. • L. Lamel, J. L. Gauvain, G. Adda, “Lightly Supervised Acoustic Model Training,” Proceedings of the ISCA ITRW ASR2000, Paris, France, September 2001. • G. Zavaliagkos, T. Colthurst, “Utilizing Untranscribed Training Data to Improve Performance,” Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, Landsdowne, Virginia, February 1998. • P. Placeway, J. Lafferty, “Cheating with Imperfect Transcripts,” Proceedings of the International Conference on Speech and Language Processing, Philadelphia, Pennsylvania, USA, pp. 2115-2118, September 1996. • T. Kemp, A. Waibel, “Unsupervised Training of a Speech Recognizer Recent Experiments,” Proceedings of ESCA Eurospeech’99, pp. 2725-2728, Budapest, Hungary, September 1999. Transcription Errors