Abstract

1. Speech recognition software is maturing, and possesses the potential to provide real-time note taking assistance in the classroom, particularly for deaf and hard of hearing students. This research talks about speech recognition in general, and reports on a practical, portable and readily deployed application that provides a cost-effective, automatic transcription system with the goal of making computer science lectures inclusive of deaf and hard of hearing students. The design of the system is described, some specific technology choices and implementation approaches are discussed, and results of two phases of an in-class evaluation of the system are analyzed. Ideas for student research projects that could extend and enhance the system also are proposed. Design of a Speech Recognition System to Assist Hearing Impaired StudentsRichard Kheir2 and Thomas P. WayDepartment of Computing Sciences, Villanova University Abstract • Applications • Four general application categories for ASR are: • Command Recognition • Dictation • Interactive Voice Response (IVR) • Assistive Technologies • System Design Part 2 - VUST • Table. Recognition accuracy for 4 classifications of classroom speech. • Classification Words Correct Total Words Percent Recognized • Planning 628 758 83% • Lecture 5930 6925 86% • Roll-call 155 254 61% • Discussion 1556 1846 84% • TOTAL 8269 9783 85% • Consists of three major components: the speech recognition software, a dictionary enhancement tool, and a transcription distribution application. • Uses an ASR system designed to be affordable, accurate and easy to set up and use. • Around one hour of speech training are enough to get good accuracy • Training through windows control panel or through the VUST instructor’s Console • Simple setup and configuration. • User friendly interface • Instructor initiates transcription • Students connect via web applet • Accurate results even without added jargon (table below) Background Motivation • Note takers and sign language interpreters are expensive to hire and provide limited assistance due to the need to paraphrase during a lecture • Developing countries provide no assistance • Commercial ASR systems are expensive to acquire • 28 million deaf and hard of hearing individuals in the US (Around 500 million world wide) • Limited benefit from hearing aids and cochlear implants as these are most useful in face to face conversations The quest for automatic speech recognition (ASR) startedin 1939 with the introduction of VODER by AT&T. System Design Part 1 - DiBS Low recognition rate for domain specific jargon is one of the keyweaknesses in ASR. DiBS was developed to solve this problem. Table: Summary of the accuracy results for five scenarios. Description Accuracy Range Usability Untrained 75% 64%-83% Poor to fair Minimal Training 88% 78%-93% Sufficient Moderate Training 90% 81%-96% Good Moderate Training and Customized dictionary 91% 83%-96% Good Moderate Training, Customized Dictionary and pronunciations 94% 86%-98% Very good Many enhancements took place on specific domains during the following years such as the introduction of the Hidden Markov Model (HMM). At the beginning of the 21st century, commercial speech recognition systems finally became practical and affordable, with many products on the market. The most popular vendors being IBM and Dragon. We have tested the ASR system with five scenarios: Untrained, some training, moderate training, moderate training and some added jargon using DiBS and moderate training with added jargon and custom pronunciation for the added jargon. SERVER With the now wide availability of ASR software, the technology has become an application area that is emerging in assistive technology. For people who are deaf and hard of hearing, the accessibility and freedom that can be afforded by using a computer to recognize speech is finally beginning to be realized. The design of such a truly usable ASR system requires an understanding of the approaches, user requirements, and available technology. Contributions & Future Work • Contributions • Proved to be an affordable and beneficial assistive system • Provides an easy to use software • Improves Recognition Accuracy • Distributed and portable application • Future work • Commercial Quality • Post speech profiles and jargon in a central repository • Evaluate other speech engines • Deploy in classrooms 2 1 1 2 Run the VUST program and selects a speech profile. Select available connection, and click “Connect”. Lecturer Student Nady UHF-3 wireless headset system 3 …click ‘Connect and Start Recognition’ to start VUST server. 3 Connect to VUST transcription server URL using web browser. Transcription is received once the lecture begins. 1 Connect wireless microphone receiver to computer and wear headset & transmitter.