Is Automatic Speech Recognition Ready for Direct Use by Classroom Teachers?

1. Is Automatic Speech Recognition Ready for Direct Use by Classroom Teachers? PEPNet 2004 - Presentation Pittsburgh, PA, Sheraton Station Square, April 24, 2004, 10:15 – 11:30 AM Presenter/Author: Kathleen Eilers Crandall, Ph.D. Contributors: Donna E. Gustina, and Stephen S. Campbell National Technical Institute for the Deaf Rochester Institute of Technology

2. The Glossograph • Fay wrote about an experimental mechanical device used to transcribe human speech, and said, • “… it is not unreasonable to hope that some instrument will yet be contrived …“ Fay, E.A. (1883). The glossograph. American Annals of the Deaf, 28, 67-69.

3. Sci-Fi or Reality? "The pen was an archaic instrument, seldom used even for signatures...Apart from very short notes, it was usual to dictate everything into the speak-write…” (Nineteen eighty-four.Orwell, 1949)

4. Project • Direct teacher use of Continuous Automatic Speech Recognition: • English Classroom/Lab Funded by a grant from the Parsons Foundation of California

5. English Classroom/Lab Project

6. English Classroom/Lab ProjectPurpose Investigate direct use of ASR by classroom teacher to learn: • Is acceptable recognition level attained? • Under what conditions? • Style of speaking • Communication mode • Language complexity

7. Use of ASR by an intermediary Intermediary, a ‘captionist,’ re-speaks professor’s words into a computer Intermediary summarizes professor’s words into a computer (‘interpreted speech’) Intermediary may use C-print (a shorthand typing system) in combination with ASR http://cprint.rit.edu/ Related Work

8. Related Work Use of ASR by the primary speaker • iCommunicator™ http://www.myicommunicator.com/product_info.html • Liberated Learning Environment http://www.liberatedlearning.com (St. Mary’s University, Halifax, Nova Scotia)

9. English Classroom/Lab Project

10. English Classroom/Lab ProjectTeacher -- Students • Teacher -- Speaker • Native speaker of American English • User of ASL as a second language • Trained the ASR equipment • Students -- Readers • Young adult college students who are deaf or hard-of-hearing • Reading and writing skills at the lowest quartile of entering students • Enrolled in basic level English language reading and writing courses

11. English Classroom/Lab ProjectEvaluation Procedures • ASR Software: • Dragon Naturally Speaking • IBM ViaVoice • Microsoft Office • Speaking styles: • Spontaneous conversation • Dictation-like speech • Communication modes: • Speaking • Simultaneously speaking and signing

12. English Classroom/Lab Teacher stationControl systemSmart Board & LCD Projector Student Stations

13. Vary by population and message predictability New vs. Known information Fluent readers vs. Language learners Reading for pleasure vs. Reading to master new information CLOZE research and prediction of missing information English Classroom/Lab ProjectAccuracy Needs

14. English Classroom/Lab ProjectResults: ASR Software

15. English Classroom/Lab ProjectResults: Communication Mode

16. English Classroom/Lab ProjectResults: Language Complexity

17. English Classroom/Lab ProjectCorrecting Text • Error correction • What to correct • When to correct • How to correct

18. Multitasking Demands • Normal tasks for speaker/teacher • Formulating ideas relevant to topic • Attending to learning needs of students • Meeting lipreading and sign language needs • Added tasks for speaker/teacher • Speaking to produce readable ASR text • Monitoring text • Making corrections

19. RecommendationsDiscussionQuestions

20. Grammatical Correctness • Is ASR accuracy affected by the grammatical correctness of the user’s speech? • Student written responses spoken as written: Accuracy – 93.8% • Student written responses spoken after corrected: Accuracy - 94.3%

21. Style of Speaking • Style of speaking that more closely resembles dictation approaches a usable accuracy rate. • Lowering the complexity does not improve accuracy.

22. Conditions of Use Direct use of ASR by a language teacher -- Useful only under very controlled conditions. • Illustrating the generation of written language • Demonstrating the use of notes and outlines to produce written text • Translating selected sign language utterances into English text during discussions

23. ASR: Classroom Use Prepared Outline Teacher’s Screen Student’s Screen

24. Considerations • Training • Critical to reach over 90% accuracy • Training with conversation • Corrections • Familiarity with strategies • Dictate, Spell, Right click • Equipment • Microphone headsets - design, comfort, and size • Demand on computer processor • Effect of optional settings

25. Tips for Better Accuracy • Powerful computer • No other programs running • Consistent microphone placement • Environment • Training • Profile • Join user groups, such as ms-speech@yahoogroups.com

26. Teaching/Learning Issues: Does ASR promote the learning of reading and writing for Deaf and Hard-of-Hearing students? How do students process this information? Do students attend to multiple inputs? Can teachers attend to this many tasks effectively? Language Processing

27. Presenter – Contact Information Kathleen Eilers-crandall, Ph.D. Department of English National Technical Institute for the Deaf Rochester Institute of Technology Lyndon Baines Johnson Building - 2264 Phone: (585) 475-5111 Fax: (585) 475-6500 Email: kecncp@rit.edu Web: http://www.rit.edu/~kecncp