New technologies supporting people with severe speech disorders

1. New technologies supporting people with severe speech disorders Mark Hawley Barnsley District General Hospital and University of Sheffield

2. Research Projects • STARDUST (Speech Training and Recognition for Disabled Users of Assistive Technology) • DoH NEAT programme • OLP (Ortho Logo Paedia) • EU Vth Framework (Quality of life and management of living resources)

3. Research Team • Department of Medical Physics and Clinical Engineering, Barnsley District General Hospital(Mark Hawley, Simon Brownsell) • Institute of General Practice and Primary Care, University of Sheffield(Pam Enderby, Mark Parker, Rebecca Palmer) • Department of Computer Science, University of Sheffield(Phil Green, Nassos Hatzis, James Carmichael)

4. Dysarthria • A neurological motor speech impairment • characterised by slow, weak, imprecise and/or uncoordinated movements of the speech musculature. • Frequently associated with other physical disabilities • 170/100 000 (Emerson & Enderby 1995) • Speech is often difficult to understand (unintelligible) and variable (inconsistent) • Severe = <40% intelligible

5. Intelligibility and Consistency • ‘Normal’ speech will be almost 100% intelligible and with few articulatory differences over time (consistent). • ‘Severe’ dysarthria may be completely unintelligible to the naïve listener and will show high variability (inconsistent) • but may show consistency of key elements which will make it more intelligible to the familiar listener. • STARDUST is concerned with consistency • OLP with intelligibility

6. STARDUST • To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers

7. Speech recognition systems • Speech-input writing programmes (eg Dragon) • large vocabulary, speaker adaptive • Speech-input control systems • small vocabulary, speaker dependent • Environmental control systems • speech-in - speech-out • Mobile phones • Smart homes

8. Speech-input writing programmes • Normal speech - with recognition training can get >90% recognition rates (Rose and Galdo, 1999) • Dysarthric speech - mild 10-15% lower recognition rates (Ferrier, 1992) • Declining as speech deteriorates - by 30-50% for single words (Thomas-Stonell, 1998, Hawley 2002)

9. Performance of a commercial speech-recognition environmental controller(in ‘ideal’ conditions)

10. STARDUST • To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers

11. Recognition technology • Small vocabulary • Speaker dependent • uses hidden Markov models • based on HTK (University of Cambridge)

12. Recognition increases with amount of training data

13. STARDUST recogniser performance(N=number of words used for training)

14. STARDUST • To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers(ie improve consistency)

15. Training tool • Quantitative and qualitative real-time visual feedback to improve consistency • at word level • at sub-word level • Can be used by the client alone or with carer or therapist • Training tool records examples of words - used to build recogniser

16. Consistency measure word level

17. OLT visual feedback (sub-word)(Hatzis and Green 1999)

18. STARDUST - conclusions • Recogniser that recognises severely dysarthric speech - well but not perfectly • next step to test in real usage • Computer-based training program to improve consistency • word level and sub-word level in future • collects lots of speech data for recogniser • Develop demonstrators of environmental controller and speech-output device

19. OLP • To supplement speech therapy by providing computer-based tools for audio-visual feedback to improve clients’ speech production • intelligibility as well as consistency • To make this available remotely using distance learning techniques

20. Partners • Institute for Language and Speech Processing, Greece • University of Sheffield, UK • Royal Institute of Technology, Sweden • Polytechnic University of Madrid, Spain • ARCHES, France • Unisoft Software Applications SA, Greece • Logos Centre fo Speech-Voice Pathology, Greece • Barnsley District General Hospital, UK

21. Client groups • Dysarthria • Pre-lingual and severe hearing impairment • Cleft lip and palate and velopharyngeal incompetence

22. Changing speech patterns • Accurate and consistent feedback • Provide target speech patterns and feed back deviations • Repeated practice • Drawbacks of conventional therapy • feedback may become inconsistent • lack of time leads to lack of practice

23. OLT visual feedback (sub-word)

24. Current systems • IBM Speech Viewer • Indiana Speech Training Aid • Video Voice Speech Training System • Speech Rehabilitation Speech Training Aid for the Hearing Impaired (HARP)

25. Desirable features • should provide a contrastive visual training ie the correct model of a reference speaker and the deviant production of the client should be shown simultaneously to be compared with each other • the visual pattern must be • attractive • easily comprehensible • shown without delay

26. OLP features • Ability to contrast desired target and undesirable utterances (on same display) • Mapping of articulatory information to 2D visual display in real-time • Motivating displays and exercises • Flexibility - can be individualised by therapist • Can be used at home by client