3D Computer Simulation of the Human Vocal Tract

1. 3D Computer Simulation of the Human Vocal Tract Towards an Extensible Infrastructure for a Three-dimensional Face and Vocal-Tract Model Sid Fels, ECE, Bryan Gick, Linguistics Florian Vogt, ECE, Ian Wilson, Linguistics, Carol Jaeger, ECE University of British Columbia Vancouver, BC, Canada V6T 1Z4

2. Introduction and Motivation • Want speech synthesis that is: • low bandwidth, • high quality, • visually coordinated, • physically based, • real-time • Important developments • computational speed increases • new imaging techniques • better understanding of vocal tract

3. Speech Synthesis Techniques • Three main synthesis techniques: • Time domain • LPC (Markel & Gray, 1972), CELP (Schroeder & Atal), Multipulse (Atal & Remde) • CELP used in cell phones - good quality at 4.8Kbps • used in text-to-speech applications too • concatenation based systems • Frequency domain • Articulatory domain

4. Time domain Text To Speech (TTS) • TTS in time domain • concatenate prerecorded speech segments • pitch change and transitions tricky • research on different ways to do this • Here’s a few examples: • CSTR Edinburgh: diphone synthesis/ non uniform unit selection 1, 2, 3 • CHATR ATR-ITL Kyoto / Japan, non uniform unit selection 1 • BellLabs-TTS-System,LPC diphone synthesis 1, 2 • PSOLA (Verhelst, 1990) and more

5. Frequency domain TTS • TTS using formants: • spectral changes are slow • should interpolate well • rules for transitions can’t be simply linear • change speaker characteristics easily • not so natural though • change intonation (somewhat) • Main synthesizers: (Klatt, 1980) and (Rye & Holmes, 1982) • Examples: • Infovox, Telia Promotor / KTH Stockholm 1 • Multilingual TTS system, TI Uni Duisburg 1 • DECtalk: regular 12, affective modification (Cahn) 123

6. Articulatory Synthesis • Parameterize human vocal tract, glottis and lungs • mechanical or electrical systems

7. 2D Articulatory Synthesis • Articulatory model • Mermelstein, 1971 model used at Haskins • Coker, 1976 • C: Tongue body Center • H: Hyoid • J: Jaw • L: Lips • T: Tongue Tip • V: Velum

8. 2D Articulatory Synthesis • To synthesize speech: • convert to area function • use acoustic tube model • activate with sound source • Sound/excitation source • waveform • model • Glottal and Lung model from (Flanagan, Ishizaka, and Shipley, 1975) • 2 masses and springs (oscillator)

9. Articulatory Synthesis: Haskins 1st frame • Examples: • /da/ • about 75msec from start to end • sound • utterance interpolated frames 2nd frame

10. Articulatory Synthesis: Haskins • Problems: • vowel sounds OK • plosives, fricatives, liquids and aspirants not OK • where to get articulator data? • Measurements • MRI • electromagnetic articulograph • ultrasound • X-ray • acoustics • Models • only use rigid 2D models

11. Articulatory Synthesis: History • Interesting history • sometimes hot (1700s, 2000) and sometimes not (1800s, 1970-1980s) • Important now because of “Talking Head” research • McGurk effect

12. Articulatory Synthesis: Talking Heads • Visual and auditory signals interact • visual signal can make auditory signal hard to hear • McGurk Effect Demo • Talking heads important for: • more natural interaction • dubbing new voices • compact encoding of voice and image • Can we create good talking head from acoustic signal? • Not so easy: i.e., Bregler, Slaney and Covell • articulatory synthesis provides necessary articulatory movement with audio waveform • see “Speech Recognition and Sensory Integration”, Massaro and Stork, American Scientist, Vol. 86, 1998.

13. Articulatory Synthesis: History • Kratzenstein resonators (1770 - Imperial Academy of St. Petersburg contest)

14. Articulatory Synthesis: AVTs • von Kempelen’s AVT (1791)

15. Articulatory Synthesis: more T.H. • R. R. Riesz's talking mechanism, 1937

16. Articulatory Synthesis: electronic AVTs • The Voder (Dudley, Riesz and Watson, 1939) • Example

17. Articulatory Synthesis: more AVTS • Alexander G. Bell and Melville Bell • moulded human skull • Glove-TalkII (Fels and Hinton, 1998) • uses gestural articulator model

18. Articulatory TTS Synthesis • Text-to-speech possible: • Pavarobotti, National Center for Voice and Speech, (U. Iowa) 1, 2 • See also Perry Cook’s work at Princeton (formerly at Stanford (CCRMA)) • http://www.cs.princeton.edu/~prc • SPASM (example song) • P. R. Cook, "Synthesis of the Singing Voice Using a Physically Parameterized Model of the Human Vocal Tract," ICMC, 1989.

19. Where we are going: 3D Articulator Synthesis • Creating infrastructure for research on 3D articulatory synthesis • Use for: • speech synthesis • speech analysis • articulation analysis

20. Where we are at: • Developing and extending scenegraph methodology for model specification • multi-scale and multi-level capable • Elaborating scenegraph model based on anatomy • base-line for refinement or simplification • Exploring techniques for synthesis • support 2D to 3D techniques • integrate source-tract models as well • Developing measurement tools • ultrasound tongue tracking; integrate with Haskins

21. Articulatory Synthesis: Project Overview • Build S/W Infrastructure to support modular approach to modeling • extrapolate and abstract from Lee, Waters and Terzopolous work Lee, Terzopolous, Waters, 1998 Haber, Kaehler, at al, 2002

22. Software Architecture • 5 main components to deal with: 1. simulator engine, 2. three-dimensional geometry module 3. graphical user interface (GUI) module, 4. synthesis engine and 5. numerics engine.

23. Graphics 3D geometry simulator numerics GUI Structure of Simulator aeroacoustics Imagingdata sources

24. Graphics 3D geometry simulator numerics GUI Structure of Simulator aeroacoustics Imagingdata sources

25. 3D Geometry: Scene Graph • Base model notation on Scene Graph • basis of 3D animation • nodes for specifying graphical models including • shapes, cameras, lights, properties, transformation, engines, selection, view etc. • Extend and add nodes to represent relationships • muscles, constraints, nerves, dynamics • may need multiple passes per iteration

26. 3D Geometry: Scene Graph • Example decomposition • Head • Skull • Mouth • jaw • teeth • tongue • hyoid • cheek, other soft structures • Pharynx • Nose • Larynx • Respiratory tract

27. Using Image Data • Data from MRI, ultrasound, EMA and other imaging devices • used in real-time or offline • create geometry • provide constraints on system

28. Graphics 3D geometry simulator numerics GUI Structure of Simulator aeroacoustics Imagingdata sources

29. Graphics • Separate out rendering of model • Integrate with other 3D animation toolse.g. Blender, Maya, 3DMax.

30. Graphics 3D geometry simulator numerics GUI Structure of Simulator aeroacoustics Imagingdata sources

31. GUI • Separate out to make simulation code clean • Allow multiple access points to reduce dependencies • command line, GUI, scripted, stdin/stdout, files (save state) • Control module behaviour • Automate as much as possible • extensions get support for GUIs

32. Graphics 3D geometry simulator numerics GUI Structure of Simulator aeroacoustics Imagingdata sources

33. Numerics • All numerical processes separated out from simulation • allow for improvements to numerical routines • allow flexibility to switch methods • i.e. Implicit vs. Explicit euler integration, FEM solver • Each pass through scene graph builds up state • numerics operate on state and return state back to scene graph

34. Graphics 3D geometry numerics GUI Structure of Simulator simulator aeroacoustics Imagingdata sources

35. Simulation • As simple as possible: • infinite loop that updates state of simulation • traverses scene graph • calculate new state • render • simulate airflow • can be thought of as an independent module as well

36. 3D Articulator Techniques & Issues 1. Vocal Tract Model • geometric versus physical model • static versus dynamic model • parameter extraction/tuning from • real data • X-Ray, MRI, EMA, Ultrasound, Electropalatography (Stone and Lundberg, 1996) • anatomy

37. 3D Articulatory Synthesis Model • Our direction • use physical model of vocal tract and tongue • articulator/muscle based model • match face model • use dynamic modeling of soft tissues • tongue, lips, cheeks, pharynx, etc. • include volume constraints • collision detection

38. 3D Articulatory Vocal Tract Model • Which technique to use for soft tissues? (see Gibson and Mirtich, 1997 for review) a) Non-physical models • splines, patches • difficult to get deformations correct • may be good for representing static 3D shape of vocal tract b) Spring-mass models • use a collection of point masses connected by springs • popular in facial animations • i.e. Waters, 1987 and Lee, Terzopolous, Waters, 1998 • may be difficult to model stiff areas well • numerical instabilities • volumetric constraints difficult to model well

39. 3D Articulatory Vocal Tract Model c) Boundary Line Integral and Boundary Element Method (James and Pai, 1999) • Use boundary integral equation formulation • of static linear elasticity • use boundary element method to solve • limited to boundary only • how to deal with heterogeneous tissue? Tongue may be difficult • Limited to linear elasticity • should be OK for small deformations d) Continuum models and Finite Element Methods • some human tissue models have been created: • Payan et al 2001, Gouret et. al., 1989, Chen and Zeltzer, 1992, Bro-Nielsen, 1997.

40. 3D Articulator Synthesis: Engwall / Badin • One attempt by (Engwall / Badin, 1999) • geometric model of vocal tract derived from articulator parameters • array of vertices (polygonal mesh) • symmetric around midsagittal plane • tongue model is set of filtered vertices • 5 parameters • synthesis model • acoustic tube

41. 3D Articulatory Synthesis Model Anatomical Models for simulation • FEM-Tongue models e.g Dung 2000 • Ultrasound Tongue models e.g. Stone 1998 • Anatomical Tongue model Takemoto 2001

42. 3D Articulatory Synthesis Model 2) Synthesis Model • simulate propagation of air pressure waves through the 3D vocal tract model • simplified source model • fluid dynamic models • maybe modification of ray-tracing • 2D acoustic tube model • dynamic models and time domain source • classical electrical analog • enhanced with airflow model (Jackson, 2000) • 2.5D and 3D acoustic tube • with and without source models • FEM or BEM analysis for flow and turbulence

43. 3D Articulatory Synthesis: Applications • Surgical prediction (Payan et al 2002) • compression • videophone, multimedia data • speech research tool • text-to-speech synthesis • lip synchronization in movies Haber, at al, 2001 • new musical instruments Vogt, etal 2001 • based on 3D models + wave propagation • Some applications may not require complete, 3D articulatory synthesis model

44. 3D Articulatory Synthesis: Summary • Continue building framework for public domain 3D Articulatory Synthesizer • Construction of modular S/W architecture • integrating simple models of vocal tract behaviour • providing support for multiple level-of-detail modeling • 2D tube model for synthesis • Developing 3D tube models • Building anatomically based vocal tract models • development of scene graph semantics and syntax

45. Articulatory Synthesis: Haskins • Haskins lab articulatory synthesis (ASY) • synthesize static vowel sounds (/u/, /i/) • vowel space mostly function of tongue position and height

46. Articulatory Synthesis: Haskins • Can also synthesize utterances • Use table (script) for specifying configuration of vocal tract (articulation) • Use table (control) that specifies timing of • source • articulation • like key framing or morphing • synthesis done every pitch pulse

47. Articulatory Synthesis: Haskins 1st frame • Examples: • /da/ • about 75msec from start to end • sound • utterance interpolated frames 2nd frame

48. Articulatory Synthesis: Haskins • Formant Tracks

49. : Articulatory Synthesis: Haskins • Spectrogram

50. Articulatory Synthesis: Haskins • Problems: • vowel sounds OK • plosives, fricatives, liquids and aspirants not OK • where to get articulator data? • Measurements • MRI • electromagnetic articulograph • ultrasound • X-ray • acoustics • Models • only use rigid 2D models