1 / 57

6-Text To Speech (TTS) Speech Synthesis

6-Text To Speech (TTS) Speech Synthesis. Speech Synthesis Concept Speech Naturalness Phone Sequence To Speech Articulatory Approaches Concatenative Approaches HMM-based Approaches Rule-Based Approaches. Speech Synthesis Concept. Text. Speech waveform. Speech waveform. Text to

lalasa
Download Presentation

6-Text To Speech (TTS) Speech Synthesis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 6-Text To Speech (TTS) Speech Synthesis • Speech Synthesis Concept • Speech Naturalness • Phone Sequence To Speech • Articulatory Approaches • Concatenative Approaches • HMM-based Approaches • Rule-Based Approaches

  2. Speech Synthesis Concept Text Speech waveform Speech waveform Text to Phone Sequence Phone Sequence to Speech Text Natural Language Processing (NLP) Speech Processing

  3. Speech Naturalness • Obviation of undesirable noise and distortion and dissociation from speech • Prosody generation • Speech energy • Duration • pitch • Intonation • Stress

  4. Speech Naturalness (Cont’d) • Intonation and Stress are very effective in speech naturalness • Intonation : Variation of Pitch frequency along speaking • Stress : Increasing the pitch frequency in a specific time

  5. Which word receives an intonation? • It depends on the context. • The ‘new’ information in the answer to a question is often accented • while the ‘old’ information is usually not. • Q1: What types of foods are a good source of vitamins? • A1: LEGUMES are a good source of vitamins. • Q2: Are legumes a source of vitamins? • A2: Legumes are a GOOD source of vitamins. • Q3: I’ve heard that legumes are healthy, but what are they a good source of ? • A3: Legumes are a good source of VITAMINS. Slide from Jennifer Venditti

  6. Same ‘tune’, different alignment LEGUMES are a good source of vitamins The main rise-fallaccent (= “I assert this”) shifts locations. Slide from Jennifer Venditti

  7. Same ‘tune’, different alignment Legumes are a GOOD source of vitamins The main rise-fallaccent (= “I assert this”) shifts locations. Slide from Jennifer Venditti

  8. Same ‘tune’, different alignment legumes are a good source of VITAMINS The main rise-fallaccent (= “I assert this”) shifts locations. Slide from Jennifer Venditti

  9. Types of Waveform Synthesis • Articulatory Synthesis: • Model movements of articulators and acoustics of vocal tract • Concatenative Synthesis: • Use databases of stored speech to assemble new utterances. • Diphone • Unit Selection • Statistical (HMM) Synthesis • Trains parameters on databases of speech • Rule-Based (Formant) Synthesis: • Start with acoustics, create rules/filters to create waveform

  10. Articulatory Synthesis • Simulation of physical processes of human articulation • Wolfgang von Kempelen (1734-1804) and others used bellows, reeds and tubes to construct mechanical speaking machines • Modern versions “simulate” electronically the effect of articulator positions, vocal tract shape, etc on air flow.

  11. Concatenative approaches • Two main approaches: • 1- Concatenating Phone Units • Example: concatenating samples of recorded diphones or syllables • 2- Unit selection • Uses several samples for each phone unit and selects the most appropriate one when synthesizing

  12. Phone Units • Paragraph ( ) • Sentence ( ) • Word (Depends on the language. Usually more than 100,000) • Syllable • Diphone & Triphone • Phoneme (Between 10 , 100)

  13. Phone Units (Cont’d) • Diphone : We model Transitions between two phonemes . . . . . p1 p3 p2 p4 p5 Diphone Phoneme

  14. Phone Units (Cont’d) • Farsi phonemes: 30 • Farsi diphones: 30*30 = 900 • Phoneme /zho/ is missing (?) • Farsi triphones: 27000 in theory • Not all of the triphones are used

  15. Phone Units (Cont’d) • Syllable = Onset (Consonant) + Rhyme • Syllable is a set of phonemes that exactly contains one vowel • Syllables in Farsi : CV , CVC , CVCC • We have about 4000 Syllables in Farsi • Syllables in English :V, CV , CVC ,CCVC, CCVCC, CCCVC, CCCVCC, . . . • Number of Syllables in English is too many

  16. Phone Sequence To Speech (Cont’d) Phone Sequence to primitive utterance primitive utterance to Natural Speech Text to Phone Sequence Speech Text NLP Speech Processing

  17. Concatenative Approaches • In this approaches we store units of natural speech for reconstruction of desired speech • We could select the appropriate phone unit for speech synthesis • we can store compressed parameters instead of main waveform

  18. Concatenative Approaches (Cont’d) • Benefits of storing compressed parameters instead of main waveform • Less memory use • General state instead of a specific storedutterance • Generating prosody easily

  19. Concatenative Approaches (Cont’d) Type of Storing Phone Unit Paragraph Sentence Word Syllable Diphone Phoneme Main Waveform Main Waveform Main Waveform Coded/Main Waveform Coded Waveform Coded Waveform

  20. Concatenative Approaches (Cont’d) • Pitch Synchronous Overlap-Add-Method (PSOLA) is a famous method in phoneme transmit smoothing • Overlap-Add-Method is a standard DSP method • PSOLA is a base action for Voice Conversion. • In this method in analysis stage we select frames that are synchronous by pitch markers.

  21. Diphone Architecture Example • Training: • Choose units (kinds of diphones) • Record 1 speaker saying 1 example of each diphone • Mark the boundaries of each diphones, • cut each diphone out and create a diphone database • Synthesizing an utterance, • grab relevant sequence of diphones from database • Concatenate the diphones, doing slight signal processing at boundaries • use signal processing to change the prosody (F0, energy, duration) of selected sequence of diphones

  22. Unit Selection • Same idea as concatenative synthesis, but database contains bigger varieties of “phone units” from diphones to sentences • Multiple examples of phone units (under different prosodic conditions) are recorded • Selection of appropriate unit therefore becomes more complex, as there are in the database competing candidates for selection

  23. Unit Selection • Unlike diphone concatenation, little or no signal processing applied to each unit • Natural data solves problems with diphones • Diphone databases are carefully designed but: • Speaker makes errors • Speaker doesn’t speak intended dialect • Require database design to be right • If it’s automatic • Labeled with what the speaker actually said • Coarticulation, schwas, flaps are natural • “There’s no data like more data” • Lots of copies of each unit mean you can choose just the right one for the context • Larger units mean you can capture wider effects

  24. Unit Selection Issues • Given a big database • For each segment (diphone) that we want to synthesize • Find the unit in the database that is the best to synthesize this target segment • What does “best” mean? • “Target cost”: Closest match to the target description, in terms of • Phonetic context • F0, stress, phrase position • “Join cost”: Best join with neighboring units • Matching formants + other spectral characteristics • Matching energy • Matching F0

  25. Unit Selection Search

  26. Joining Units • unit selection, just like diphone, need to join the units • Pitch-synchronously • For diphone synthesis, need to modify F0 and duration • For unit selection, in principle also need to modify F0 and duration of selection units • But in practice, if unit-selection database is big enough (commercial systems) • no prosodic modifications (selected targets may already be close to desired prosody)

  27. Unit Selection Summary • Advantages • Quality is far superior to diphones • Natural prosody selection sounds better • Disadvantages: • Quality can be very bad in some places • HCI problem: mix of very good and very bad is quite annoying • Synthesis is computationally expensive • Needs more memory than diphone synthesis

  28. Rule-Based Approach Stages • Determine the speech model and model parameters • Determine type of phone units • Determine some parameter amount for each phone unit • Substitute sequence of phone units by its equivalent parameter sequence • Put parameter sequence in speech model

  29. KLATT 80 Model

  30. KLATT 88 Model

  31. THE KLSYN88 CASCADE PARALLEL FORMANT SYNTHESIZER FNP FNZ FTP FTZ F1 B1 BNP BNZ BTP BTZ DF1 DB1 F2 B2 F3 B3 F4 B4 F5 B5 GLOTTAL SOUND SOURCES TL CASCADE VOCAL TRACT MODEL LARYNGEAL SOUND SOURCES F0 AV OO FL DI SS CP + AH ANV SO A1V + + - + - - B2F A2F A2V + B3F A3F AF A3V B4F A4F A4V B5F + - + - + - A5F ATV B6F F6 A6F PARALLEL VOCAL TRACT MODEL LYRYNGEAL SOUND SOURCES (NORMALLY NOT USED) AB BYPASS PATH PARALLEL VOCAL TRACT MODEL FRICATION SOUND SOURCES

  32. Three Voicing Source Model In KLATT 88 • The old KLSYN impulsive source • The KLGLOTT88 model • The modified LF model

  33. HMM-Based Synthesis • Corpus-based, statistical parametric synthesis • Proposed in mid-'90s, becomes popular since mid-'00s • Large data + automatic training => Automatic voice building • Source-filter model + statistical acoustic model Flexible to change its voice characteristics • HMM as its statistical acoustic model • We focus on HMM-based speech synthesis

  34. First extract parametric representations of speech including spectral and excitation parameters from a speech database • Model them by using a set of generative models (e.g., HMMs) • Training • Synthesis

  35. Speech Parameter Modeling Based on HMM • Spectral parameter modeling • Excitation parameter modeling • State duration modeling

  36. Spectral parameter modeling • Mel-cepstral analysis has been used for spectral estimation • A continuous density HMM is used for the vocal tract modeling in the same way as speech recognition systems. • The continuous density Markov model is a finite state machine which makes one state transition at each time unit (i.e. frame). First, a decision is made to which state to succeed (including the state itself). Then an output vector is generated according to the probability density function (pdf) for the current state

  37. Cont’d…

  38. F0 parameter modeling • While the observation of F0 has a continuous value in the voiced region, there exists no value for the unvoiced region. We can model this kind of observation sequence assuming that the observed F0 value occurs from one-dimensional spaces and the “unvoiced” symbol occurs from the zero-dimensional space.

  39. Calculation of dynamic feature: • As was mentioned, mel-cepstral coefficient is used as spectral parameter, Their dynamic feature Δc and Δ2c are calculated as follows: • Dynamic features for F0: In unvoiced region, pt, Δpt and Δ2pt are defined as a discrete symbol. • When dynamic features at the boundary between voiced and unvoiced cannot be calculated, they are defined as a discrete symbol.

  40. Effect of dynamic feature • By using dynamic features, the generated speech parameter vectors reflect not only the means of static and dynamic feature vectors but also the covariances of those • Estimation will be smoother • Good and bad effect

  41. Multi-Stream HMM structure: • The sequence of mel-cepstral coefficient vector and F0 pattern are modeled by a continuous density HMM and multi-space probability distribution HMM • Putting all this together has some advantages

  42. Synthesis part • An arbitrarily given text to be synthesized is converted to a context-based label sequence. • The text is converted a context dependent label sequence by a text analyzer. • For the TTS system, the text analyzer should have the ability to extract contextual information. However, no text analyzer has the ability to extract accentual phrase and to decide accent type of accentual phrase.

  43. Some contextual factors • When we have HMM for each phoneme • {preceding, current, succeeding} phoneme • position of breath group in sentence • {preceding, current, succeeding} part-of-speech • position of current accentual phrase in current breath group • position of current phoneme in current accentual phrase

  44. According to the obtained state durations, a sequence of mel-cepstral coefficients and F0 values including voiced/unvoiced decisions is generated from the sentence HMM by using the speech parameter generation algorithm • Finally, speech is synthesized directly from the generated mel-cepstral coefficients and F0 values by the MLSA filter

  45. Spectral representation & corresponding filter • cepstrum: LMA filter • generalized cepstrum: GLSA filter • mel-cepstrum: MLSA (Mel Log Spectrum Approximation) filter • mel-generalized cepstrum: MGLSA filter • LSP: LSP filter • PARCOR: all-pole lattice filter • LPC: all-pole filter

  46. Advantages • Most of the advantages of statistical parametric synthesis against unit-selection synthesis are related to its flexibility due to the statistical modeling process. • Transforming voice characteristics, speaking styles, and emotions. • Also Combination of unit selection and voice conversion (VC) techniques can alleviate this problem, high-quality voice-conversion is still problematic.

  47. Adaptation (mimicking voices):Techniques of adaptation were originally developed in speech recognition to adjust general acoustic model , These techniques have also been applied to HMM-based speech synthesis to obtain speaker-specific synthesis systems with a small amount of speech data • Interpolation (mixing voices): Interpolate parameters among representative HMM sets - Can obtain new voices even no adaptation data is available - Gradually change speakers. & speaking styles

More Related