Speech Recognition

1. Speech Recognition • SR has long been a goal of AI • If the computer is an electronic brain equivalent to a human, we should be able to communicate with that brain • It would be very convenient to communicate by natural language means • SR/Voice input • Natural language understanding • Natural language generation • Speech synthesis output • There are three distinct problems and AI researchers have not tried to tackle them as a combined unit although • SR often includes part of the NLU • The study of speech synthesis has led to a greater understanding of SR

2. Speech Recognition as a Problem • We find a range of possible parameters for the speech recognition task • Speaking mode: isolated words vs. continuous speech • Speaking style: read speech vs. conversational • Dependence: speaker dependent (designed for one speaker) vs. speaker independent trained vs. speaker independent untrained • Vocabulary: small (<= 50 words), moderate (50-200 words), large (1000 words), very large (5000 words), entire language • Language model: full grammar, limited grammar, finite state (HMM, Bayes network) using n-grams, knowledge base, neural network, other • Semantics: small lexicon/restricted domain, restricted domain, general purpose • Speech to noise ratio and other hardware (quality of microphone, is the environment noisy?) • Real-time versus off-line processing

3. SR as a Process • Most researchers view SR as a bottom-up mapping process • Acoustic signal is the data • Use various forms of signal processing to obtain spectral data • Generate phonetic units to account for groups of data (phonemes, diphones, demisyllables, syllables, words) • Combine phonetic units into syllables and words • Possibly use syntax (and semantic knowledge) to ensure the words make sense • Use top-down processing as needed

4. Radio Rex • A toy from 1922 • A dog mounted on an iron base with an electromagnetic to counteract the force of a spring that would push “Rex” out of his house • The electromagnetic was interrupted if an acoustic signal at 500 Hz was detected • The sound “e” (/eh/) as found in Rex is at about 500 Hz • So the dog comes when called

5. Speech Analysis and Synthesis Models • The first important model was discovered in the 1930s at Bell Labs when the speech spectrum was first mapped • Speech produces a distribution of power (energy) across multiple frequencies Spectral plot of a speech signal and two particular “time slices” showing the energy across the various frequencies

6. The Task Pictorially The speech signal is divided into overlapping segments These are processed to create small “windows” of speech by using FFTs and LPC analysis which provides a series of energy patterns at different frequencies

7. Identifying Speech • With the speech signal “carved up” we can analyze the frequency of the energy at each time slice • We can try to identify the cause of each time slice – that is, what phonetic unit was uttered to create that signal • There are identifying characteristics that we can look for • Constants are usually short bursts of energy • Stop consonants are shortest • Fricatives are consonants created by constriction in the vocal tract (by the lips) • Approximants are consonants that are created using less energy and construction like l and r • We can also look for voicing to help narrow down the specific consonant

8. Vowels and Formants • When vowels are uttered, they are longer in duration and create formants • Formants are acoustic resonances which have distinctive characteristics • lengthy, relatively parallel “lines” in the spectrum, high frequency • typically a vowel creates two formants although sometimes three are created • Identifying the vowel is sometimes a matter of determining at what frequencies the F1 and F2 formants are found at (and how they change over time) F1 F2 F3

9. Example Formant Centers • Here is a breakdown of the frequencies for the formants for various vowel sounds • Unfortunately, there are a number of factors they influence the formats including • age, gender • excitement level, volume • speed of speech • the sounds are on both sides of the vowel • formants iin nine will appear differently than the formants for the i in time!

10. Vowel Formant Frequencies

11. Consonant Place of Articulation

12. Co-articulation • As stated on the previous slide, the impact of one articulatory sound into another causes variation in the speech spectrum • The result is that it isn’t a simple mapping of frequency  phonetic unit • but instead a very complex, poorly understood situation • In speech recognition, the problem can be simplified by insisting on discrete speech • Pauses between every pair of words • Within a word, we might model how the entire word should appear (that is, our recognition units are words, not phonemes, letters or syllables) or we might try to model how co-articulation impacts the speech spectrum • But in continuous speech, the problem is magnified because different combinations of words will create vastly different spectra

13. Phonetic Dependence • Below are two wave forms created by uttering the same vowel sound, “ee” as in three (on the left) and tea (on the right) • notice how dissimilar the “ee” portion is • the one on the right is longer and the one on the left has higher frequency formants

14. Isolated Speech Recognition • Early speech recognition concentrated on isolated speech because continuous speech recognition was just not possible • Even today, accurate continuous speech recognition is extremely challenging • There are many advantages to isolated speech recognition but the primary advantages are • The speech signal is segmented so that it is easy to determine the starting point and stopping point of each word • Co-articulation effects are minimized A distinct gap (silence) appears between words

15. Early Speech Recognition • Bell labs 1952 implemented a system for isolated digit recognition (of a single speaker) by comparing the formants of the speech signal to expected frequencies • RCA labs implemented an isolated speech recognition system for a single speaker on 10 different syllables • MIT Lincoln Lab constructed a speaker independent recognizer for 10 vowels

16. In the 1960s • First, specialized hardware was developed for phoneme recognition, vowel recognition and number recognition • In England, a system was developed to recognize 4 vowels and 9 consonants using statistical information about allowable sequences as found in English • Another advancement was by adopting a non-uniform time scale so that sounds were not expected to be of equal duration • This included dynamic programming in an attempt to try various alignments of the phonetic units to the speech signal

17. Continued • In the late 1960s, Linear Predictive Coding (LPC) was developed • A speech signal is largely generated by creating a “buzzing” type of sound • the LPC estimates the locations of the formants • we remove the formants from the signal to further estimate the frequency and energy remaining behind that buzzing • this energy can give us an indication of the consonants (while the formats give us evidence of the vowels) • The result of LPC is a list of coefficients, numbers that can be used to “describe” the sound and thus be used for identification

18. In the 1970s • ARPA initiated wide scale research around 1971 on multispeaker continuous speech of large vocabularies • Multispeaker speech – people speak at different frequencies, particularly if speakers are of different sexes and widely different ages • Continuous speech – the speech signal for a given sound is dependent on the preceding and succeeding sounds, and in continuous speech, it is extremely hard to find the beginning of a new word, making the search process even more computationally difficult • this was a significant challenge, discrete speech over small vocabularies had been partially solved, but no system did continuous speech at this time

19. Continued • Large vocabularies – to this point, speech recognition systems were limited to a few or a few dozen words, ARPA wanted 1000-word vocabularies • this not only complicated the search process by introducing 10-100 times the complexity in words to match against, but also required the ability to handle ambiguity at the lexical and syntactical levels • Language model – ARPA permitted a restricted syntax but it still had to allow for a multitude of sentence forms, and thus required some natural language capabilities (syntactic parsing) • ARPA demanded real-time performance • Four systems were developed out of this research

20. Harpy • Developed at CMU • Harpy created a giant lattice space of possible utterances as combinations of • the phonemes of every word in the lexicon and by using the syntax, all possible word combinations • The syntax was specified as a series of production rules • The lattice consists of about 15,000 nodes, each node equivalent to a phoneme or part of a phoneme • includes silence and connector phonemes – what we expect as the co-articulation affect between two phonemes • the lattice took 13 hours to generate on a PDP-10 • each node was annotated with expected LPC coefficients for matching knowledge • Harpy performed a beam search to find the most likely path through the lattice • about 3% of the entire lattice would be searched during the average case

21. Harpy Lattice and Beam Search

23. More on Harpy • Because all phonemes are represented as nodes in the lattice, and the search algorithm moves from node to node for each “time slice” of the utterance • it is implied that all phonemes have the same (or similar) durations, which is not true • to get around this problem, phoneme nodes might point to related phoneme nodes to extend the duration and better match the utterance • Below is a sample lattice for the word “please” • Notice how it is possible to go from PL to L indicating that the “L” might have a longer duration than expected

24. Hearsay-II • Hearsay-II attempted to solve the problem through symbolic problem solvers • each problem solver is known as a knowledge source (KS) • the knowledge sources communicate through a blackboard architecture • the blackboard is a place for data, partial results and finished results • each KS knew what part of the BB to read from and where to post partial conclusions to • a scheduler would use a complex algorithm to decide which KG should be invoked next based on priority of KGs and what knowledge was currently available on the BB

25. Blackboard Architecture Hearsay could recognize 1011 words of continuous speech and several speakers with a limited syntax with an accuracy of around 90%

26. Hearsay Architecture

27. Hearsay Knowledge Sources • Signal Acquisition, Parameter Extraction, Segmentation, and Labeling: • SEG: Digitizes the signal, measures parameters, and produces a labeled segmentation • Word Spotting • POM: Creates syllable-class hypotheses from segments. • MOW: Creates word hypotheses from syllable classes. • WORD-CTL: Controls the number of word hypotheses that MOW creates. • Phrase-Island Generation“ • WORD-SEQ: Creates word-sequence hypotheses that represent potential phrases from word hypotheses and weak grammatical knowledge. • WORD-SEQ-CTL: Controls the number of hypotheses that WORD-SEQ creates • PARSE: Attempts to parse a word sequence and, if successful, creates a phrase hypothesis from it.

28. Continued • Phrase Extending: • PREDICT: Predicts all possible words that might syntactically precede or follow a given phrase. • VERIFY: Rates the consistency between segment hypotheses and a contiguous word-phrase • CONCAT: Creates a phrase hypothesis from a verified contiguous word-phrase pair. • Rating, Halting, and Interpretation. • RPOL: Rates the credibility of each new or modified hypothesis, using information placed on the hypothesis by other KSs • STOP: Decides to halt processing (detects a complete sentence with a sufficiently high rating, or notes the system has exhausted its available resources) and selects the best phrase hypothesis or set of complementary phrase hypotheses as the output. • SEMANT: Generates an unambiguous interpretation for the reformation-retrieval system which the user has queried

29. Blackboard: Phonetic Units/Syllables Sentence is “Are any by Feigenbaum and Feldman?”

30. Blackboard:Creating Words From Phonetic Units

31. Blackboard: Sentence Structure and Syntax

32. More Hearsay Details • The original system was implemented between 1971 and 1973 • Hearsay-II was an improved version implemented between 1973 and 1976 • Hearsay-II’s grammar was simplified as a 1011x1011 binary matrix to indicate which words could follow which words (matrix generated from training sentences) • Hearsay is more noted for pioneering the blackboard architecture as a platform for distributed processing in AI whereby a number of different problem solvers (agents) can tackle a problem • Unlike Harpy, Hearsay had to spend a good deal of time deciding which problem solver (KS) should execute next • this was the role of the scheduler • So this took time away from the task of processing the speech input

33. BBN and SRI • BBN attempted to tackle the problem through HWIM (hear what I mean) • The system is knowledge-based, much like Hearsay although scheduling was more implicit, based on how humans attempted to solve the problem • Also, the process involved first identifying the most certain words and using them as “islands of certainty” to work both bottom-up and top-down to extend what could be identified • KS would call each other and pass processed data rather than use a central mechanism • Bayes probabilities were used for scoring phoneme and word hypotheses (probabilities were derived from statistics obtained by analyzing training data) • The syntax was represented by an Augmented Transition Network so this system had a greater challenge at the syntax level than Hearsay and Harpy (we look at ATNs next week) • After an initial start, SRI teamed with SDC to build a system, which was also knowledge-based, primarily rule-based • But the system was never completed having achieved very poor accuracy in early tests

34. ARPA Results • Results are shown in the table below • Of all of the systems, only Harpy showed promise • However, none of the systems were thought to be scalable to larger sizes • The knowledge-based approach would require a great deal more effort in encoding the knowledge to recognize new phonemes • With more words and a larger syntax, the branching factor would cause accuracy to decrease to an even poorer rate • Generating Harpy’s lattice was only possible because of the limited size of the lexicon and syntax • Harpy’s approach though provided the most accuracy and this led to the adoption of HMMs for SR

35. BBN’s Byblos • In the early 80s, work at AT&T Bell Labs led to better performance on speaker independent SR by developing • Clustering algorithms • Better understanding of signal processing techniques to identify spectral distance measures • these concepts and others led to the introduction of HMMs as an approach for solving the SR problem • Byblos was the first noteworthy system to use HMMs to solve SR • There is an HMM for each phoneme in English • Trigrams were implemented for transition knowledge (phone-to-phone and word-to-word)

36. N-Grams • An N-Gram is a collection of n states to compute the transition probability of the particular sequence • To create the n-grams, take every sequence of n items in the domain • e.g., all combinations of n words in sequence • In order to obtain the transition probabilities for the n-gram, we have to use the training data • Problem: the training data will not have all possible combinations • Solution: provide a default minimum probability • Question: what should n be? • 2 or 3 is common (bigrams, trigrams)

37. Example: Bi-grams for English 2 Letter Sequences

38. Sample HMM Grammars • Unigrams (SWB): • Most Common: “I”, “and”, “the”, “you”, “a” • Rank-100: “she”, “an”, “going” • Least Common: “Abraham”, “Alastair”, “Acura” • Bigrams (SWB): • Most Common: “you know”, “yeah SENT!”, “!SENT um-hum”, “I think” • Rank-100: “do it”, “that we”, “don’t think” • Least Common: “raw fish”, “moisture content”, “Reagan Bush” • Trigrams (SWB): • Most Common: “!SENT um-hum SENT!”, “a lot of”, “I don’t know” • Rank-100: “it was a”, “you know that” • Least Common: “you have parents”, “you seen Brooklyn”

39. Discrete Speech Using HMMs A 5-stage phoneme model Discrete speech of the 10 digits

40. More on Byblos • The HMM phoneme model contains three states to represent the beginning of a phoneme, the middle of a phoneme and the end of a phoneme • The states can loop back onto themselves and state 1 can go right to state 3, this provides the HMM with the ability to match the actual duration of the phoneme in the utterance • Specialized phoneme models were developed for phone-to-phone transitions to better model the impact of co-articulation • Transition probabilities were generated using triphones as stated on the previous slide, but what about the output probabilities? The word “seeks”

41. Continuous Speech with HMM • Many simplifications made for discrete speech do not work for continuous speech • HMMs will have to model smaller units, possibly phonemes • To reduce the search space, use a beam search • To ease the word-to-word transitions, use bigrams or trigrams The process is similar to slide 39 where all phoneme HMMs are searched using Viterbi, but here, transition probabilities are included along with more codebooks to handle the phoneme-to-phoneme transitions and word-to-word transitions This 7-stage phoneme model is used in the Sphinx system (in this case, the phoneme /d/)

42. Continuous Speech Search within a word compared to searching across words

43. Codebooks • One of the new innovations for Byblos is the codebook • A codebook is a discretization of a time slice of the speech signal, that is, converting the numeric data into a series of descriptors • these descriptors are usually vectors of LPC data • Byblos used 256 different codebooks, so a time slice would be classified as the closest matching codebook available • The output probability is how likely a given codebook would be seen for a given phonetic node • e.g., for the /a/ phone first state, the likelihood of seeing codebook 1 is .05 and the likelihood of seeing codebook 17 is .13, etc • Output probabilities would be derived using training data

44. The codebook is used as the observation The input signal is decomposed into time slices where each time slice is a vector quantization (e.g., LPC coefficients) The VQ is then mapped into the nearest matching codebook using some distance metric (e.g., Euclidean distance) More codebooks mean larger HMMs but more accurate results HMM Codebook HMM systems will commonly use at least 256 codebooks

45. 5 0 0 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 0 1 2 3 4 6 7 8 9 Creating the Codebooks • Training data are used • Data are clustered using the training data • centroids are located for each cluster • Each cluster then is described as a vector • The centroid + vector make up each codebook • see the next slide

46. 6 5 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 6 6 6 6 7 7 7 7 8 8 8 8 9 9 9 9 0 1 2 3 4 6 7 8 9 0 1 2 3 4 5 7 8 9 Continued

47. The HMM Speech Process • Speech signal is processed into feature vectors (VQs) • The HMM is traversed where the observation for each time unit is the most closely matching codebook for that time period’s feature vector • Output probabilities are at first random and then trained using Baum-Welch, transition probabilities are based on trigrams

48. Computing bj(Ot) Here, the VQ for time slice t is shown as a star, and the central codebook is the one selected as the closest match • For time slice t, we have the VQ • The probability for bj(Ot) (that is, the probability of seeing codebook Ot from state j) is computed as the number of vectors t in state j / number of vectors in state j • The selected codebook t is based on which codebook comes closest to the actual vector VQt • Using the beam search allows us to consider more than one codebook – we might be looking over as many as 10 closely matching codebooks at a time

49. Byblos Results • The initial system proved capable of handling 12 different speakers • Each speaker would first have to train the system to their speech through training sentences (600 sentences each, average 8 words per sentence) • They tested the system with and without the triphone grammar and with 1 and 3 codebooks • 3 codebooks per time slice provided a greater variety of data to use by decomposing the feature vectors into three different sets of data • Newer versions of Byblos have been created that achieve better accuracy

50. HMM Approach In Detail • The HMM approach views speech recognition as finding a path through a graph of connected phonetic and/or grammar models, each of which is an HMM • The speech signal is the observable (although what we actually use are the most closely matching codebook from the VQ) • The unit of speech (phoneme) uttered is the hidden state • Separate HMMs are developed for every phonetic unit • there may be multiple paths through a single HMM to allow for differences in duration caused by co-articulation and other effects