1 / 98

Tutorial on neural probabilistic language models

Tutorial on neural probabilistic language models. Piotr Mirowski , Microsoft Bing London South England NLP Meetup @ UCL April 30, 2014. Ackowledgements. AT&T Labs Research Srinivas Bangalore Suhrid Balakrishnan Sumit Chopra (now at Facebook) New York University

charper
Download Presentation

Tutorial on neural probabilistic language models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tutorial on neural probabilistic language models Piotr Mirowski, Microsoft Bing London South England NLP Meetup @ UCL April 30, 2014

  2. Ackowledgements • AT&T Labs Research • Srinivas Bangalore • SuhridBalakrishnan • Sumit Chopra (now at Facebook) • New York University • YannLeCun (now at Facebook) • Microsoft • AbhishekArun

  3. About the presenter • NYU (2005-2010) • Deep learning for time series • Epileptic seizure prediction • Gene regulation networks • Text categorization of online news • Statistical language models • Bell Labs (2011-2013) • WiFi-based indoor geolocation • SLAM and robotics • Load forecasting in smart grids • Microsoft Bing (2013-) • AutoSuggest(Query Formulation)

  4. Objective of this tutorial Understand deep learning approachesto distributional semantics:word embeddings and continuous space language models

  5. Outline • Probabilistic Language Models (LMs) • Likelihood of a sentence and LM perplexity • Limitations of n-grams • Neural Probabilistic LMs • Vector-space representation of words • Neural probabilistic language model • Log-Bilinear (LBL) LMs (loss function maximization) • Long-range dependencies • Enhancing LBL with linguistic features • Recurrent Neural Networks (RNN) • Applications • Speech recognition and machine translation • Sentence completion and linguistic regularities • Bag-of-word-vector approaches • Auto-encoders for text • Continuous bag-of-words and skip-gram models • Scalability with large vocabularies • Tree-structured LMs • Noise-contrastive estimation

  6. Outline • Probabilistic Language Models (LMs) • Likelihood of a sentence and LM perplexity • Limitations of n-grams • Neural Probabilistic LMs • Vector-space representation of words • Neural probabilistic language model • Log-Bilinear (LBL) LMs • Long-range dependencies • Enhancing LBL with linguistic features • Recurrent Neural Networks (RNN) • Applications • Speech recognition and machine translation • Sentence completion and linguistic regularities • Bag-of-word-vector approaches • Auto-encoders for text • Continuous bag-of-words and skip-gram models • Scalability with large vocabularies • Tree-structured LMs • Noise-contrastive estimation

  7. ProbabilisticLanguage Models • Probabilityof a sequence of words: • Conditionalprobabilityof an upcoming word: • Chain rule of probability: • (n-1)th order Markov assumption

  8. Learning probabilisticlanguage models • Learn joint likelihood of training sentencesunder(n-1)th order Markov assumptionusing n-grams • Maximize the log-likelihood: • Assuming a parametric model θ • Could we take advantage of higher-order history? target word word history

  9. Evaluating language models: perplexity mushrooms 0.1 pepperoni 0.1 anchovies 0.01 …. fried rice 0.0001 …. and 1e-100 • How well can we predict next word? • A random predictor would give each word probability 1/Vwhere V is the size of the vocabulary • A better model of a text should assign a higher probability to the word that actually occurs • Perplexity: I always order pizza with cheese and ____ The 33rd President of the US was ____ I saw a ____ Slide courtesy of Abhishek Arun

  10. Limitations of n-grams the the the my cat cat cat sat sat sat on on on the the the mat hat sat cat sat on the mat the cat sat on the rug • Conditional likelihood of seeing a sub-sequence of length nin available training data • Limitation: discrete model (each word is a token) • Incomplete coverage of the training datasetVocabulary of size V words: Vn possible n-grams (exponential in n) • Semantic similarity between word tokens is not exploited

  11. Workarounds for n-grams • Smoothing • Adding non-zero offset to probabilities of unseen words • Example: Kneyser-Ney smoothing • Back-off • No such trigram? try bigrams… • No such bigram? try unigrams… • Interpolation • Mix unigram, bigram, trigram, etc… [Katz, 1987; Chen & Goodman, 1996; Stolcke, 2002]

  12. Outline • Probabilistic Language Models (LMs) • Likelihood of a sentence and LM perplexity • Limitations of n-grams • Neural Probabilistic LMs • Vector-space representation of words • Neural probabilistic language model • Log-Bilinear (LBL) LMs • Long-range dependencies • Enhancing LBL with linguistic features • Recurrent Neural Networks (RNN) • Applications • Speech recognition and machine translation • Sentence completion and linguistic regularities • Bag-of-word-vector approaches • Auto-encoders for text • Continuous bag-of-words and skip-gram models • Scalability with large vocabularies • Tree-structured LMs • Noise-contrastive estimation

  13. Continuous Space Language Models • Word tokens mapped to vectors in a low-dimensional space • Conditional word probabilities replaced bynormalized dynamical models on vectors of word embeddings • Vector-space representation enables semantic/syntactic similaritybetween words/sentences • Use cosine similarity as semantic word similarity • Find nearest neighbours: synonyms, antonyms • Algebra on words: {king} – {man} + {woman} = {queen}?

  14. Vector-space representation of words 1 “One-hot” of “one-of-V”representation of a word token at position tin the text corpus, with vocabulary of size V Vector-space representationof the prediction of target word wt(we predict a vector of size D) ẑt v V zt-1 Vector-space representation of the tth word history:e.g., concatenation of n-1 vectors of size D 1 zt-2 Vector-space representation of any word vin the vocabularyusing a vector of dimension D Also calleddistributed representation zv zt-1 D

  15. Learning continuous space language models • Input: • word history (one-hot or distributed representation) • Output: • target word (one-hot or distributed representation) • Function that approximates word likelihood: • Linear transform • Feed-forward neural network • Recurrent neural network • Continuous bag-of-words • Skip-gram • …

  16. Learning continuous space language models • How do we learn the word representations zfor each word in the vocabulary? • How do we learn the model that predicts the next word or its representation ẑtgiven a word history? • Simultaneous learning of modeland representation

  17. Vector-space representation of words • Compare two words using vector representations: • Dot product • Cosine similarity • Euclidean distance • Bi-Linear scoring function at position t: • Parametric model θpredicts next word • Bias bvfor word v related to unigram probabilities of word v • Given a predicted vector ẑt, the actual predicted word is the 1-nearest neighbour of ẑt • Exhaustive search in large vocabularies (V in millions) can be computationally expensive… [Mnih & Hinton, 2007]

  18. Word probabilities fromvector-space representation • Normalized probability: • Using softmax function • Bi-Linear scoring function at position t: • Parametric model θpredicts next word • Bias bvfor word v related to unigram probabilities of word v • Given a predicted vector ẑt, the actual predicted word is the 1-nearest neighbour of ẑt • Exhaustive search in large vocabularies (V in millions) can be computationally expensive… [Mnih & Hinton, 2007]

  19. Loss function • Log-likelihood model: • Numerically more stable • Loss function to maximize: • Log-likelihood • In general, loss defined as: score of the right answer + normalization term • Normalization term is expensive to compute

  20. Neural Probabilistic Language Model h A B zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD Neural network100 hidden unitsV output unitsfollowed bysoftmax word embedding in dimensionD=30 R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat functionz_hist= Embedding_FProp(model, w) % Get the embeddings for all words in w z_hist= model.R(:, w); z_hist = reshape(z_hist, length(w)*model.dim_z, 1); [Bengio et al, 2001, 2003; Schwenk et al, “Connectionist language modelling for large vocabulary continuous speech recognition”, ICASSP 2002]

  21. Neural Probabilistic Language Model h A B functions = NeuralNet_FProp(model, z_hist) % One hidden layer neural network o = model.A * z_hist + model.bias_a; h = tanh(o); S = model.B * h + model.bias_b; zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD Neural network100 hidden unitsV output unitsfollowed bysoftmax word embedding in dimensionD=30 R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Bengio et al, 2001, 2003; Schwenk et al, “Connectionist language modelling for large vocabulary continuous speech recognition”, ICASSP 2002]

  22. Neural Probabilistic Language Model h A B zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD Neural network100 hidden unitsV output unitsfollowed bysoftmax functionp = Softmax_FProp(s) % Probability estimation p_num = exp(s); p = p_num / sum(p_num); word embedding in dimensionD=30 R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Bengio et al, 2001, 2003; Schwenk et al, “Connectionist language modelling for large vocabulary continuous speech recognition”, ICASSP 2002]

  23. Neural Probabilistic Language Model h A B zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD Neural network100 hidden unitsV output unitsfollowed bysoftmax word embedding in dimensionD=30 R R R R R discrete word space {1, ..., V}V=18k words Outperforms best n-grams(Class-based Kneyser-Neyback-off 5-grams) by 7% wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat Took months to train(in 2001-2002) on AP Newscorpus (14M words) Complexity: (n-1)×D + (n-1)×D×H + H×V [Bengio et al, 2001, 2003; Schwenk et al, “Connectionist language modelling for large vocabulary continuous speech recognition”, ICASSP 2002]

  24. Log-BilinearLanguage Model functionz_hat= LBL_FProp(model, z_hist) % Simple linear transform Z_hat = model.C * z_hist + model.bias_c; zt-5 zt-4 zt-3 zt-2 zt-1 zt ẑt word embedding space ℜD E C Simple matrixmultiplication word embedding in dimensionD=100 R R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mnih & Hinton, 2007]

  25. Log-BilinearLanguage Model zt-5 zt-4 zt-3 zt-2 zt-1 zt ẑt word embedding space ℜD E C Simple matrixmultiplication functions = ...Score_FProp(z_hat, model) s = model.R’ * z_hat + model.bias_v; word embedding in dimensionD=100 R R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mnih & Hinton, 2007]

  26. Log-BilinearLanguage Model zt-5 zt-4 zt-3 zt-2 zt-1 zt ẑt word embedding space ℜD E C Simple matrixmultiplication word embedding in dimensionD=100 R R R R R R discrete word space {1, ..., V}V=18k words Slightly better thanbest n-grams(Class-based Kneyser-Neyback-off 5-grams) wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat Takes days to train(in 2007) on AP Newscorpus (14 million words) Complexity: (n-1)×D + (n-1)×D×D + D×V [Mnih & Hinton, 2007]

  27. Nonlinear Log-BilinearLanguage Model h A B zt-5 zt-4 zt-3 zt-2 zt-1 zt ẑt word embedding space ℜD E Neural network200 hidden unitsV output unitsfollowed bysoftmax word embedding in dimensionD=100 R R R R R R discrete word space {1, ..., V}V=18k words Outperforms best n-grams(Class-based Kneyser-Neyback-off 5-grams) by 24% wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat Took weeks to train(in 2009-2010) on AP Newscorpus (14M words) Complexity: (n-1)×D + (n-1)×D×H + H×D + D×V [Mnih & Hinton, Neural Computation, 2009]

  28. Learning neural language models • Maximize the log-likelihood of observed data,w.r.t. parameters θ of the neural language model • Parameters θ(in a neural language model): • Word embedding matrix R and bias bv • Neural weights: A, bA, B, bB • Gradient descent with learning rate η:

  29. Maximizing the loss function • Maximum Likelihood learning: • Gradient of log-likelihood w.r.t. parametersθ: • Use the chain rule of gradients

  30. Maximizing the loss function:example of LBL • Maximum Likelihood learning: • Gradient of log-likelihood w.r.t. parametersθ: • Neural net: back-propagate gradient function[dL_dz_hat, dL_dR, dL_dbias_v, w] = ...Loss_BackProp(z_hat, model, p, w) % Gradient of loss w.r.t. word bias parameter dL_dbias_v= -p; dL_dbias_v(w) = 1 - p; % Gradient of loss w.r.t. prediction of (N)LBL model dL_dz_hat = model.R(:, w) – model.R * p; % Gradient of loss w.r.t. vocabulary matrix R dL_dR = –z_hat * p’; dL_dR(:, w) = z_hat * (1 – p(w)); 1 R=(zv) D 1 V

  31. Learning neural language models Randomly choose a mini-batch(e.g., 1000 consecutive words) Forward-propagatethrough word embeddingsand through model Estimate word likelihood (loss) Back-propagate loss Gradient step to update model

  32. Nonlinear Log-BilinearLanguage Model h A B FProp Look-up embeddingsof the wordsin the n-gram using R Forward propagatethrough the neural net Look-up ALL vocabularywords using Rand computeenergy and probabilities(computationally expensive) zt-5 zt-4 zt-3 zt-2 zt-1 zt ẑt word embedding space ℜD E Neural network200 hidden unitsV output unitsfollowed bysoftmax word embedding in dimensionD=100 R R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mnih & Hinton, Neural Computation, 2009]

  33. Nonlinear Log-BilinearLanguage Model h A B BackProp Compute gradientsof loss w.r.t. outputof the neural net,back-propagatethrough neural netlayers B and A(computationally expensive) Back-propagatefurther down to wordembeddingsR Compute gradientsof loss w.r.t. wordsof all vocabulary,back-propagate to R zt-5 zt-4 zt-3 zt-2 zt-1 zt ẑt word embedding space ℜD E Neural network200 hidden unitsV output unitsfollowed bysoftmax word embedding in dimensionD=100 R R R R R R discrete word space {1, ..., V}V=18k words wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mnih & Hinton, Neural Computation, 2009]

  34. Stochastic Gradient Descent (SGD) • Choice of the learning hyperparameters • Learning rate? • Learning rate decay? • Regularization (L2-norm) of the parameters? • Momentum term on the parameters? • Use cross-validationon validation set • E.g., on AP News (16M words) • Training set: 14M words • Validation set: 1M words • Test set: 1M words

  35. Limitations of these neural language models • Computationally expensive to train • Bottleneck: need to evaluate probability of each word over the entire vocabulary • Very slow training time (days, weeks) • Ignoreslong-range dependencies • Fixed time windows • Continuous version of n-grams

  36. Outline • Probabilistic Language Models (LMs) • Likelihood of a sentence and LM perplexity • Limitations of n-grams • Neural Probabilistic LMs • Vector-space representation of words • Neural probabilistic language model • Log-Bilinear (LBL) LMs • Long-range dependencies • Enhancing LBL with linguistic features • Recurrent Neural Networks (RNN) • Applications • Speech recognition and machine translation • Sentence completion and linguistic regularities • Bag-of-word-vector approaches • Auto-encoders for text • Continuous bag-of-words and skip-gram models • Scalability with large vocabularies • Tree-structured LMs • Noise-contrastive estimation

  37. the Adding language features to neural LMs cat sat on the h A B DT NN VBD DT IN Additional featurescan be added as inputsto the neural net / linearprediction function. We tried POS(part-of-speechtags)and super-tags derivedfrom incomplete parsing. zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD feature embedding space ℜF zt ẑt E C feature embedding F F F F F word embedding R R R R R R discrete POSfeatures {0,1}P DT NN VBD DT IN discrete word space {1, ..., V} wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mirowski, Chopra, Balakrishnan and Bangalore (2010) “Feature-rich continuous language models for speech recognition”, SLT;Bangalore & Joshi (1999) “Supertagging: an approach to almost parsing”, Computational Linguistics]

  38. Constraining word representations h A B zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD Using the WordNet hierarchicalsimilarity between words,we tried to force some wordsto remain similar to a smallset of WordNet neighbours. feature embedding space ℜF zt ẑt E C feature embedding F F F F F word embedding R R R R R R discrete POSfeatures {0,1}P No significant change intraining time or languagemodel performance. WordNet graph of words DT NN VBD DT IN discrete word space {1, ..., V} wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mirowski, Chopra, Balakrishnan and Bangalore (2010) “Feature-rich continuous language models for speech recognition”, SLT;http://wordnet.princeton.edu]

  39. Topic mixtures of language models h(k) h(1) A(k) A(1) B(k) B(1) We pre-computed theunsupervised topicmodel representation ofeach sentence in trainingusing LDA (Latent DirichletAllocation) [Blei et al, 2003]with 5 topics.On test data, estimate topic usingtrained LDA model. zt-5 zt-4 zt-3 zt-2 zt-1 word embedding space ℜD θ1 feature embedding space ℜF θk zt ẑt E θ1 C(1) feature embedding θk F F F F F C(k) word embedding R R R R R R discrete POSfeatures {0,1}P DT NN VBD DT IN Enables to model long-rangedependencies at sentence level. sentence or document topic(5 topics) discrete word space {1, ..., V} f wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mirowski, Chopra, Balakrishnan and Bangalore (2010) “Feature-rich continuous language models for speech recognition”, SLT;David Blei (2003) "Latent Dirichlet Allocation", JMLR]

  40. Word embeddings obtained on Reuters • Example of word embeddings obtained using our language model on the Reuters corpus(1.5 million words, vocabulary V=12k words), vector space of dimension D=100 • For each word, the 10 nearest neighbours in the vector space retrieved using cosine similarity: [Mirowski, Chopra, Balakrishnan and Bangalore (2010) “Feature-rich continuous language models for speech recognition”, SLT]

  41. Word embeddings obtained on AP News [Mirowski (2010) “Time series modelling with hidden variables and gradient-based algorithms”, NYU PhD thesis] Example of word embeddings obtained using our LM on AP News(14M words, V=17k), D=100 The word embedding matrix R was projected in 2D by Stochastic t-SNE[Van der Maaten, JMLR 2008]

  42. Word embeddings obtained on AP News Example of word embeddings obtained using our LM on AP News(14M words, V=17k), D=100 The word embedding matrix R was projected in 2D by Stochastic t-SNE[Van der Maaten, JMLR 2008] [Mirowski (2010) “Time series modelling with hidden variables and gradient-based algorithms”, NYU PhD thesis]

  43. Word embeddings obtained on AP News Example of word embeddings obtained using our LM on AP News(14M words, V=17k), D=100 The word embedding matrix R was projected in 2D by Stochastic t-SNE[Van der Maaten, JMLR 2008] [Mirowski (2010) “Time series modelling with hidden variables and gradient-based algorithms”, NYU PhD thesis]

  44. Word embeddings obtained on AP News Example of word embeddings obtained using our LM on AP News(14M words, V=17k), D=100 The word embedding matrix R was projected in 2D by Stochastic t-SNE[Van der Maaten, JMLR 2008] [Mirowski (2010) “Time series modelling with hidden variables and gradient-based algorithms”, NYU PhD thesis]

  45. Recurrent Neural Net (RNN) language model h W Time-delay 1-layerneural networkwith D output units word embedding space ℜDin dimension D=30 to 250 zt-1 zt Word embedding matrix U V o discrete word space {1, ..., M}M>100k words Handles longer word history(~10 words) as well as 10-gram feed-forward NNLM Training algorithm: BPTTBack-Propagation Through Time wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat Complexity: D×D + D×D + D×V [Mikolov et al, 2010, 2011]

  46. Context-dependent RNN language model h W Time-delay 1-layerneural networkwith D output units word embedding space ℜDin dimension D=200 zt-1 zt F sentence or document topic(K=40 topics) U V f o G discrete word space {1, ..., M}M>100k words Compute topicmodel representationword-by-word on last 50 wordsusing approximate LDA [Blei et al, 2003]with Ktopics.Enables to model long-rangedependencies at sentence level. wt-5 wt-4 wt-3 wt-2 wt-1 wt the cat sat on the mat [Mikolov & Zweig, 2012]

  47. Perplexity of RNN language models Penn TreeBank V=10k vocabulary Train on 900k words Validate on 80k words Test on 80k words AP News V=17k vocabulary Train on 14M words Validate on 1M words Test on 1M words [Mirowski, 2010; Mikolov & Zweig, 2012;RNN toolbox: http://research.microsoft.com/en-us/projects/rnn/default.aspx]

  48. Outline • Probabilistic Language Models (LMs) • Likelihood of a sentence and LM perplexity • Limitations of n-grams • Neural Probabilistic LMs • Vector-space representation of words • Neural probabilistic language model • Log-Bilinear (LBL) LMs • Long-range dependencies • Enhancing LBL with linguistic features • Recurrent Neural Networks (RNN) • Applications • Speech recognition and machine translation • Sentence completion and linguistic regularities • Bag-of-word-vector approaches • Auto-encoders for text • Continuous bag-of-words and skip-gram models • Scalability with large vocabularies • Tree-structured LMs • Noise-contrastive estimation

  49. Performance of LBL on speech recognition HUB-4 TV broadcast transcripts Vocabulary V=25k(with proper nouns & numbers) Train on 1M words Validate on 50k words Test on 800 sentences Re-rank top 100candidate sentences, provided for each spoken sentenceby a speech recognitionsystem (acoustic model + simple trigram) [Mirowski et al, 2010]

  50. Performance of RNN on machine translation RNN with 100 hidden nodes Trained using 20-step BPTT Uses lattice rescoring RNN trained on 2M words already improves over n-gram trained on 1.15B words [Auli et al, 2013] [Image credits: Auli et al (2013) “Joint Language and Translation Modeling with Recurrent Neural Networks”, EMNLP]

More Related