Linguistic Regularities in Sparse and Explicit Word Representations

1. Linguistic Regularities in Sparse and Explicit Word Representations Omer Levy Yoav Goldberg Bar-Ilan University Israel

2. Papers in ACL 2014* * Sampling error: +/- 100%

3. Neural Embeddings • Dense vectors • Each dimension is a latent feature • Common software package: word2vec • “Magic” king man woman queen (analogies)

4. Representing words as vectors is not new!

5. Explicit Representations (Distributional) • Sparse vectors • Each dimension is an explicit context • Common association metric: PMI, PPMI • Does the same “magic” work for explicit representations too? • Baroni et al. (2014) showed that embeddings outperform explicit, but…

6. Questions • Are analogies unique to neural embeddings? Compare neural embeddings with explicit representations • Why does vector arithmetic reveal analogies? Unravel the mystery behind neural embeddings and their “magic”

7. Background

8. Mikolov et al. (2013a,b,c) • Neural embeddings have interesting geometries

10. Mikolov et al. (2013a,b,c) • Neural embeddings have interesting geometries • These patterns capture “relational similarities” • Can be used to solve analogies: man is to woman as king is to queen

11. Mikolov et al. (2013a,b,c) • Neural embeddings have interesting geometries • These patterns capture “relational similarities” • Can be used to solve analogies: is to as is to • Can be recovered by “simple” vector arithmetic:

12. Mikolov et al. (2013a,b,c) • Neural embeddings have interesting geometries • These patterns capture “relational similarities” • Can be used to solve analogies: is to as is to • With simple vector arithmetic:

13. Mikolov et al. (2013a,b,c)

14. Mikolov et al. (2013a,b,c)

15. Mikolov et al. (2013a,b,c) king man woman queen

16. Mikolov et al. (2013a,b,c) Tokyo Japan France Paris

17. Mikolov et al. (2013a,b,c) best good strong strongest

18. Mikolov et al. (2013a,b,c) best good strong strongest vectors in

19. Are analogies unique to neural embeddings?

20. Are analogies unique to neural embeddings? • Experiment: compare embeddings to explicit representations

21. Are analogies unique to neural embeddings? • Experiment: compare embeddings to explicit representations

22. Are analogies unique to neural embeddings? • Experiment: compare embeddings to explicit representations • Learn different representations from the samecorpus:

23. Are analogies unique to neural embeddings? • Experiment: compare embeddings to explicit representations • Learn different representations from the samecorpus: • Evaluate with the samerecovery method:

24. Analogy Datasets • 4 words per analogy: is to as is to • Given 3 words: is to as is to • Guess the best suiting from the entire vocabulary • Excluding the question words • MSR:8000 syntactic analogies • Google:19,000 syntactic and semantic analogies

25. Embedding vs Explicit (Round 1)

26. Embedding vs Explicit (Round 1) Many analogies recovered by explicit, but many more by embedding.

27. Why does vector arithmetic reveal analogies?

28. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity: Problem: one similarity might dominate the rest.

29. Why does vector arithmetic reveal analogies? • We wish to find the closest to

30. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity:

31. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity:

32. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity: vector arithmetic similarity arithmetic

33. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity: vector arithmetic similarity arithmetic

34. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity: vector arithmetic similarityarithmetic

35. Why does vector arithmetic reveal analogies? • We wish to find the closest to • This is done with cosine similarity: vector arithmetic similarityarithmetic royal? female?

36. What does each similarity term mean? • Observe the joint features with explicit representations!

37. Can we do better?

38. Let’s look at some mistakes…

39. Let’s look at some mistakes… England London Baghdad ?

40. Let’s look at some mistakes… England London Baghdad Iraq

41. Let’s look at some mistakes… England London Baghdad Mosul?

42. The Additive Objective

43. The Additive Objective

44. The Additive Objective

45. The Additive Objective

46. The Additive Objective

47. The Additive Objective • Problem: one similarity might dominate the rest • Much more prevalent in explicit representation • Might explain why explicit underperformed

48. How can we do better?

49. How can we do better? • Instead of adding similarities, multiply them!

50. How can we do better? • Instead of adding similarities, multiply them!