1 / 25

Machine Learning in Natural Language More on Discriminative models

Machine Learning in Natural Language More on Discriminative models. Dan Roth University of Illinois, Urbana-Champaign danr@cs.uiuc.edu http://L2R.cs.uiuc.edu/~danr. How to Compare?. Generalization (since the representation is the same) How many examples are needed

hadar
Download Presentation

Machine Learning in Natural Language More on Discriminative models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Machine Learning in Natural LanguageMore on Discriminative models Dan Roth University of Illinois, Urbana-Champaign danr@cs.uiuc.edu http://L2R.cs.uiuc.edu/~danr

  2. How to Compare? • Generalization (since the representation is the same) How many examples are needed to get to a given level of accuracy? • Efficiency How long does it take to learn a hypothesis and evaluate it (per-example)? • Robustness; Adaptation to a new domain, ….

  3. Sentence Representation S= I don’t know whether to laugh or cry • Define a set offeatures: • featuresare relations that hold in the sentence • Map a sentence to its feature-based representation • The feature-based representation will give some of the information in the sentence • Use this as an example to your algorithm -

  4. Sentence Representation S= I don’t know whether to laugh or cry • Define a set offeatures: featuresare relations that hold in the sentence • Conceptually, there are two steps in coming up with a feature-based representation • What are the information sources available? Sensors: words, order of words, properties (?) of words • What features to construct based on these? Why needed?

  5. Whether Weather Embedding New discriminator in functionally simpler

  6. Domain Characteristics • The number of potential features is very large • The instance space is sparse • Decisions depend on a small set of features (sparse) • Want to learn from a number of examples that is small relative to the dimensionality

  7. Generalization • Dominated by the sparseness of the function space Most features are irrelevant # of examples required by multiplicative algorithms depends mostly on # of relevant features (Generalization bounds depend on ||w||;) • Lesser issue: Sparseness of features space: advantage to additive. Generalization depend on ||x|| (Kivinen/Warmuth 95)

  8. Mistakes bounds for 10 of 100 of n Function: At least 10 out of fixed 100 variables are active Dimensionality is n Perceptron,SVMs # of mistakes to convergence Winnow n: Total # of Variables (Dimensionality)

  9. Efficiency • Dominatedby the size of the feature space • Most features are functions (e.g., n-grams) of raw attributes • Additive algorithms allow the use of Kernels No need to explicitly generate the complex features • Could be more efficient since work is done in the original feature space.

  10. SNoW • Update rule: Multiplicative /Additive/NB (+ regularization) • Feature space: Infinite Attribute Space - examples of variable size: only active features - determined in a data driven way • Multi Class Learner • Several approaches are possible • Makes Possible: • Generation of many complex/relational types of features • Only a small fraction is actually represented • Computationally efficient (on-line!)

  11. Other Issues in Classification • Other methods are used broadly today in NLP: • SVM, AdaBoost, • Mutli class classification • Dealing with lack of data: • Semi-supervised learning • Missing data

  12. Whether Weather Embedding New discriminator in functionally simpler

  13. Kernel Based Methods • A method to run Perceptron on a very large feature set, without incurring the cost of keeping a very large weight vector. • Computing the weight vector is done in the original space. • Notice: this pertains only to efficiency. • Generalization is still relative to the real dimensionality. • This is the main trick in SVMs. (Algorithm - different) (although many applications actually use linear kernels).

  14. Kernel Base Methods • Let I be the set t1,t2,t3…of monomials (conjunctions) over The feature space x1, x2… xn. • Then we can write a linear function over this new feature space.

  15. Kernel Based Methods • Great Increase in expressivity • Can run Perceptron (and Winnow) but the convergence bound may suffer exponential growth. • Exponential number of monomials are true in each example. • Also, will have to keep many weights.

  16. The Kernel Trick(1) • Consider the value of w used in the prediction. • Each previous mistake, on example z, makes an additive contribution of +/-1 to w, iff t(z) = 1. • The value of wis determined by the number of mistakes on which t() was satisfied.

  17. The Kernel Trick(2) • P – set of examples on which we Promoted • D – set of examples on which we Demoted • M = P D

  18. The Kernel Trick(3) • P – set of examples on which we Promoted • D – set of examples on which we Demoted • M = P D • Where S(z)=1 if z P and S(z) = -1 if z D. Reordering:

  19. The Kernel Trick(4) • S(y)=1 if y P and S(y) = -1 if y D. • A mistake on z contributes the value +/-1 to all monomials satisfied by z. The total contribution of z to the sum is equal to the number of monomials that satisfy both x and z. • Define a dot product in the t-space: • We get the standard notation:

  20. Kernel Based Methods • What does this representation give us? • We can view this Kernel as the distance between x,z in the t-space. • But, K(x,z) can be measured in the original space, without explicitly writing the t-representation of x, z

  21. Kernel Based Methods • Consider the space of all 3n monomials (allowing both positive and negative literals). • Then, • Where same(x,z) is the number of features that have the same value for both x and z.. We get: • Example: Take n=2; x=(00), z=(01), …. • Other Kernels can be used.

  22. Implementation • Simply run Perceptron in an on-line mode, but keep track of the set M. • Keeping the set M allows to keep track of S(z). • Rather than remembering the weight vector w, • remember the set M (P and D) – all those examples on which we made mistakes. Dual Representation

  23. Summary – Kernel Based Methods I • A method to run Perceptron on a very large feature set, without incurring the cost of keeping a very large weight vector. • Computing the weight vector can still be done in the original feature space. • Notice: this pertains only to efficiency: The classifier is identical to the one you get by blowing up the feature space. • Generalization is still relative to the real dimensionality. • This is the main trick in SVMs. (Algorithm - different) (although most applications actually use linear kernels)

  24. Efficiency-Generalization Tradeoff • There is a tradeoff between the computationalefficiency with which these kernels can be computed and the generalization ability of the classifier. • For example, using such kernels the Perceptron algorithm can make an exponential number of mistakes even when learning simple functions. • In addition, computing with kernels depends strongly on the number of examples. It turns out that sometimes working in the blown up space is more efficient than using kernels. • Next: Kernel methods in NLP

  25. Other Issues in Classification • Other methods are used broadly today in NLP: • SVM, AdaBoost, • Mutliclass classification • Dealing with lack of data: • Semi-supervised learning • Missing data: EM

More Related