Naïve Bayes Classifier

Naïve Bayes Classifier Adopted from slides by Ke Chen from University of Manchester and YangQiu Song from MSRA

Generative vs. Discriminative Classifiers • Training classifiers involves estimating f: X  Y, or P(Y|X) • Discriminative classifiers (also called ‘informative’ by Rubinstein&Hastie): • Assume some functional form for P(Y|X) • Estimate parameters of P(Y|X) directly from training data • Generative classifiers • Assume some functional form for P(X|Y), P(X) • Estimate parameters of P(X|Y), P(X) directly from training data • Use Bayes rule to calculate P(Y|X= xi)

Bayes Formula

Generative Model • Color • Size • Texture • Weight • …

Discriminative Model • Logistic Regression • Color • Size • Texture • Weight • …

Comparison • Generative models • Assume some functional form for P(X|Y), P(Y) • Estimate parameters of P(X|Y), P(Y) directly from training data • Use Bayes rule to calculate P(Y|X= x) • Discriminative models • Directly assume some functional form for P(Y|X) • Estimate parameters of P(Y|X) directly from training data

Probability Basics • Prior, conditional and joint probability for random variables • Prior probability: • Conditional probability: • Joint probability: • Relationship: • Independence: • Bayesian Rule

Probability Basics • Quiz: We have two six-sided dice. When they are tolled, it could end up with the following occurance: (A) dice 1 lands on side “3”, (B) dice 2 lands on side “1”, and (C) Two dice sum to eight. Answer the following questions:

Probabilistic Classification • Establishing a probabilistic model for classification • Discriminative model Discriminative Probabilistic Classifier

Probabilistic Classification • Establishing a probabilistic model for classification (cont.) • Generative model Generative Probabilistic Model for Class 2 Generative Probabilistic Model for Class L Generative Probabilistic Model for Class 1

Probabilistic Classification • MAP classification rule • MAP: Maximum APosterior • Assign x to c* if • Generative classification with the MAP rule • Apply Bayesian rule to convert them into posterior probabilities • Then apply the MAP rule

Naïve Bayes • Bayes classification • Difficulty: learning the joint probability • Naïve Bayes classification • Assumption that all input attributes are conditionally independent! • MAP classification rule: for

Naïve Bayes • Naïve Bayes Algorithm (for discrete input attributes) • Learning Phase: Given a training set S, • Output: conditional probability tables; for elements • Test Phase: Given an unknown instance , • Look up tables to assign the label c* to X’ if

Example • Example: Play Tennis

Example • Learning Phase P(Play=Yes) = 9/14 P(Play=No) = 5/14

Relevant Issues • Violation of Independence Assumption • For many real world tasks, • Nevertheless, naïve Bayes works surprisingly well anyway! • Zero conditional probability Problem • If no example contains the attribute value • In this circumstance, during test • For a remedy, conditional probabilities estimated with

Relevant Issues • Continuous-valued Input Attributes • Numberless values for an attribute • Conditional probability modeled with the normal distribution • Learning Phase: • Output: normal distributions and • Test Phase: • Calculate conditional probabilities with all the normal distributions • Apply the MAP rule to make a decision

Conclusions • Naïve Bayes based on the independence assumption • Training is very easy and fast; just requiring considering each attribute in each class separately • Test is straightforward; just looking up tables or calculating conditional probabilities with normal distributions • A popular generative model • Performance competitive to most of state-of-the-art classifiers even in presence of violating independence assumption • Many successful applications, e.g., spam mail filtering • A good candidate of a base learner in ensemble learning • Apart from classification, naïve Bayes can do more…

Extra Slides

Naïve Bayes (1) • Revisit • Which is equal to • Naïve Bayes assumes conditional independency • Then the inference of posterior is

Naïve Bayes (2) • Training: Observation is multinomial; Supervised, with label information • Maximum Likelihood Estimation (MLE) • Maximum a Posteriori (MAP): put Dirichlet prior • Classification

Naïve Bayes (3) • What if we have continuous Xi？ • Generative training • Prediction

Naïve Bayes (4) • Problems • Features may overlapped • Features may not be independent • Size and weight of tiger • Use a joint distribution estimation (P(X|Y), P(Y))to solve a conditional problem (P(Y|X= x)) • Can we discriminatively train? • Logistic regression • Regularization • Gradient ascent

Naïve Bayes Classifier