Bayes Rule

1. Bayes Rule Rev. Thomas Bayes(1702-1761) • The product rule gives us two ways to factor a joint probability: • Therefore, • Why is this useful? • Can get diagnostic probability P(Cavity | Toothache) from causal probabilityP(Toothache | Cavity) • Can update our beliefs based on evidence • Key tool for probabilistic inference

2. Bayes Rule example • Marie is getting married tomorrow, at an outdoor ceremony in the desert. In recent years, it has rained only 5 days each year (5/365 = 0.014). Unfortunately, the weatherman has predicted rain for tomorrow. When it actually rains, the weatherman correctly forecasts rain 90% of the time. When it doesn't rain, he incorrectly forecasts rain 10% of the time. What is the probability that it will rain on Marie's wedding?

3. Bayes Rule example • Marie is getting married tomorrow, at an outdoor ceremony in the desert. In recent years, it has rained only 5 days each year (5/365 = 0.014). Unfortunately, the weatherman has predicted rain for tomorrow. When it actually rains, the weatherman correctly forecasts rain 90% of the time. When it doesn't rain, he incorrectly forecasts rain 10% of the time. What is the probability that it will rain on Marie's wedding?

4. Bayes rule: Another example • 1% of women at age forty who participate in routine screening have breast cancer.  80% of women with breast cancer will get positive mammographies.9.6% of women without breast cancer will also get positive mammographies.  A woman in this age group had a positive mammography in a routine screening.  What is the probability that she actually has breast cancer?

5. Independence • Two events A and B are independent if and only if P(A  B) = P(A) P(B) • In other words, P(A | B) = P(A) and P(B | A) = P(B) • This is an important simplifying assumption for modeling, e.g., Toothache and Weather can be assumed to be independent • Are two mutually exclusive events independent? • No, but for mutually exclusive events we have P(A  B) = P(A) + P(B) • Conditional independence: A and B are conditionally independent given C iffP(A  B | C) = P(A | C) P(B | C)

6. Conditional independence: Example • Toothache:boolean variable indicating whether the patient has a toothache • Cavity:boolean variable indicating whether the patient has a cavity • Catch: whether the dentist’s probe catches in the cavity • If the patient has a cavity, the probability that the probe catches in it doesn't depend on whether he/she has a toothache P(Catch | Toothache, Cavity) = P(Catch | Cavity) • Therefore, Catchis conditionally independent of Toothachegiven Cavity • Likewise, Toothacheis conditionally independent of Catchgiven Cavity P(Toothache | Catch, Cavity) = P(Toothache | Cavity) • Equivalent statement: P(Toothache, Catch | Cavity) = P(Toothache | Cavity) P(Catch | Cavity)

7. Conditional independence: Example • How many numbers do we need to represent the joint probability table P(Toothache, Cavity, Catch)? 23 – 1 = 7 independent entries • Write out the joint distribution using chain rule: P(Toothache, Catch, Cavity) = P(Cavity) P(Catch | Cavity) P(Toothache | Catch, Cavity) = P(Cavity) P(Catch | Cavity) P(Toothache | Cavity) • How many numbers do we need to represent these distributions? 1 + 2 + 2 = 5 independent numbers • In most cases, the use of conditional independence reduces the size of the representation of the joint distribution from exponential in n to linear in n

8. Probabilistic inference • Suppose the agent has to make a decision about the value of an unobserved query variable X given some observed evidence variable(s)E = e • Partially observable, stochastic, episodic environment • Examples: X = {spam, not spam}, e = email messageX = {zebra, giraffe, hippo}, e = image features • Bayes decision theory: • The agent has a loss function, which is 0 if the value of X is guessed correctly and 1 otherwise • The estimate of X that minimizes expected loss is the one that has the greatest posterior probability P(X = x | e) • This is the Maximum a Posteriori (MAP) decision

9. MAP decision • Value x of X that has the highest posterior probability given the evidence E = e: • Maximum likelihood (ML) decision: posterior likelihood prior

10. Naïve Bayes model • Suppose we have many different types of observations (symptoms, features) E1, …, En that we want to use to obtain evidence about an underlying hypothesis X • MAP decision involves estimating • If each feature Fi can take on k values, how many entries are in the (conditional) joint probability table P(E1, …, En |X = x)?

11. Naïve Bayes model • Suppose we have many different types of observations (symptoms, features) E1, …, En that we want to use to obtain evidence about an underlying hypothesis X • MAP decision involves estimating • We can make the simplifying assumption that the different features are conditionally independent given the hypothesis: • If each feature can take on k values, what is the complexity of storing the resulting distributions?

12. Naïve Bayes Spam Filter • MAP decision: to minimize the probability of error, we should classify a message as spam if P(spam | message) > P(¬spam | message)

13. Naïve Bayes Spam Filter • MAP decision: to minimize the probability of error, we should classify a message as spam if P(spam | message) > P(¬spam | message) • We have P(spam | message) P(message | spam)P(spam)and ¬P(spam | message) P(message | ¬spam)P(¬spam)

14. Naïve Bayes Spam Filter • We need to find P(message | spam) P(spam) and P(message | ¬spam) P(¬spam) • The message is a sequence of words (w1, …, wn) • Bag of words representation • The order of the words in the message is not important • Each word is conditionally independent of the others given message class (spam or not spam)

15. Naïve Bayes Spam Filter • We need to find P(message | spam) P(spam) and P(message | ¬spam) P(¬spam) • The message is a sequence of words (w1, …, wn) • Bag of words representation • The order of the words in the message is not important • Each word is conditionally independent of the others given message class (spam or not spam) • Our filter will classify the message as spam if

16. Bag of words illustration US Presidential Speeches Tag Cloudhttp://chir.ag/projects/preztags/

17. Bag of words illustration US Presidential Speeches Tag Cloudhttp://chir.ag/projects/preztags/

18. Bag of words illustration US Presidential Speeches Tag Cloudhttp://chir.ag/projects/preztags/

19. Naïve Bayes Spam Filter posterior prior likelihood

20. Parameter estimation • In order to classify a message, we need to know the prior P(spam) and the likelihoods P(word | spam) and P(word | ¬spam) • These are the parameters of the probabilistic model • How do we obtain the values of these parameters? prior P(word | spam) P(word | ¬spam) spam: 0.33 ¬spam: 0.67

21. Parameter estimation • How do we obtain the prior P(spam) and the likelihoods P(word | spam) and P(word | ¬spam)? • Empirically: use training data • This is the maximum likelihood (ML) estimate, or estimate that maximizes the likelihood of the training data: # of word occurrences in spam messages P(word | spam) = total # of words in spam messages d: index of training document, i: index of a word

22. Parameter estimation • How do we obtain the prior P(spam) and the likelihoods P(word | spam) and P(word | ¬spam)? • Empirically: use training data • Parameter smoothing: dealing with words that were never seen or seen too few times • Laplacian smoothing: pretend you have seen every vocabulary word one more time than you actually did # of word occurrences in spam messages P(word | spam) = total # of words in spam messages # of word occurrences in spam messages + 1 P(word | spam) = total # of words in spam messages + V (V: total number of unique words)

23. Summary of model and parameters • Naïve Bayes model: • Model parameters: Likelihood of spam Likelihood of ¬spam prior P(w1 | spam) P(w2 | spam) … P(wn | spam) P(w1 | ¬spam) P(w2 | ¬spam) … P(wn | ¬spam) P(spam) P(¬spam)

24. Bag-of-word models for images Csurka et al. (2004), Willamowski et al. (2005), Grauman & Darrell (2005), Sivic et al. (2003, 2005)

25. Bag-of-word models for images • Extract image features

26. Bag-of-word models for images • Extract image features

27. Bag-of-word models for images • Extract image features • Learn “visual vocabulary”

28. Bag-of-word models for images • Extract image features • Learn “visual vocabulary” • Map image features to visual words

29. Bayesian decision making: Summary • Suppose the agent has to make decisions about the value of an unobserved query variable X based on the values of an observed evidence variableE • Inference problem: given some evidence E = e, what is P(X | e)? • Learning problem: estimate the parameters of the probabilistic model P(X | E) given a training sample{(x1,e1), …, (xn,en)}