Machine Learning

1. Machine Learning Probability and Bayesian Networks Doug Downey (adapted from Bryan Pardo, Northwestern University)

2. An Introduction • Bayesian Decision Theory came long before Version Spaces, Decision Tree Learning and Neural Networks. It was studied in the field of Statistical Theory and more specifically, in the field of Pattern Recognition. Doug Downey (adapted from Bryan Pardo, Northwestern University)

3. An Introduction • Bayesian Decision Theory is at the basis of important learning schemes such as… • Naïve Bayes Classifier • Bayesian Belief Networks • EM Algorithm • Bayesian Decision Theory is also useful as it provides a framework within which many non-Bayesian classifiers can be studied • See [Mitchell, Sections 6.3, 4,5,6]. Doug Downey (adapted from Bryan Pardo, Northwestern University)

4. Discrete Random Variables • A is a Boolean random variable if it denotes an event where there is uncertainty about whether it occurs • Examples • The next US president will be Barack Obama • You will get an A in the course • P(A) = probability of A = the fraction of all possible worlds where A is true Doug Downey (adapted from Bryan Pardo, Northwestern University)

5. Vizualizing P(A) All Possible Worlds Worlds where A is True Doug Downey (adapted from Bryan Pardo, Northwestern University)

6. Axioms of Probability • Let there be a space S composed of a countable number of events • The probability of each event is between 0 and 1 • The probability of the whole sample space is 1 • When two events are mutually exclusive, their probabilities are additive Doug Downey (adapted from Bryan Pardo, Northwestern University)

7. Vizualizing Two Boolean RVs A B Doug Downey (adapted from Bryan Pardo, Northwestern University)

8. Conditional Probability NOT Independent Can we do the following? The conditional probability of A given B is represented by the following formula A B Only if A and B are independent Doug Downey (adapted from Bryan Pardo, Northwestern University)

9. Independence • variables A and B are said to be independent if knowing the value of A gives you no knowledge about the likelihood of B…and vice-versa P(A|B) = P(A) and P(B|A) = P(B) Doug Downey (adapted from Bryan Pardo, Northwestern University)

10. An Example: Cards • Take a standard deck of 52 cards. • On the first draw I pull the Ace of Spades. • I don’t replace the card. • What is the probability I’ll pull the Ace of Spades on the second draw? • Now, I replace the Ace after the 1st draw, shuffle, and draw again. • What is the chance I’ll draw the Ace of Spades on the 2nd draw? Doug Downey (adapted from Bryan Pardo, Northwestern University)

11. Discrete Random Variables • A is a discrete random variable if it takes a countable number of distinct values • Examples • Your grade G in the course • The number of heads k in n coin flips • P(A=k) = the fraction of all possible worlds where A equals k • Notation: PD(A = k) prob. relative to a distribution D • Pfair grading(G = “A”), Pcheating(G = “A”) Doug Downey (adapted from Bryan Pardo, Northwestern University)

12. Bayes Theorem • Definition of Conditional Probability • Corollary: The Chain Rule • Bayes Rule (Thomas Bayes, 1763) Doug Downey (adapted from Bryan Pardo, Northwestern University)

13. ML in a Bayesian Framework • Any ML technique can be expressed as reasoning about probabilities • Goal: Find hypothesis h that is most probable given training data D • Provides a more explicit way of describing & encoding our assumptions

14. Some Definitions • Prior probability of h, P(h): • The background knowledge we have about the chance that h is a correct hypothesis (before having observed the data). • Prior probability of D, P(D): • the probability that training data D will be observed given no knowledge about which hypothesis h holds. • Conditional Probability of D, P(D|h): • the probability of observing data D given that hypothesis h holds. • Posterior probability of h, P(h|D): • the probability that h is true, given the observed training data D. • the quantity that Machine Learning researchers are interested in. Doug Downey (adapted from Bryan Pardo, Northwestern University)

15. Maximum A Posteriori (MAP) • Goal: To find the most probable hypothesis h from a set of candidate hypotheses H given the observed data D. • MAP Hypothesis, hMAP Doug Downey (adapted from Bryan Pardo, Northwestern University)

16. Maximum Likelihood (ML) • ML hypothesis is a special case of the MAP hypothesis where all hypotheses are equally likely to begin with Doug Downey (adapted from Bryan Pardo, Northwestern University)

17. Example: Brute Force MAP Learning • Assumptions • The training data D is noise-free • The target concept c is in the hypothesis set H • All hypotheses are equally likely • Choice: Probability of D given h Doug Downey (adapted from Bryan Pardo, Northwestern University)

18. Brute Force MAP (continued) Bayes Theorem Given our assumptions VSH,D is the version space Doug Downey (adapted from Bryan Pardo, Northwestern University)

19. Find-S as MAP Learning • We can characterize the FIND-S learner (chapter 2) in Bayesian terms • Again P(D | h) is 1 if h is consistent on D, and 0 otherwise • P(h) increases with… • specificity of h • Then: MAP hypothesis = output of Find-S Doug Downey (adapted from Bryan Pardo, Northwestern University)

20. Neural Nets in a Bayesian Framework • Under certain assumptions regarding noise in the data, minimizing the mean squared error (what multilayer perceptrons do) corresponds to computing the maximum likelihood hypothesis. Doug Downey (adapted from Bryan Pardo, Northwestern University)

21. Least Squared Error = ML f hML e Assume e is drawn from a normal distribution Doug Downey (adapted from Bryan Pardo, Northwestern University)

22. Least Squared Error = ML Doug Downey (adapted from Bryan Pardo, Northwestern University)

23. Least Squared Error = ML Doug Downey (adapted from Bryan Pardo, Northwestern University)

24. Decision Trees in Bayes Framework • Decent choice for P(h): simpler hypotheses have higher probability • Occam’s razor • This can be encoded in terms of finding the “Minimum Description Length” encoding • Provides a way to “trade off” hypothesis size for training error • Potentially prevents overfitting Doug Downey (adapted from Bryan Pardo, Northwestern University)

25. Most Compact Coding • Lets minimize the bits used to encode a message • Idea: • Assign shorter codes to more probable messages • According to Shannon & Weaver • An optimal code assigns –log2P(i) bits to encode item i • thus… Doug Downey (adapted from Bryan Pardo, Northwestern University)

26. Minimum Description Length (MDL) Doug Downey (adapted from Bryan Pardo, Northwestern University)

27. Minimum Description Length (MDL) Doug Downey (adapted from Bryan Pardo, Northwestern University)

28. Minimum Description Length (MDL) Doug Downey (adapted from Bryan Pardo, Northwestern University)

29. What does all that mean? • The “optimal” hypothesis is the one that is the smallest when we count… • How long the hypothesis description must be • How long the data description must be, given the hypothesis • Key idea: since we’re given h, we need only encode h’s mistakes Doug Downey (adapted from Bryan Pardo, Northwestern University)

30. What does all that mean? • If the hypothesis is perfect, we don’t need to encode any data. • For each misclassification, we must • say which item is misclassified • Takes log2m bits, where m = size of the dataset • Say what the right classification is • Takes log2k bits, where k = number of classes Doug Downey (adapted from Bryan Pardo, Northwestern University)

31. The best MDL hypothesis • The best hypothesis is the best tradeoff between • Complexity of the hypothesis description • Number of times we have to tell people where it screwed up. Doug Downey (adapted from Bryan Pardo, Northwestern University)

32. Is MDL always MAP? • Only given significant assumptions: • If we know a representation scheme such that size of h in H is -log2P(h) • Likewise, the size of the exception representation must be –log2P(D|h) • THEN • MDL = MAP Doug Downey (adapted from Bryan Pardo, Northwestern University)

33. Making Predictions The reason we learned h to begin with Does it make sense to choose just one h? h1 :Looks matter h2 :Money matters h3 :Ideas matter Obama Elected President We want a prediction: yes or no? Doug Downey (adapted from Bryan Pardo, Northwestern University)

34. Maximum A Posteriori (MAP) Find most probable hypothesis Use the predictions of that hypothesis h1 :Looks matter h2 :Money matters h3 :Ideas matter …. do we really want to ignore the other hypotheses? Imagine 8 hypotheses. Seven of them say “yes” and have a probability of 0.1 each. One says “no” and has a probability of 0.3. Who do you believe? Doug Downey (adapted from Bryan Pardo, Northwestern University)

35. Bayes Optimal Classifier • Bayes Optimal Classification: The most probable classification of a new instance is obtained by combining the predictions of all hypotheses, weighted by their posterior probabilities: …where V is the set of all the values a classification can take and vis one possible such classification. No other method using the same H and prior knowledge is better (on average). Doug Downey (adapted from Bryan Pardo, Northwestern University)

36. Naïve Bayes Classifier • Unfortunately, Bayes Optimal Classifier is usually too costly to apply! ==> Naïve Bayes Classifier We’ll be seeing more of these… Doug Downey (adapted from Bryan Pardo, Northwestern University)

37. The Joint Distribution Make a truth table listing all combinations of variable values Assign a probability to each row Make sure the probabilities sum to 1 Doug Downey (adapted from Bryan Pardo, Northwestern University)

38. Using The Joint Distribution Find P(A) Sum the probabilities of all rows where A=1 P(A=1) = 0.05+ 0.2 + 0.25+ 0.05 = 0.55 P(A) = Doug Downey (adapted from Bryan Pardo, Northwestern University)

39. Using The Joint Distribution Find P(A|B) P(A=1 | B=1)=P(A=1, B=1)/P(B=1)=(0.25+0.05)/ (0.25+0.05+0.1+0.05) Doug Downey (adapted from Bryan Pardo, Northwestern University)

40. Using The Joint Distribution Are A and B Independent? NO. They are NOT independent Doug Downey (adapted from Bryan Pardo, Northwestern University)

41. Why not use the Joint Distribution? Given m boolean variables, we need to estimate 2m-1 values. 20 yes-no questions = a million values How do we get around this combinatorial explosion? Assume independence of variables!! Doug Downey (adapted from Bryan Pardo, Northwestern University)

42. …back to Independence The probability I have an apple in my lunch bag is independent of the probability of a blizzard in Japan. This is DOMAIN Knowledge, typically supplied by the problem designer Doug Downey (adapted from Bryan Pardo, Northwestern University)

43. Naïve Bayes Classifier • Cases described by a conjunction of attribute values • These attributes are our “independent” hypotheses • The target function has a finite set of values, V • Could be solved using the joint distribution table • What if we have 50,000 attributes? • Attribute j is a Boolean signaling presence or absence of the jth word from the dictionary in my latest email. Doug Downey (adapted from Bryan Pardo, Northwestern University)

44. Naïve Bayes Classifier Doug Downey (adapted from Bryan Pardo, Northwestern University)

45. Naïve Bayes Continued Conditional independence step Instead of one table of size 250000 we have 50,000 tables of size 2 Doug Downey (adapted from Bryan Pardo, Northwestern University)

46. Bayesian Belief Networks • Bayes Optimal Classifier • Often too costly to apply (uses full joint probability) • Naïve Bayes Classifier • Assumes conditional independence to lower costs • This assumption often overly restrictive • Bayesian belief networks • provide an intermediate approach • allows conditional independence assumptions that apply to subsets of the variable. Doug Downey (adapted from Bryan Pardo, Northwestern University)

47. Example • I'm at work, neighbor John calls to say my alarm is ringing, but neighbor Mary doesn't call. Sometimes it's set off by minor earthquakes. Is there a burglar? • Variables: Burglary, Earthquake, Alarm, JohnCalls, MaryCalls • Network topology reflects "causal" knowledge: • A burglar can set the alarm off • An earthquake can set the alarm off • The alarm can cause Mary to call • The alarm can cause John to call Doug Downey (adapted from Bryan Pardo, Northwestern University)

48. Example contd. Doug Downey (adapted from Bryan Pardo, Northwestern University)

49. Bayesian Networks Parents Pa of Alarm P(A | B,E) E B Burglary Earthquake e b 0.95 0.05 e b 0.94 0.06 e b Alarm 0.29 0.01 0.999 e b 0.001 MaryCalls JohnCalls Together: Define a unique distribution in a factored form Quantitative part: Set of conditional probability distributions [Pearl 91] • Qualitative part: • Directed acyclic graph (DAG) • Nodes - random vars. • Edges - direct influence Traditional Approaches Doug Downey (adapted from Bryan Pardo, Northwestern University)

50. Compactness • A CPT for Boolean Xi with k Boolean parents has 2k rows for the combinations of parent values • Each row requires one number p for Xi = true(the number for Xi = false is just 1-p) • If each variable has no more than k parents, the complete network requires O(n · 2k) numbers • I.e., grows linearly with n, vs. O(2n)for the full joint distribution • For burglary net, 1 + 1 + 4+ 2 + 2 = 10 numbers (vs. 25-1 = 31) Doug Downey (adapted from Bryan Pardo, Northwestern University)