1 / 34

So far…

So far…. Bayesian Learning What does it mean to be Bayesian? Na ï ve Bayes Independence assumptions EM Algorithm Learning with hidden variables Today: Representing arbitrary probability distributions Inference Exact inference; Approximate inference

vanig
Download Presentation

So far…

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. So far… • Bayesian Learning • What does it mean to be Bayesian? • Naïve Bayes • Independence assumptions • EM Algorithm • Learning with hidden variables • Today: • Representing arbitrary probability distributions • Inference • Exact inference; Approximate inference • Learning Representations of Probability Distributions

  2. Unsupervised Learning • We get as input (n+1) tuples: (X1, X2, … Xn, Xn+1) • There is no notion of a class variable or a label. • After seeing a few examples, we would like to know something about the domain: • correlations between variables, probability of certain events, etc. • We want to learn the most likely model that generated the data

  3. Unsupervised Learning • In general, the problem is very hard. But, under some assumptions on the distribution we have shown that we can do it. (exercise: show it’s the most likely distribution) • Assumptions: (conditional independence given y) • P(xi | xj,y) = P(xi|y) 8i,j • Can these assumptions be relaxed ? • Can we learn more general probability distributions ? • (These are essential in many applications: language, vision.) We can compute the probability of any event or conditional event over the n+1 variables.

  4. Graphical Models of Probability Distributions • Bayesian Networks represent the joint probability distribution over a set of variables. • Independence Assumption: x is independent of its non-descendants given its parents • With these conventions, the joint probability distribution is given by: Y z is a parent of x x is a descendant of y This is a theorem. The terms are called CPTs (Conditional Probability tables) and they completely define the probability distribution. Z Z1 Z2 Z3 X10 X X2

  5. Graphical Models of Probability Distributions • For general Bayesian Networks • The learning problem is hard • The inference problem (given the network, evaluate the probability of a given event) is hard (#P Complete) P(y) Y Z P(z3 | y) Z1 Z2 Z3 X2 X10 X P(x | z1, z2 ,z, z3)

  6. Tree Dependent Distributions Y P(y) • Directed Acyclic graph • Each node has at most one parent • Independence Assumption: • x is independent of its non-descendants given its parents • (x is independent of other nodes give z; v is independent of w given u;) • Need to know two numbers for each link: p(x|z), and a prior for the root p(y) P(s|y) Z S U W P(x|z) T X V

  7. Tree Dependent Distributions Y P(y) • This is a generalization of naïve Bayes. • Inference Problem: • Given the Tree with all the associated probabilities, evaluate the probability of an event p(x) ? P(s|y) Z S U W P(x|z) T X V • P(x=1) = = P(x=1|z=1)P(z=1) + P(x=1|z=0)P(z=0) • Recursively, go up the tree: P(z=1) = P(z=1|y=1)P(y=1) + P(z=1|y=0)P(y=0) P(z=0) = P(z=0|y=1)P(y=1) + P(z=0|y=0)P(y=0) Linear Time Algorithm Now we have everything in terms of the CPTs (conditional probability tables)

  8. Tree Dependent Distributions Y P(y) • This is a generalization of naïve Bayes. • Inference Problem: • Given the Tree with all the associated probabilities, evaluate the probability of an event p(x,y) ? P(s|y) Z S U W P(x|z) T X V • P(x=1,y=0) = = P(x=1|y=0)P(y=0) • Recursively, go up the tree along the path from x to y: P(x=1|y=0) = z=0,1 P(x=1|y=0, z)P(z|y=0) = = z=0,1P(x=1|z)P(z|y=0) Now we have everything in terms of the CPTs (conditional probability tables)

  9. Tree Dependent Distributions Y P(y) • This is a generalization of naïve Bayes. • Inference Problem: • Given the Tree with all the associated probabilities, evaluate the probability of an event p(x,u) ? • (No direct path from x to u) P(s|y) Z S U W P(x|z) T X V • P(x=1,u=0) = P(x=1|u=0)P(u=0) • Let y be a parent of x and u (we always have one) P(x=1|u=0) = y=0,1 P(x=1|u=0, y)P(y|u=0) = = y=0,1 P(x=1|y)P(y|u=0) = Now we have reduced it to cases we have seen

  10. Tree Dependent Distributions Y P(y) • Inference Problem: • Given the Tree with all the associated probabilities, we “showed” that we can evaluate the probability of all events efficiently. • There are more efficient algorithms • The idea was to show that the inference is this case is a simple application of Bayes rule and probability theory. P(s|y) Z S U W P(x|z) T X V • Projects • Presentation on 12/15 9am • Final Exam 12/11, in 340x • FinalPractice 12/6 • Problem set 12/7 • No time extension

  11. Tree Dependent Distributions Y P(y) • Learning Problem: • Given data (n tuples) assumed to be sampled from a tree-dependent distribution • What does that mean? • Generative model • Find the tree representation of the distribution. • What does that mean? P(s|y) Z S U W P(x|z) T X V • Among all trees, find the most likely one, given the data: P(T|D) = P(D|T) P(T)/P(D)

  12. Tree Dependent Distributions Y P(y) • Learning Problem: • Given data (n tuples) assumed to be sampled from a tree-dependent distribution • Find the tree representation of the distribution. P(s|y) Z S U W P(x|z) T X V • Assuming uniform prior on trees, the Maximum Likelihood approach is to maximize P(D|T), TML = argmaxT P(D|T) = argmaxT¦ {x} PT (x1, x2, … xn) • Now we can see why we had to solve the inference problem first; it is required for learning.

  13. Tree Dependent Distributions Y P(y) • Learning Problem: • Given data (n tuples) assumed to be sampled from a tree-dependent distribution • Find the tree representation of the distribution. P(s|y) Z S U W P(x|z) T X V • Assuming uniform prior on trees, the Maximum Likelihood approach is to maximize P(D|T), TML = argmaxT P(D|T) = argmaxT¦ {x} PT (x1, x2, … xn) = = argmaxT¦ {x}¦iPT (xi|Parents(xi)) • Try this for naïve Bayes

  14. Example: Learning Distributions Are these representations of the same distribution? Given a sample, which of these generated it? • Probability Distribution 1: 0000 0.1 0001 0.1 0010 0.1 0011 0.1 0100 0.1 0101 0.1 0110 0.1 0111 0.1 1000 0 1001 0 1010 0 1011 0 1100 0.05 1101 0.05 1110 0.05 1111 0.05 • Probability Distribution 2: • Probability Distribution 3 X4 P(x4) P(x3|x4) P(x2|x4) P(x1|x4) X2 X3 X1 P(x4) X4 P(x2|x4) X2 X1 P(x1|x4) X3 P(x3|x2)

  15. Example: Learning Distributions We are given 3 data points: 1011; 1001; 0100 Which one is the target distribution? • Probability Distribution 1: 0000 0.1 0001 0.1 0010 0.1 0011 0.1 0100 0.1 0101 0.1 0110 0.1 0111 0.1 1000 0 1001 0 1010 0 1011 0 1100 0.05 1101 0.05 1110 0.05 1111 0.05 • Probability Distribution 2: • Probability Distribution 3 X4 P(x4) P(x3|x4) P(x2|x4) P(x1|x4) X2 X3 X1 P(x4) X4 P(x2|x4) X2 X1 P(x1|x4) X3 P(x3|x2)

  16. Example: Learning Distributions We are given 3 data points: 1011; 1001; 0100 Which one is the target distribution? • Probability Distribution 1: 0000 0.1 0001 0.1 0010 0.1 0011 0.1 0100 0.1 0101 0.1 0110 0.1 0111 0.1 1000 0 1001 0 1010 0 1011 0 1100 0.05 1101 0.05 1110 0.05 1111 0.05 • What is the likelihood that this table generated the data? P(T|D) = P(D|T) P(T)/P(D) • Likelihood(T) = P(D|T) = P(1011|T) P(1001|T)P(0100|T) • P(1011|T)= 0 • P(1001|T)= 0.1 • P(0100|T)= 0.1 • P(Data|Table)=0

  17. Example: Learning Distributions X4 P(x4) • Probability Distribution 2: • What is the likelihood that the data was sampled from Distribution 2? • Need to define it: • P(x4=1)=1/2 • p(x1=1|x4=0)=1/2 p(x1=1|x4=1)=1/2 • p(x2=1|x4=0)=1/3 p(x2=1|x4=1)=1/3 • p(x3=1|x4=0)=1/6 p(x3=1|x4=1)=5/6 • Likelihood(T) = P(D|T) = P(1011|T) P(1001|T)P(0100|T) • P(1011|T)= p(x4=1)p(x1=1|x4=1)p(x2=0|x4=1)p(x3=1|x4=1)=1/2 1/2 1/3 5/6= 5/72 • P(1001|T)= = 1/2 1/2 2/3 5/6=10/72 • P(0100|T)= =1/2 1/2 2/3 5/6=10/72 • P(Data|Tree)=125/4*36 P(x3|x4) P(x2|x4) P(x1|x4) X2 X3 X1

  18. Example: Learning Distributions P(x4) X4 • Probability Distribution 3: • What is the likelihood that the data was sampled from Distribution 2? • Need to define it: • P(x4=1)=2/3 • p(x1=1|x4=0)=1/3 p(x1=1|x4=1)=1 • p(x2=1|x4=0)=1p(x2=1|x4=1)=1/2 • p(x3=1|x2=0)=2/3 p(x3=1|x2=1)=1/6 • Likelihood(T) = P(D|T) = P(1011|T) P(1001|T)P(0100|T) • P(1011|T)= p(x4=1)p(x1=1|x4=1)p(x2=0|x4=1)p(x3=1|x2=1)=2/3 1 1/2 2/3= 2/9 • P(1001|T)= = 2/3 1 1/2 1/3=1/9 • P(0100|T)= =1/3 2/3 1 5/6=10/54 • P(Data|Tree)=10/37 P(x2|x4) X2 X1 P(x1|x4) X3 P(x3|x2) Distribution 2 is the most likely distribution to have produced the data.

  19. Example: Summary • We are now in the same situation we were when we decided which of two coins, fair (0.5,0.5) or biased (0.7,0.3) generated the data. • But, this isn’t the most interesting case. • In general, we will not have a small number of possible distributions to choose from, but rather a parameterized family of distributions. (analogous to a coin with p 2 [0,1] ) • We need a systematic way to search this family of distributions.

  20. Example: Summary • First, let’s make sure we understand what we are after. • We have 3 data points that have been generated according to our target distribution: 1011; 1001; 0100 • What is the target distribution ? • We cannot find THE target distribution. • What is our goal? • As before – we are interested in generalization – • Given the above 3 data points, we would like to know P(1111) or P(11**), P(***0) etc. • We could compute it directly from the data, but…. • Assumptions about the distribution are crucial here

  21. Learning Tree Dependent Distributions Y P(y) • Learning Problem: • 1. Given data (n tuples) assumed to be sampled from a tree-dependent distribution • find the most probable tree representation of the distribution. • 2. Given data (n tuples) • find the tree representation that best approximates the distribution (without assuming that the data is sampled from a tree-dependent distribution.) P(s|y) Z S U W P(x|z) T X V • The simple minded algorithm for learning a tree dependent distribution requires • (1) for each tree, compute its likelihood L(T) = P(D|T) = = ¦ {x} PT (x1, x2, … xn) = = ¦{x}¦iPT (xi|Parents(xi)) • (2) Find the maximal one

  22. 1. Distance Measure • To measure how well a probability distribution P is approximated by probability distribution T we use here the Kullback-Leibler cross entropy measure (KL-divergence): • Non negative. • D(P,T)=0 iff P and T are identical • Non symmetric. Measures how much P differs from T.

  23. 2. Ranking Dependencies • Intuitively, the important edges to keep in the tree are edges (x---y) for x, y which depend on each other. • Given that the distance between the distribution is measured using the KL divergence, the corresponding measure of dependence is the mutual information between x and y, (measuring the information x gives about y) • which we can estimate with respect to the empirical distribution (that is, the given data).

  24. Learning Tree Dependent Distributions • The algorithm is given mindependent measurements from P. • For each variable x, estimate P(x) (Binary variables – n numbers) • For each pair of variables x, y, estimate P(x,y) (O(n2) numbers) • For each pair of variables compute the mutual information • Build a complete undirected graph with all the variables as vertices. • Let I(x,y) be the weights of the edge (x,y) • Build a maximum weighted spanning tree

  25. Spanning Tree • Sort the weights • Start greedily with the largest one. • Add the next largest as long as it does not create a loop. • In case of a loop, discard this weight and move on to the next weight. • This algorithm will create a tree; • It is a spanning tree, in the sense that it touches all the vertices. • It is not hard to see that this is the maximum weighted spanning tree • The complexity is O(n2 log(n))

  26. Learning Tree Dependent Distributions • The algorithm is given mindependent measurements from P. • For each variable x, estimate P(x) (Binary variables – n numbers) • For each pair of variables x, y, estimate P(x,y) (O(n2) numbers) • For each pair of variables compute the mutual information • Build a complete undirected graph with all the variables as vertices. • Let I(x,y) be the weights of the edge (x,y) • Build a maximum weighted spanning tree • Transform the resulting undirected tree to a directed tree. • Choose a root variable and set the direction of all the edges away from it. • Place the corresponding conditional probabilities on the edges. (2) (3) (1)

  27. Correctness (1) • Place the corresponding conditional probabilities on the edges. • Given a tree t, defining probability distribution Tby forcing the conditional probabilities along the edges to coincide with those computed from a sample taken from P, gives the best tree dependent approximation to P • Let Tbe the tree-dependent distribution according to the fixed tree t. T(x) = T(xi|Parent(xi)) = P(xi| (xi)) • Recall:

  28. Correctness (1) • Place the corresponding conditional probabilities on the edges. • Given a tree t, defining Tby forcing the conditional probabilities along the edges to coincide with those computed from a sample taken from P, gives the best t-dependent approximation to P • When is this maximized? • That is, how to define T(xi|(xi))? Slight abuse of notation at the root

  29. Correctness (1) P(xi|¼(xi) log T(xi|¼(xi)) takes its maximal value when we set: T(xi|¼(xi)) = P(xi|¼(xi)) Definition of expectation:

  30. Correctness (2) • Let I(x,y) be the weights of the edge (x,y). Maximizing the sum of the information gains minimizes the distributional distance. • We showed that: • However: • This gives: D(P,T) = -H(x) - 1,n I(xi,¼(xi)) - 1,nx P(xi) log P(xi) • 1st and 3rd term do not depend on the tree structure. Since the distance is non negative, minimizing it is equivalent to maximizing the sum of the edges weights.

  31. Correctness (2) • Let I(x,y) be the weights of the edge (x,y). Maximizing the sum of the information gains minimizes the distributional distance. • We showed that the T is the best tree approximation of P if it is chosen to maximize the sum of the edges weights. D(P,T) = -H(x) - 1,nI(xi,¼(xi)) - 1,nxP(xi) log P(xi) • The minimization problem is solved without the need to exhaustively consider all possible trees. • This was achieved since we transformed the problem of finding the best tree to that of finding the heaviest one, with mutual information on the edges.

  32. Correctness (3) • Transform the resulting undirected tree to a directed tree. (Choose a root variable and direct of all the edges away from it.) • What does it mean that you get the same distribution regardless of the chosen root? (Exercise) • This algorithm learns the best tree-dependent approximation of a distribution D. L(T) = P(D|T) = ¦ {x}¦i PT (xi|Parent(xi)) • Given data, this algorithm finds the tree that maximizes the likelihood of the data. • The algorithm is called the Chow-Liu Algorithm. Suggested in 1968 in the context of data compression, and adapted by Pearl to Bayesian Networks. Invented a couple more times, and generalized since then.

  33. Example: Learning tree Dependent Distributions • We have 3 data points that have been generated according to the target distribution: 1011; 1001; 0100 • We need to estimatesome parameters: • P(A=1) = 2/3, P(B=1)=1/3, P(C=1)=1/3), P(D=1)=2/3 • For the values 00, 01, 10, 11 respectively, we have that: • P(A,B)=0; 1/3; 2/3; 0 P(A,B)/P(A)P(B)=0; 3; 3/2; 0 I(A,B) ~ 9/2 • P(A,C)=1/3; 0; 1/3; 1/3 P(A,C)/P(A)P(C)=3/2; 0; 3/4; 3/2 I(A,C) ~ 15/4 • P(A,D)=1/3; 0; 0; 2/3 P(A,D)/P(A)P(D)=3; 0; 0; 3/2 I(A,D) ~ 9/2 • P(B,C)=1/3; 1/3; 1/3;0 P(B,C)/P(B)P(C)=3/4; 3/2; 3/2; 0 I(B,C) ~ 15/4 • P(B,D)=0; 2/3; 1/3;0 P(B,D)/P(B)P(D)=0; 3; 3/2; 0 I(B,D) ~ 9/2 • P(C,D)=1/3; 1/3; 0; 1/3 P(C,D)/P(C)P(D)=3/2; 3/4; 0; 3/2 I(C,D) ~ 15/4 • Generate the tree; place probabilities.

  34. Learning tree Dependent Distributions • Chow-Liu algorithm finds the tree that maximizes the likelihood. • In particular, if D is a tree dependent distribution, this algorithm learns D. (what does it mean ?) • Less is known on how many examples are needed in order for it to converge. (what does that mean?) • Notice that we are taking statistics to estimate the probabilities of some event in order to generate the tree. Then, we intend to use it to evaluate the probability of other events. • One may ask the question: why do we need this structure ? Why can’t answer the query directly from the data ? • (Almost like making prediction directly from the data in the badges problem)

More Related