130 likes | 234 Views
Information Bottleneck versus Maximum Likelihood. Felix Polyakov. Likelihood of the Data. Probability of a head. A simple example. A coin is known to be biased The coin is tossed three times – two heads and one tail Use ML to estimate the probability of throwing a head. Model:
E N D
Information Bottleneck versus Maximum Likelihood Felix Polyakov
Likelihood of the Data Probability of a head A simple example... A coin is known to be biased The coin is tossed three times – two heads and one tail Use ML to estimate the probability of throwing a head • Model: • p(head) = P • p(tail) = 1 - P • Try P = 0.2 L(O) = 0.2 * 0.2 * 0.8 = 0.032 • Try P = 0.4 L(O) = 0.4 * 0.4 * 0.6 = 0.096 • Try P = 0.6 L(O) = 0.6 * 0.6 * 0.4 = 0.144 • Try P = 0.8 L(O) = 0.8 * 0.8 * 0.2 = 0.128
A bit more complicated example… :Mixture Model • Three baskets with white (O= 1), grey (O = 2), and black (O = 3) balls B1 B2 B3 • 15 balls were drawn as follows: • Choose a basket according to p(i) = bi • Draw the ball j from basket i with probability • Use ML to estimate given the observations: sequence of balls’ colors
Likelihood of observations • Log Likelihood of observations • Maximal Likelihood of observations
Likelihood of the observed data • x – hidden random variables [e.g. basket] • y – observed random variables [e.g. color] • - model parameters [e.g. they define p(y|x)] 0 – current estimate of model parameters
2. Maximization • EM algorithm converges to local maxima Expectation-maximization algorithm (I) • Expectation • Compute • Get
Jensen’s inequality for concave function EM – another approach • Goal:
2. Maximization Expectation-maximization algorithm (II) • Expectation (I) and (II) are equivalent
Scheme of the approach Expectation Maximization