Maximum Likelihood

1. Maximum Likelihood Benjamin Neale Boulder Workshop 2012

2. We will cover • Easy introduction to probability • Rules of probability • How to calculate likelihood for discrete outcomes • Confidence intervals in likelihood • Likelihood for continuous data

3. Starting simple • Let’s think about probability

4. Starting simple • Let’s think about probability • Coin tosses • Winning the lottery • Roll of the die • Roulette wheel

5. Starting simple • Let’s think about probability • Coin tosses • Winning the lottery • Roll of the die • Roulette wheel • Chance of an event occurring

6. Starting simple • Let’s think about probability • Coin tosses • Winning the lottery • Roll of the die • Roulette wheel • Chance of an event occurring • Written as P(event) = probability of the event

7. Simple probability calculations • To get comfortable with probability, let’s solve these problems: • Probability of rolling an even number on a six-sided die • Probability of pulling a club from a deck of cards

8. Simple probability calculations • To get comfortable with probability, let’s solve these problems: • Probability of rolling an even number on a six-sided die ½ or 0.5 • Probability of pulling a club from a deck of cards ¼ or 0.25

9. Simple probability rules • P(A and B) = P(A)*P(B)

10. Simple probability rules • P(A and B) = P(A)*P(B) • E.g. what is the probability of tossing 2 heads in a row?

11. Simple probability rules • P(A and B) = P(A)*P(B) • E.g. what is the probability of tossing 2 heads in a row? • A = Heads and B = Heads so,

12. Simple probability rules • P(A and B) = P(A)*P(B) • E.g. what is the probability of tossing 2 heads in a row? • A = Heads and B = Heads so, • P(A) = ½, P(B) = ½,P(A and B) = ¼

13. Simple probability rules • P(A and B) = P(A)*P(B) • E.g. what is the probability of tossing 2 heads in a row? • A = Heads and B = Heads so, • P(A) = ½, P(B) = ½, P(A and B) = ¼ *We assume independence

14. Simple probability rules cnt’d • P(A or B) = P(A) + P(B) – P(A and B)

15. Simple probability rules cnt’d • P(A or B) = P(A) + P(B) – P(A and B) • What is the probability of rolling a 1 or a 4?

16. Simple probability rules cnt’d • P(A or B) = P(A) + P(B) – P(A and B) • What is the probability of rolling a 1 or a 4? • A = rolling a 1 and B = rolling a 4

17. Simple probability rules cnt’d • P(A or B) = P(A) + P(B) – P(A and B) • What is the probability of rolling a 1 or a 4? • A = rolling a 1 and B = rolling a 4 • P(A) = , P(B) = , P(A or B) = 1 1 1 6 6 3

18. Simple probability rules cnt’d • P(A or B) = P(A) + P(B) – P(A and B) • What is the probability of rolling a 1 or a 4? • A = rolling a 1 and B = rolling a 4 • P(A) = , P(B) = , P(A or B) = 1 1 1 6 6 3 *We assume independence

19. Recap of rules • P(A and B) = P(A)*P(B) • P(A or B) = P(A) + P(B) – P(A and B) • Sometimes things are ‘exclusive’ such as rolling a 6 and rolling a 4. It cannot occur in the same trial implies P(A and B) = 0 Assuming independence

20. Conditional probabilities • P(X | Y) = the probability of X occurring given Y.

21. Conditional probabilities • P(X | Y) = the probability of X occurring given Y. • Y can be another event (perhaps that predicts X)

22. Conditional probabilities • P(X | Y) = the probability of X occurring given Y. • Y can be another event (perhaps that predicts X) • Y can be a probability or set of probabilities

23. Conditional probabilities • Roll two dice in succession

24. 1 12 Conditional probabilities • Roll two dice in succession • P(total = 10) =

25. 1 12 Conditional probabilities • Roll two dice in succession • P(total = 10) = • What is P(total = 10 | 1st die = 5)?

26. 1 1 12 6 Conditional probabilities • Roll two dice in succession • P(total = 10) = • What is P(total = 10 | 1st die = 5)? • P(total = 10 | 1st die = 5) =

27. Binomial probabilities • Used for two conditions such as coin toss • Determine the chance of any outcome:

28. Binomial probabilities • Used for two conditions such as coin toss • Determine the chance of any outcome:

29. Binomial probabilities • Used for two conditions such as coin toss • Determine the chance of any outcome: Probability # of k results # of trials

30. Binomial probabilities • Used for two conditions such as coin toss • Determine the chance of any outcome: Number of combinations of n choose k ! = factorial; n! = n*(n-1)*(n-2)*…*2*1 and factorials are bad for big numbers

31. Binomial probabilities • Used for two conditions such as coin toss • Determine the chance of any outcome: Probability of not k occurring Probability of k occurring

32. Binomial probabilities • Used for two conditions such as coin toss • Determine the chance of any outcome: Probability of not k occurring Number of combinations of n choose k Probability # of positive results Probability of k occurring # of trials ! = factorial; n! = n*(n-1)*(n-2)*…*2*1 and factorials are bad for big numbers

33. Combinations piece long way • Does it work? Let’s try: How many combinations for 3 heads out of 5 tosses?

34. Combinations piece long way • Does it work? Let’s try: How many combinations for 3 heads out of 5 tosses? • HHHTT, HHTHT, HHTTH, HTHHT, HTHTH, HTTHH, THHHT, THHTH, THTHH, TTHHH = 10 possible combinations

35. Combinations piece formula • Does it work? Let’s try: How many combinations for 3 heads out of 5 tosses? • We have 5 choose 3 = 5!/(3!)*(2!) • =(5*4)/2 • =10

36. Probability roundup • We assumed the ‘true’ parameter values • E.g. P(Heads) = P(Tails) = ½

37. Probability roundup • We assumed the ‘true’ parameter values • E.g. P(Heads) = P(Tails) = ½ • What happens if we have data and want to determine the parameter values?

38. Probability roundup • We assumed the ‘true’ parameter values • E.g. P(Heads) = P(Tails) = ½ • What happens if we have data and want to determine the parameter values? • Likelihood works the other way round: what is the probability of the observed data given parameter values?

39. Concrete example • Likelihood aims to calculate the range of probabilities for observed data, assuming different parameter values.

40. Concrete example • Likelihood aims to calculate the range of probabilities for observed data, assuming different parameter values. • The set of probabilities is referred to as a likelihood surface

41. Concrete example • Likelihood aims to calculate the range of probabilities for observed data, assuming different parameter values. • The set of probabilities is referred to as a likelihood surface • We’re going to generate the likelihood surface for a coin tossing experiment

42. Concrete example • Likelihood aims to calculate the range of probabilities for observed data, assuming different parameter values. • The set of probabilities is referred to as a likelihood surface • We’re going to generate the likelihood surface for a coin tossing experiment • The set of parameter values with the best probability is the maximum likelihood

43. Coin tossing • I tossed a coin 10 times and get 4 heads and 6 tails

44. Coin tossing • I tossed a coin 10 times and get 4 heads and 6 tails • From this data, what does likelihood estimate the chance of heads and tails for this coin?

45. Coin tossing • I tossed a coin 10 times and get 4 heads and 6 tails • From this data, what does likelihood estimate the chance of heads and tails for this coin? • We’re going to calculate: • P(4 heads out of 10 tosses| P(H) = *) • where star takes on a range of values

46. Calculations • P(4 heads out of 10 tosses | P(H)=0.1) = We can make this easier, as it will be constant C* across all calculations

47. Calculations • P(4 heads out of 10 tosses | P(H)=0.1) = Now all we do is change the values of p and q

48. Calculations • P(4 heads out of 10 tosses | P(H)=0.1) =

49. Table of likelihoods

50. Table of likelihoods Largest probability observed = maximum likelihood