Probability Theory Overview and Analysis of Randomized Algorithms

1. Probability Theory Overview and Analysis of Randomized Algorithms Analysis of Algorithms Prepared by John Reif, Ph.D.

2. Probability Theory Topics • Random Variables: Binomial and Geometric • Useful Probabilistic Bounds and Inequalities

3. Readings • Main Reading Selections: • CLR, Chapter 5 and Appendix C

4. Probability Measures • A probability measure (Prob) is a mapping from a set of events to the reals such that: • For any event A 0  Prob(A)  1 • Prob (all possible events) = 1 • If A, B are mutually exclusive events, then Prob(A  B) = Prob (A) + Prob (B)

5. Conditional Probability • Define

6. Bayes’ Theorem • If A1, …, An are mutually exclusive and contain all events then

7. Random Variable A (Over Real Numbers) • Density Function

8. Random Variable A (cont’d) • Prob Distribution Function

9. Random Variable A (cont’d) • If for Random Variables A,B • Then “A upper bounds B” and “B lower bounds A”

10. Expectation of Random Variable A • Ā is also called “average of A” and “mean of A” = A

11. Variance of Random Variable A

12. Variance of Random Variable A (cont’d)

13. Discrete Random Variable A

14. Discrete Random Variable A (cont’d)

15. Discrete Random Variable A Over Nonnegative Numbers • Expectation

16. Pair-Wise Independent Random Variables • A,B independent if Prob(A  B) = Prob(A) X Prob(B) • Equivalent definition of independence

17. Bounding Numbers of Permutations • n! = n x (n-1) x 2 x 1= number of permutations of n objects • Stirling’s formula n! = f(n) x (1+o(1)), where

18. Bounding Numbers of Permutations (cont’d) • Note • Tighter bound = number of permutations of n objects taken k at a time

19. Bounding Numbers of Combinations = number of (unordered) combinations of n objects taken k at a time • Bounds (due to Erdos & Spencer, p. 18)

20. Bernoulli Variable • Ai is 1 with prob P and 0 with prob 1-P • Binomial Variable • B is sum of n independent Bernoulli variables Ai each with some probability p

22. B is Binomial Variable with Parameters n,p

23. B is Binomial Variable with Parameters n,p (cont’d)

24. Poisson Trial • Ai is 1 with prob Piand 0 with prob 1-Pi • Suppose B’ is the sum of n independent Poisson trialsAi with probability Pi for i > 1, …, n

25. Hoeffding’s Theorem • B’ is upper bounded by a Binomial Variable B • Parameters n,p where

26. procedure GEOMETRIC parameter p ¬ begin V 0 loop with probability 1-p goto exit Geometric Variable V • parameter p

27. Probabilistic Inequalities • For Random Variable A

28. Markov and Chebychev Probabilistic Inequalities • Markov Inequality (uses only mean) • Chebychev Inequality (uses mean and variance)

29. Example of use of Markov and Chebychev Probabilistic Inequalities • If B is a Binomial with parameters n,p

30. Gaussian Density Function

31. Normal Distribution • Bounds x > 0 (Feller, p. 175)

32. Sums of Independently Distributed Variables • Let Sn be the sum of n independently distributed variables A1, …, An • Each with mean and variance • So Sn has mean  and variance2

33. Strong Law of Large Numbers: Limiting to Normal Distribution • The probability density function of to normal distribution(x) • Hence Prob

34. Strong Law of Large Numbers (cont’d) • So Prob (since 1- (x)  (x)/x)

35. Advanced Material Moment Generating Functions and Chernoff Bounds

36. Moments of Random Variable A (cont’d) • n’th Moments of Random Variable A • Moment generating function

37. Moments of Random Variable A (cont’d) • Note S is a formal parameter

38. Moments of Discrete Random Variable A • n’th moment

39. Probability Generating Function of Discrete Random Variable A

40. Moments of AND of Independent Random Variables • If A1, …, An independent with same distribution

41. Generating Function of Binomial Variable B with Parameters n,p • Interesting fact

42. Generating Function of Geometric Variable with parameter p

43. Chernoff Bound of Random Variable A • Uses all moments • Uses moment generating function By setting x = ’ (s) 1st derivative minimizes bounds

44. Chernoff Bound of Discrete Random Variable A • Choose z = z0 to minimize above bound • Need Probability Generating function

45. Chernoff Bounds for Binomial B with parameters n,p • Above mean x  

46. Chernoff Bounds for Binomial B with parameters n,p (cont’d) • Below mean x  

47. Anguin-Valiant’s Bounds for Binomial B with Parameters n,p • Just above mean • Just below mean

48. Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d)  Tails are bounded by Normal distributions

49. Binomial Variable B with Parameters p,N and Expectation  = pN • By Chernoff Bound for p < ½ • Raghavan-Spencer bound for any  > 0

50. Probability Theory Analysis of Algorithms Prepared by John Reif, Ph.D.