Background Knowledge

1. Background Knowledge • Brief Review on • Counting, • Probability, • Statistics, • I. Theory

2. Counting: Permutations • Permutations: • The number of possible permutations of r objects from n objects is n ( n-1) (n-2) … (n –r +1) = n! / (n-r)! • We denote this number as nPr Remember the factorial of a number x = x! is defined as x! = (x) (x-1) (x-2) …. (2)(1)

3. Counting: Permutations • Permutations with indistinguishable objects: • Assume we have a total of n objects. • r1 are alike, r2 are alike,.., rk are alike. • The number of possible permutations of n objects is n! / r1! r2! … rk!

4. Counting: Combinations • Combinations: • Assume we wish to select r objects from n objects. • In this case we do not care about the order in which we select the r objects. • The number of possible combinations of r objects from n objects is n ( n-1) (n-2) … (n –r +1) / r! = n! / (n-r)! r! • We denote this number as C(n,r)

5. Statistical and Inductive Probability • Statistical: • Relative frequency of occurrence after many trials • Inductive: • Degree of belief on certain event We will be concerned with the statistical view only. Law of large numbers 0.5 Proportion of heads Number of flips of a coin

6. The Sample Space • The space of all possible outcomes of a given process or situation is called the sample space S. Example: cars crossing a check point based on color and size: S red & small blue & small blue & large red & large

7. An Event • An event is a subset of the sample space. Example: Event A: red cars crossing a check point irrespective of size S blue & small red & small red & large blue & large A

8. The Laws of Probability • The probability of the sample space S is 1, P(S) = 1 • The probability of any event A is such that 0 <= P(A) <= 1. • Law of Addition If A and B are mutually exclusive events, then the probability that either one of them will occur is the sum of the individual probabilities: P(A or B) = P(A) + P(B) If A and B are not mutually exclusive: P(A or B) = P(A) + P(B) – P(A and B) B A

9. Conditional Probabilities • Given that A and B are events in sample space S, and P(B) is different of 0, then the conditional probability of A given B is P(A|B) = P(A and B) / P(B) • If A and B are independent then P(A|B) = P(A)

10. The Laws of Probability • Law of Multiplication What is the probability that both A and B occur together? P(A and B) = P(A) P(B|A) where P(B|A) is the probability of B conditioned on A. If A and B are statistically independent: P(B|A) = P(B) and then P(A and B) = P(A) P(B)

11. Random Variable Definition: A variable that can take on several values, each value having a probability of occurrence. • There are two types of random variables: • Discrete. Take on a countable number of values. • Continuous. Take on a range of values. Discrete Variables • For every discrete variable X there will be a probability function P(x) = P(X = x). • The cumulative probability function for X is defined as F(x) = P(X <= x).

12. Random Variable Continuous Variables: • Concept of histogram. • For every variable X we will associate a probability density function f(x). The probability is the area lying between two values. Prob(x1 < X <= x2) = ∫x1 f(x) dx • The cumulative probability function is defined as F(x) = Prob( X <= x) = ∫-infinity f(u) du x2 x

13. Multivariate Distributions • P(x,y) = P( X = x and Y = y). • P’(x) = Prob( X = x) = ∑y P(x,y) It is called the marginal distribution of X The same can be done on Y to define the marginal distribution of Y, P”(y). • If X and Y are independent then P(x,y) = P’(x) P”(y)

14. Expectations: The Mean • Let X be a discrete random variable that takes the following values: x1, x2, x3, …, xn. Let P(x1), P(x2), P(x3),…,P(xn) be their respective probabilities. Then the expected value of X, E(X), is defined as E(X) = x1P(x1) + x2P(x2) + x3P(x3) + … + xnP(xn) E(X) = Σi xi P(xi)

15. The Binomial Distribution • What is the probability of getting x successes in n trials? • Assumption: all trials are independent and the probability of success remains the same. Let p be the probability of success and let q = 1-p then the binomial distribution is defined as P(x) = nCx p x q n-x for x = 0,1,2,…,n The mean equals n p

16. The Multinomial Distribution • We can generalize the binomial distribution when the random variable takes more than just two values. • We have n independent trials. Each trial can result in k different values with probabilities p1, p2, …, pk. • What is the probability of seeing the first value x1 times, the second value x2 times, etc. P(x1,x2,…,xk) = [n! / (x1!x2!…xk!)] p1x1 p2x2 … pk xk

17. Other Distributions • Poisson P(x) = e-u ux / x! • Geometric f(x) = p(1-p)x-1 • Exponential f(x) = λ e-λx • Others: • Normal • χ2, t, and F

18. Entropy of a Random Variable A measure of uncertainty or entropy that is associated to a random variable X is defined as H(X) = - Σ pi log pi where the logarithm is in base 2. This is the “average amount of information or entropy of a finite complete probability scheme” (Introduction to I. Theory by Reza F.).

19. Example of Entropy There are two possible complete events A and B (Example: flipping a biased coin). • P(A) = 1/256, P(B) = 255/256 • H(X) = 0.0369 bit • P(A) = 1/2, P(B) = 1/2 H(X) = 1 bit • P(A) = 7/16, P(B) = 9/16 H(X) = 0.989 bit

20. Entropy of a Binary Source It is a function concave downward. 1 bit 0 0.5 1

21. Derived Measures Average information per pairs H(X,Y) = - ΣxΣy P(x,y) log P(x,y) Conditional Entropy: H(X|Y) = - ΣxΣy P(x,y) log P(x|y) Mutual Information: I(X;Y) = ΣxΣy P(x,y) log [P(x,y) / (P(x) P(y))] = H(X) + H(Y) – H(X,Y) = H(X) – H(X|Y) = H(Y) – H(Y|X)