CPSC 531:Probability & Statistics: Review

1. CPSC 531:Probability & Statistics: Review Instructor: Anirban Mahanti Office: ICT 745 Email: mahanti@cpsc.ucalgary.ca Class Location: TRB 101 Lectures: TR 15:30 – 16:45 hours Class web page: http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531 Notes derived from “Probability and Statistics” by M. DeGroot and M. Schervish, Third edition, Addison Wesley, 2002. CPSC 531: Probability Review

2. Outline • Experiments, sample space, and events • Review of set theory • Probability: definition, property, interpretation • Finite sample space • Counting methods • Conditional probability • Independent events • Law of total probability • Bayes’ Theorem CPSC 531: Probability Review

3. Experiments, sample space, and events • Experiment – in probability theory refers to a process whose outcome is not known in advance with certainty • E.g., An experiment consists of tossing a coin 10 times. What is the probability that at least four heads are obtained in a row? • Sample space (S) – is the collection of all possible outcomes of an experiment. Each outcome is a sample point. • E.g., Rolling a six-sided dice: S = {1, 2, 3, 4, 5, 6} • Event – is a subset of the sample space • E.g., A is an event that an even number is rolled: A = {2, 4, 6} CPSC 531: Probability Review

4. Set Theory (1) • Some sets contain only finite number of elements, while others contain infinitely many elements. An infinite set can be classified as: • Countable - if there is a one-to-one correspondence between elements of the set and the set of natural numbers • Uncountable – converse of countable; e.g., set of real numbers, numbers in the interval [0, 1] • Empty set: Ø is the “null” set • Union: A  B = is the event containing all outcomes that are in A or B, or both CPSC 531: Probability Review

5. Set Theory (2) • Intersection: A  B = event contained in both A and B • Complement: AC is the complement of event A and contains all outcomes in the sample space S that do not belong to A • Idempotent laws • A  A = A • A  A = A • Associative laws • (A  B)  C = A  (B  C) = (A  B)  C • (A  B)  C = A  (B  C) = (A  B)  C CPSC 531: Probability Review

6. Set Theory (3) • Commutative laws • A  B = B  A • A  B = B  A • Distributive laws • A  (B  C) = (A  B)  (A  C) • A  (B  C) = (A  B)  (A  C) • Identity laws • A  Ø = A • A U = A • A U = U • A  Ø = Ø CPSC 531: Probability Review

7. Probability - definition • A probability distribution, or simply a probability, on a sample space S is a specification of numbers P(.) that satisfies the following axioms: • For every event A, P(A) ≥ 0 • P(S) = 1 • For every infinite sequence of mutually exclusive events A1, A2, … CPSC 531: Probability Review

8. Properties of Probability • For any event A, P(AC) = 1 – P(A) • Since Ø = SC, P(Ø) = 0 • If A  B, P(A) ≤ P(B) • For every event A, 0 ≤ P(A) ≤ 1 • Home work: Recall Venn diagrams … they can help in proving the above properties. CPSC 531: Probability Review

9. Example (1) • Select one ball from a box containing white (W), red (R), blue (B), and green (G) balls. Suppose that P(R) = 0.1 and P(B) = 0.5. What is the probability of selecting a white or a green ball? • Solution: W U R U B U G = S From axiom 3, we have P(S) = P(W) + P(R) + P(B) + P(G) Also, P(WUG) = P(W)+P(G) = 1 – P(R) – P(B) = 0.4 CPSC 531: Probability Review

10. Example (2) • Rolling a fair dice: P(1) = P(2) = P(3) = P(4) = P(5) = P(6) = 1/6, where P(i) is the probability of rolling a face with i dots. • Question: • P({1, 3}) = ? • P({2, 4, 6}) = ? CPSC 531: Probability Review

11. Interpretations of probability • Classical view: Probability measures the long-term likelihood of outcomes. This is, if an experiment is repeated “many” times, event A should occur roughly P(A) fraction of the time. • Can “many” be quantified? • Bayesian view: Probability represents a personal estimate of how likely an event is going to occur. • Note: the theory of probability does not depend on the interpretation. Good! CPSC 531: Probability Review

12. Finite Sample Spaces • A common assumption for a finite sample space S containing n outcomes s1, s2, … sn is that each outcome is equally likely. • If an event A in this sample space contains exactly m outcomes, then P(A) = m/n • Example: Toss three fair coins simultaneously. What is the probability of obtaining exactly two heads? S = {(HHH), (HHT), (HTH), (THH), (TTH), (HTT), (THT), (TTT)} P(2 H’s) = 3/8 CPSC 531: Probability Review

13. Counting Methods • Multiplication rule • Permutations • Combinations CPSC 531: Probability Review

14. Multiplication rule • If an experiment has k “parts” (k ≥ 2), and the ith part has ni possible outcomes, then the total number of possible outcomes is n1 x n2 … x nn • Example: 6 coins are tossed • 6 parts to the experiments: toss 1, 2, …, 6 • 2 possible outcomes per part (i.e., per toss) • Possible outcomes = 26 = 64 CPSC 531: Probability Review

15. Permutations (1) • Consider an experiment that involves sampling without replacement. E.g., arrange 5 different books on a shelf. How many ways can the books be arranged? • Think of the shelf as having 5 slots • 5 choices for filling the first slot • 4 choices for filling the second slot • 3 choices for filling the third slot • 2 choices for filling the fourth slot …. • So, we have 5x4x3x2x1 = 120 possibilities CPSC 531: Probability Review

16. Permutations (2) • Given n items, the number of permutations (or arrangements) of all n items equals n! • n! = n x (n-1) x (n-2) x … x 2 x 1 • Definition for 0! - It equals 1 since there is only 1 way to arrange zero items. • When not all items are ordered: E.g., a club has 25 members and 2 officers (a president and a secretary). These officers are to be chosen from the members of the club. How many ways can the officers be selected? • 25 ways to select the first officer and 24 ways to select the second officer. Answer = 25x24 = 600 CPSC 531: Probability Review

17. Permutations (3) • Generalization: The number of ordered ways k items can be selected from a pool of n items is: Pn,k = n x (n-1) x … X (n-k+1) = n!/(n-k)! • Home work problem: What is the probability that at least two people share the same birthday? You may assume the following: • 365 possible birthdays (ignore leap years) • Our group consists of 2 ≤ k ≤ 365 people who are unrelated (no twins) • Each birthday is equally likely (birth rate independent of the time of year) CPSC 531: Probability Review

18. Combinations (1) • Consider choosing a subset containing k items from a set of n distinct items. How many different subsets are possible? • The arrangement of elements in the subset is irrelevant. We treat each subset as an unit. • Such unordered samples are called combinations. The number of combinations possible when selecting k items from a set of n items is denoted by Cn,k. • We will use the multiplication rule to find Cn,k. CPSC 531: Probability Review

19. Combinations (2) • Note that one way to find Pn,k is to compute Cn,k (i.e., unordered selection of k items) and multiply by the number of ways of ordering k items (i.e., k!). • Therefore, Pn,k = Cn,k x k! • That is, CPSC 531: Probability Review

20. Combinations (3) • Example: A fair coin is tossed 10 times. Determine the probability, p, of obtaining exactly 3 heads. Solution • Each experiment has two possible outcome, heads or tails. Since the coins are fair, we can assume that each outcome is equally likely. Therefore, the total number of outcomes in this experiment is = 210 • The number of different arrangements possible with 3 heads and 7 tails is = C10,3 • Therefore, CPSC 531: Probability Review

21. Conditional Probability (1) • The conditional probability of event A given that event B has occurred is • The conditional probability P(A|B) is undefined if P(B)=0 • The frequency interpretation: If an experiment is repeated a large number of times, then event B occurs approximately P(B) fraction of the time and the event A and B both occur approximately P(A  B) portion of time. Therefore, among the repetitions in which B occurs, the proportion of events in which A also occurs is P(A  B) /P(B). P(A  B) also denoted as P(AB). CPSC 531: Probability Review

22. Conditional Probability (2) • Example: Two balls are selected at random, without replacement, from a bag containing r red balls and b blue balls. Determine probability p that the first selected ball is red and the second selected ball is blue. Solution A = event that first ball is red; B = event that second ball is blue P(A) = prob. first ball is red = r/(r+b) P(B|A) = prob. blue in second draw = b/(r+b-1) P(B|A) = P(AB)/P(A) P(AB) = (r.b)/((r+b)(r+b-1)) CPSC 531: Probability Review

23. Conditional Probability (3) • P(AB) = P(A) P(A|B); extended, as shown next • Suppose that A1, A2, A3, …, An are events such that P(B) > 0 and P(A1A2A3 …An-1) > 0. Then, P(A1A2A3…An) = P(A1)P(A2|A1) P(A3|A1A2)… P(An|A1A2A3 …An-1) • Example: Draw 4 balls, one by one, without replacement, from a box containing r red balls and b blue balls. Probability of sequence red, red, blue, blue? Solution: Let Rj and Bj denote event that a red and blue ball is drawn, respectively. P(R1R2B3B4) = P(R1)P(R2|R1)P(B3|R1R2) P(B4|R1R2B3) complete the calculations on your own … CPSC 531: Probability Review

24. Independent Events (1) • Independent events are those that don’t have any effect on each other. That is, knowing one of them occurs does not provide any information about the occurrence of the other event. Mathematically, A and B are independent if P(A|B) = P(A) and P(B|A) = P(B) • From conditional probability definition, we have P(AB) = P(A|B)P(B). Therefore, if A and B are independent events, P(AB) = P(A)P(B) CPSC 531: Probability Review

25. Independent Events (2) • Example: Determine the probability pn that exactly n tosses are required for a head to appear for the first time. Solution We need to determine the probability of a sequence of (n-1) tails followed by a head. The probability of obtaining a heads or tails (for a fair coin) is ½. Therefore, pn = (½)n. CPSC 531: Probability Review

26. Law of Total Probability B2 B1 A B3 B4 • Let B1, B2, B3, …, Bk be mutually disjoint events such that B1 U B2 U B3 U … Bk equal the sample space S. Then, for any event A in S, we have Explanation A = (B1A)U (B2A)U… (BkA). The (BjA)’s are disjoint events. Therefore, using laws of conditional probability we get: CPSC 531: Probability Review

27. Bayes’ Theorem • Partition: The events B1, B2, B3, …, Bk form a partition of a set S if they are mutually disjoint and • Bayes’ Theorem: Suppose that B1, B2, B3, …, Bk form a partition of sample space S such that P(Bj)> 0 for j = 1, …, k. Let A be an event in S such that P(A)>0. Then, for i = 1, …, k, CPSC 531: Probability Review

28. Summary • A review of basic probability and statistics was presented. You probably enjoyed it! • Next, we will talk about discrete and continuous distributions. CPSC 531: Probability Review