C HAPTER 2 Basic Concepts of Probability Theory

1. CHAPTER 2 Basic Concepts of Probability Theory Prof. Sang-Jo Yoo sjyoo@inha.ac.kr http://multinet.inha.ac.kr

2. Random Experiments • Random Experiments • An experiment in which the outcome varies in an unpredictable fashion when the experiment is repeated under the same conditions • The specification of a random experiment must include an unambiguous statement of exactly what is measured or observed. • Example 2.1 • E1: Select a ball from an urn containing balls numbered 1 to 50 • E2: Select a ball from an urn containing balls numbered 1 to 4. Balls 1 & 2 are black and balls 3 & 4 are white. Note number and color. • E3: Toss a coin three times and note the sequence of heads and tails • E4: Toss a coin three times and note the number of heads • E7: Pick a number at random between zero and one • E12: Pick two numbers at random between zero and one • E13: Pick a number X at random between zero and one, then pick a number Y at random between zero and X

3. Sample Space • Sample space of a random experiment • The set of all possible outcomes • Outcomes(or sample points) are mutually exclusive in the sense that they cannot occur simultaneously. • When we perform a random experiment, one and only one outcome occurs. • A sample space can be finite, countably infinite or uncountably infinite. • Example 2.2 • E1: Select a ball from an urn containing balls numbered 1 to 50 • E2: Select a ball from an urn containing balls numbered 1 to 4. Balls 1 & 2 are black and balls 3 & 4 are white. Note the number and color. • E3: Toss a coin three times and note the sequence of heads and tails • E4: Toss a coin three times and note the number of heads • E7: Pick a number at random between zero and one • E12: Pick two numbers at random between zero and one • E13: Pick a number X at random between zero and one, then pick a number Y at random between zero and X

4. Sample Space • Toss a dice until a ‘six’ appears and count the number of times the dice was tossed. S = {1, 2, 3, …}; S is discrete and countably infinite (one-to-one correspondence with positive integers) • Pick a number X at random between zero and one, then pick a number Y at random between zero and X. S = {(x, y): 0 ≤ y ≤ x ≤ 1}; S is a continuous sample space. 4

5. Events • Events • Outcomes that satisfies certain conditions • A subset of S • Certain event consists of all outcomes, always occurs • Null event contains no outcomes, never occurs • Event class is the events of interests: set of sets • Example 2.3 • E1: “An even-numbered ball is selected,” A1= ? • E2: “The ball is white and even-numbered,” A2= ? • E3: “The three tosses give the same outcome,” A3= ? • E4: “The number of heads equals the number of tails,” A4= ? • E7: “The number selected is non-negative,” A5= ? • E12: “The two numbers differ by less than 1/10,” A6= ?

6. Review of Set Theory • Terminologies • U: universal set consists of all possible objects • Objects are called the elements or points of the set • Subset of B : A  B • Empty set:  • Equal: A=B • Union: AB={x: xA or x B} • Intersection: AB={x: xA and x B} • Disjoint (mutually exclusive): if AB=  • Complement: Ac={x: xA} • Relative complement (difference): A-B={x: xA and xB} • Notations

7. Set Operations

8. Properties of Set Operations • Commutative properties • Associative properties • Distributive properties • DeMorgan’s Rules

9. Properties of Set Operations • De Morgan’s rules(A ∩ B)c = Ac ∪ Bc and (A ∪ B)c = Ac ∩ Bc • Proof of the second rule: • Suppose x ∈ (A ∪ B)c⇔ x is not contained in any ofthe events A and B⇔ x is contained in Ac and Bc⇔ x ∈ Ac ∩ Bc. • Proof of the first rule: • Based on the second rule, take A → Ac and B → Bc, we then have(Ac ∪ Bc)c = A ∩ B. • Taking complement on both sides, we obtain the first rule. 9

10. The Axioms of Probability • Axioms 1 : • Axioms 2 : • Axioms 3 : • Axioms 3’ :

11. Properties of Probability • Corollary 1 : • Corollary 2 : • Corollary 3 : • Corollary 4 : • Corollary 5 : • Corollary 6 : • Corollary 7 :

12. Discrete Sample Space • Example 2.9 • An urn contains identical balls numbered 0.1,…,9. Randomly select a ball from the urn • A = “number of ball selected is odd,” • B = “number of ball selected is a multiple of 2,” • C = “number of ball selected is less than 5,”

13. Continuous Sample Space • Continuous sample space • Outcome are numbers. • The events consist of intervals of the real line or rectangular regions in the plane. • Example 2.12: Pick a number between zero and one • Sample space S = ? • The probability that the outcome falls in a subinterval of A = the length of the subinterval • A = the event when the outcome is at least 0.3 away from the center of the unit interval. P [A] = ?

14. Conditional Probability • Conditional probability • To determine whether two events A and B are related in the sense that knowledge about the occurrence of B alters the likelihood of occurrence of A.

15. Example: Conditional Probability • Example 2.24 • A ball is selected from an urn containing two black balls (1, 2) and two white balls (3, 4). The sample space is S ={(1, b), (2, b), (3, w), (4, w)}. Assuming that the outcome are equally likely, find P [A|B], P [A|C], where A, B, and C are the following events: A = {(1, b), (2, b), (3, w), (4, w)}, “black ball selected” B = {(1, b), (2, b), (3, w), (4, w)}, “even-numbered ball selected’” C = {(1, b), (2, b), (3, w), (4, w)}, “number of ball is greater than 2” • Example Tossing 2 dice • Suppose the first die is ‘3’; given this information, what is the probability that the sum of the 2 dice equals 8? • There are 6 possible outcomes, given the first dice is ‘3’:(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), and (3, 6). • The outcomes are equally probable. Hence, the probability that the sum is 8, given the first dice is ‘3’, is 1/6 . “” • Conditional Probability Note

16. Theorem On Total Probability • Let B1 , B2 , …, Bn be mutually exclusive events whose union equals the sample space S. Any event can be represented as the union of mutually exclusive events:

17. Example: Total Probability • Example 2.28 • A manufacturing process produces a mix of “good” memory chips and “bad” memory chips. The lifetime of good chips follows the exponential law with a rate of α. The life time of bad chips also follows the exponential law, but the rate of failure is 1000α. Suppose that the fraction of good chips is 1-p and of bad chips p. Find the probability that a randomly selected chip is still functioning after t seconds. • C = the event “chip still functioning after t seconds” • G = the event “chip is good” • B = the event “chip is bad”

18. Example: Total Probability • Example • In a trial, the judge is 65% sure that Susan has committed a crime. Person F (friend) and Person E (enemy) are two witnesses who know whether Susan is innocent or guilty. • Person F is Susan’s friend and will lie with probability 0.25 if Susan is guilty. He will tell the true if Susan is innocent. • Person E is Susan’s enemy and will lie with probability 0.30 if Susan is innocent. Person E will tell the truth if Susan is guilty. • What is the probability that Person F and Person E will give conflicting testimony? • Solution • Let I and G be the two mutually exclusive events that Susan is innocent and guilty, respectively. Let C be the event that the two witnesses will give conflicting testimony. Find P[C] based on P[C|I] and P[C|G]. • By the law of total probabilities 18

19. Bayes’ Rule • Let B1 , B2 , …, Bn be a partition of a sample space S. • Suppose that event A occurs. What is the probability of event Bj ?

20. Example: Bayes’ Rule • Example 2.30 • A manufacturing process produces a mix of “good” memory chips and “bad” memory chips. Suppose that in order to “weed out” the bad chips, every chip is tested for t seconds prior to leaving the factory. • Find the value of t for which 99% of chips sent out to customers are good. • C = the event “chip still functioning after t seconds” • G = the event “chip is good” • B = the event “chip is bad” • We want to find the value of t for which • Example 2.26 & 2.29 (Note)

21. Example: Bayes’ Rule • Example • A judge is 65% sure that a suspect has committed a crime. During the course of the trial, a witness convinces the judge that there is an 85% chance that the criminal is left-handed. If 23% of the population is left-handed and the suspect is also left-handed. With this new information,how certain should the judge be of the guilt of the suspect? • SolutionG = event that the suspect is guiltyI = event that the suspect is innocentL = event that the suspect is left-handed • Since {G, I} forms a partition of the sample space, by Bayes’ Theorem: 21

22. Independence of Events • Event A is independent of B if knowledge of the occurrence of B does not alter the probability of A • Three events A, B, and C are independent if

23. Properties of independence and mutual exclusiveness (1) If A ∩ B = φ, then P[A|B]= 0. A can never occur if B has occurred. (2) If A ≠ φ, B ≠ φ and A and B are independent, then A and B are not mutually exclusive. This is becauseP[A ∩ B]= P[A]P[B] ≠ 0. Remarks we have seen that if A and B are mutually exclusive, then A and B cannot be independent. (3) If A and B are independent and A ∩ B = φ, then either A = φ or B = φ or both. This is because0 = P[A ∩ B]= P[A] P[B]so that either P[A]= 0, P[B] = 0 or both are equal. 23

24. Example: Independent Events • Example 2.31 • A ball is selected from an urn containing two black balls (1, 2) and two white balls (3, 4). The sample space is S = {(1, b), (2, b), (3, w), (4, w)}. Let events A, B, and C are defined as follows: A = {(1, b), (2, b)}, “black ball selected” B = {(2, b), (4, w)}, “even-numbered ball selected’” C = {(3, w), (4, w)}, “number of ball is greater than 2” Are events A and B independent? Are events A and C independent?  In general if two events have nonzero probability and are mutually exclusive, then they cannot be independent.

25. Example: Independent Events Example Two numbers x and y are selected at random between zero and one. Let events A, B and C be defined by However, 25

26. Sequential Experiment • Sequential of independent experiment • Suppose that a random experiment consists of performing experiments E1, E2, …, En. • Outcome of the experiment • Sample space of the experiment • Let A1 , A2 , …, An be events such that Ak concerns only the outcome of the kth sub-experiment. If the sub-experiments are independent, then

27. Example: Sequential Events • Example 2.36 • Suppose that 10 numbers are selected at random from the interval [0, 1]. Find the probability that the first 5 numbers are less than ¼ and the last 5 numbers are greater than ½. Let x1, x2, …, x10 be the sequence of 10 numbers.

28. Binomial Probability Law • Bernoulli trial • Performance an experiment once and nothing whether a particular event A occurs. • Outcome = “success” if A occurs, outcome = “fail” otherwise • Binomial probability law • Probability of k successes in n independent repetitions of a Bernoulli trial

29. Example: Binomial Probability Law • Example 2.37 • Suppose that a coin is tossed three times. Assume that the tosses are independent and the probability of heads is p P [{HHH}] = P [{HHT}] = P [{HTH}] = P [{THH}] = P [{TTH}] = P [{THT}] = P [{HTT}] = P [{TTT}] = • Let k be the number of heads in three trials P [k=0] = P [k=1] = P [k=2] = P[ k=3] =

30. Example: Binomial Probability Law • Example 2.39 Voice Communication System • Let k be the number of active speakers in a group of eight independent speakers. Suppose that a speaker is active with probability 1/3. • Find the probability that the number of active speakers is greater than six. • Example 2.40: Error Correction Coding • A communication system transmits binary information over a channel that introduces random bit errors with probability =10-3. the transmitter transmits each information bit three times and a decoder takes a majority vote of the received bits to decide on what the transmitted bit was. • Find the probability that the receiver will make an incorrect decision.

31. Multinomial Probability Law • Let B1,B2, …, BM be a partition of the sample space S of some random experiment and P[Bj]=Pj. • The events are mutually exclusive • P1+P2+…+PM=1 • Suppose the n independent repetitions of the experiment are performed. • Let kj be the number of time event Bj occurs, then the vector (k1,k2,…,kM) specifies the number of times each of the event Bj occurs • Multinomial probability law • M=2: Binomial probability law

32. Geometric Probability Law • A sequential experiment in which we repeat independent Bernoulli trials until the occurrence of the first success. • Let the outcome of this experiment be m. • Geometric probability law • The probability p(m), that m trials are required.