Introduction to stochastic process

1. Introduction to stochastic process Dr. NurAiniMasruroh

2. Outlines Concept of probability Random variables Expected value Conditional probability Limit theorem Stochastic processes

3. Probability in Industrial Engineering • Random arrivals of jobs • Random service time • Random requests for resources • Probability of queue overflow • Service delays • Scheduling problems • Flow control and routing • Revenue management • Risk assessment for decision analysis • Etc

4. Basic probability • Random experiment: an experiment whose outcome cannot be determined in advanced • Sample space (S): set of all possible outcomes • Event (E): a subset of sample space, it occurs if the outcome of the experiment is an element of that subset • A number P(E) is defined and satisfies: • 0 ≤ P(E) ≤ 1 • P(S) =1 • For any sequence of events E1, E2, … that are mutually exclusive

5. Random variables • Consider a random experiment having sample space S. • A random variable X is a function that assigns a real value to each outcome in S • The distribution function F of the random variable X is defined for any real number x by • A random variable X is said to be discrete if its set of possible values is countable. For discrete random variables, • A random variable is called continuous if there exists a function f(x), called the probability density function, such that For every set B

6. Random variables • Since it follows that • The join distribution function F of two random variables X and Y is defined by F(x, y) = P{X≤ x, Y ≤ y} • The distribution functions of X and Y, Fx(x) = P{X≤ x} and FY(y) = P{Y≤y} It can be obtained from F(x, y) by making use of the continuity property of the probability operator. Continuity property: Similarly, F(x, y) = Fx(x)FY(y) if X and Y are independent

7. Cumulative Distribution Function (CDF) • CDF, F, of the random variable X is defined by all real numbers b, -∞ < b < ∞, by F(b) = P(X ≤ b) • F(b) denotes the cumulative probability that the random variable X takes on a value that is less or equal to b (the total probability mass) • CDF is a non-decreasing function: P(X≤ a) ≤ P(X ≤ b) if a < b and thus F(a) ≤ F(b) for a < b • F(∞) = 1 and F(-∞)=0: bounded • CDF is a right continuous function: for any b, F(b+), value of F(b) just to the right of b, equals to F(b) as b+ get closer to b. The CDF is right continuous at all values, but may be left discontinuous at some values

8. Probabilities from CDF • P(X ≤ b) = F(b) • P(X<b) = F(b – ε) = F(b-) • ε is a small number • P(X=b) = P(X ≤ b) – P(X < b) = F(b) – F(b-) • P(a < X ≤ b) = F(b) – F(a) • P(a ≤ X ≤ b) = F(b) – F(a-) • P(a ≤ X < b) = F(b-) – F(a-) • P(a < X < b) = F(b-) – F(a)

9. Probability Density Function (pdf) • The pdf of a continuous random variable X is defined by the derivative of the continuous CDF at differentiable intervals : dF(b)/db = f(b) • The pdf of a continuous random variable tells us the density of the mass at each point on the sample space • The value of pdf is not probability, for example, if f(x) = 2x, 0 ≤ x ≤ 1, and 0 otherwise, f(1) = 2 is obviously not a probability value • Graphically, pdf is the area under f(x) from ain interval a,b. • Note: if pmf has meaning itself, the value itself for pdf has no meaning

10. Discrete random variable: summary • A random variable X associates a number with each outcome of an experiment • A discrete random variable takes on a finite number • The probabilistic behavior of a discrete random variable X is described by its probability mass function (pmf), p(u) • P(uj) = p(X=uj) • All the probabilistic information about the discrete random variable X is summarized in its pmf • A discrete random variable X has a CDF F(b) which is: • Right-continuous • Staircase CDF

11. Continuous random variable: summary • No mass to define pmf: every event has zero probability • Example: p(height of people= 173.7897654) 0 • However P(147< height of people < 187) 1 • The set of possible values are uncountable while the set of possible values were finite or countably infinite for a discrete random variable • Sample space is not a discrete set, but a continuous space (or interval) • A continuous random variable X has a CDF F(b) which is: • Continuous at all b, -∞ < b < ∞ • Differentiable at all b (except possibly at finite set of points)

12. Expected value • Expectation of the random variable X, • The variance of the random variable X is defined by Var X = E[(X – E[X])2] = E[X2] – E2[X] • Two jointly distributed random variables X and Y are said to be uncorrelated if their covariance defined by Cov (X, Y) = E[(X – EX)(Y – EY)] = E[XY] – E[X]E[Y]

13. Expected value • Expectation of a sum of random variables is equal to the sum of the expectations • Variances:

14. Conditional probabilities • In general, given information about the outcome of some events, we may revise our probabilities of other events • We do this through the use of conditional probabilities • The probability of an event X given specific outcomes of another event Y is called the conditional probability X given Y • The conditional probability of event X given event Y and other background information ξ, is denoted by p(X|Y, ξ) and is given by

15. Bayes’ Theorem • Given two uncertain events X and Y. Suppose the probabilities p(X|ξ) and p(Y|X, ξ) are known, then

16. Factorization rule for joint probability

17. Limit theorem • Strong Law of Large Numbers • If X1, X2, … are independent and identically distributed with mean μ, then • Central Limit Theorem • If X1, X2, … are independent and identically distributed with mean μ and variance σ2, then • Thus if we let where X1, X2, … are independent and identically distributed, then the Strong of LLN states that, with probability1, Sn/n will converge to E[Xi]; whereas the CLT states that Sn will have an asymptotic normal distribution as n → ∞

18. Stochastic process X(t) is the state of the process (measurable characteristic of interest) at time t • the state space of the a stochastic process is defined as the set of all possible values that the random variables X(t) can assume • when the set T is countable, the stochastic process is a discrete time process; denote by {Xn, n=0, 1, 2, …} • when T is an interval of the real line, the stochastic process is a continuous time process; denote by {X(t), t≥0}

19. Stochastic process Hence, • a stochastic process is a family of random variables that describes the evolution through time of some (physical) process. • usually, the random variables X(t) are dependent and hence the analysis of stochastic processes is very difficult. • Discrete Time Markov Chains (DTMC) is a special type of stochastic process that has a very simple dependence among X(t) and renders nice results in the analysis of {X(t), t∈T} under very mild assumptions.

20. Example of stochastic processes Refer to X(t) as the state of the process at time t  A stochastic process {X(t), t∈T} is a time indexed collection of random variables • X(t) might equal the total number of customers that have entered a supermarket by time t • X(t) might equal the number of customers in the supermarket at time t • X(t) might equal the stock price of a company at time t

21. Counting process • Definition: • A stochastic process {N(t), t≥0} is a counting process if N(t) represents the total number of “events” that have occurred up to time t

22. Counting process • Examples: • If N(t) equal the number of persons who have entered a particular store at or prior to time t, then {N(t), t≥0} is a counting process in which an event corresponds to a person entering the store • If N(t) equal the number of persons in the store at time t, then {N(t), t≥0} would not be a counting process. Why? • If N(t) equals the total number of people born by time t, then {N(t), t≥0} is a counting process in which an event corresponds to a child is born • If N(t) equals the number of goals that Ronaldo has scored by time t, then {N(t), t≥0} is a counting process in which an event occurs whenever he scores a goal

23. Counting process • A counting process N(t) must satisfy • N(t)≥0 • N(t) is integer valued • If s ≤t, then N(s) ≤ N(t) • For s<t, N(t)-N(s) equals the number of events that have occurred in the interval (s,t), or the increments of the counting process in (s,t) • A counting process has • Independent increments if the number of events which occur in disjoint time intervals are independent • Stationary increments if the distribution of the number of events which occur in any interval of time depends only on the length of the time interval

24. Independent increment • This property says that numbers of events in disjoint intervals are independent random variables. • Suppose that t1< t2≤ t3< t4. Then N(t2)-N(t1), the number of events occurring in (t1,t2], is independent of N(t4)-N(t3), the number of events occurring in (t3, t4].