Stochastic Processes: Characterization and Classification

Probability and Statistics with Reliability, Queuing and Computer Science Applications: Chapter 6 on Stochastic Processes Kishor S. Trivedi Visiting Professor Dept. of Computer Science and Engineering Indian Institute of Technology, Kanpur

What is a Stochastic Process? • Stochastic Process: is a family of random variables {X(t) | t ε T} (T is an index set; it may be discrete or continuous) • Values assumed by X(t) are called states. • State space (I): set of all possible states • Sometimes called a random process or a chance process

Stochastic Process Characterization • At a fixed time t=t1, we have a random variable X(t1). Similarly, we have X(t2), .., X(tk). • X(t1) can be characterized by its distribution function, • We can also consider the joint distribution function, • Discrete and continuous cases: • States X(t) (i.e. time t) may be discrete/continuous • State space I may be discrete/continuous

Classification of Stochastic Processes • Four classes of stochastic processes: • discrete-state process  chain • discrete-time process  stochastic sequence {Xn | n є T} (e.g., probing a system every 10 ms.)

Example: a Queuing System • Interarrival times Y1, Y2, … (common dist. Fn. FY) • Service times: S1, S2, … (iid with a common cdf FS) • Notation for a queuing system: FY /FS/m • Some interarrival/service time distributions types are: • M: Memoryless (i.e., EXP) • D: Deterministic • Ek: k-stage Erlang etc. • Hk: k-stage Hyper exponential distribution • G: General distribution • GI: General independent inter arrival times • M/M/1  Memoryless interarrival/service times with a single server

Discrete/Continuous Stochastic Processes • Nk: Number of jobs waiting in the system at the time of kth job’s departure  Stochastic process {Nk| k=1,2,…}: • Discrete time, discrete state Nk Discrete k Discrete

Continuous Time, Discrete Space • X(t): Number of jobs in the system at time t. {X(t) | t є T} forms a continuous-time, discrete-state stochastic process, with, X(t) Discrete Continuous

Discrete Time, Continuous Space • Wk: waiting time for the kth job. Then {Wk | k є T}forms a Discrete-time, Continuous-state stochastic process, where, Wk Continuous k Discrete

Continuous Time, Continuous Space • Y(t): total service time for all jobs in the system at time t. Y(t) forms a continuous-time, continuous-state stochastic process, Where, Y(t) t

Further Classification • Similarly, we can define nth order distribution: • Formidable task to provide nth order distribution for all n. (1st order distribution) (2nd order distribution)

Further Classification (contd.) • Can the nth order distribution be simplified? • Yes. Under some simplifying assumptions: • Independence • As example, we have the Renewal Process • Discrete time independent process {Xn | n=1,2,…} (X1, X2, .. are iid, non-negative rvs), e.g., repair/replacement after a failure. • Markov process introduces a limited form of dependence • Markov Process • Stochastic proc. {X(t) | t є T} is Markov if for any t0 < t1< … < tn< t, the conditional distribution satisfies the Markov property:

Markov Process • We will only deal with discrete state Markov processes i.e., Markov chains • In some situations, a Markov chain may also exhibit time-homogeneity • Future of process (probabilistically) determined by its current state, independent of how it reached this particular state; but in a non homogeneous case, current time can also determine the future. • For a homogeneous Markov chain current time is also not needed to determine the future. • Let Y: time spent in a given state in a hom. CTMC

Homogeneous CTMC-Sojourn time • Since Y, the sojourn time, has the memoryless prop. • This result says that for a homogeneous continuous time Markov chain, sojourn time in a state follows EXP( ) distribution (not true for non-hom CTMC) • Hom. DTMC sojourn time dist. Is geometric. • Semi-Markov process is one in which the sojourn time in a state is generally distributed.

Bernoulli Process • A sequence of iid Bernoulli rvs, {Yi | i=1,2,3,..}, Yi =1 or 0 • {Yi} forms a Bernoulli Process, an example of a renewal process. • Define another stochastic process , {Sn | n=1,2,3,..}, where Sn = Y1 +Y2 +…+ Yn (i.e. Sn :sequence of partial sums) • Sn = Sn-1+ Yn (recursive form) • P[Sn = k | Sn-1= k] = P[Yn = 0] = (1-p)and, • P[Sn = k | Sn-1= k-1] = P[Yn = 1] = p • {Sn |n=1,2,3,..}, forms a Binomial process, an example of a homogeneous DTMC

Renewal Counting Process • Renewal counting process: # of renewals (repairs, replacements, arrivals) by time t: a continuous time process: • If time interval between two renewals follows EXP distribution, then  Poisson Process

Note: • For a fixed t, N(t) is a random variable (in this case a discrete random variable known as the Poisson random variable) • The family {N(t), t 0} is a stochastic process, in this case, the homogeneous Poisson process • {N(t), t 0} is a homogeneous CTMC as well

Poisson Process • A continuous time, discrete state process. • N(t): no. of events occurring in time (0, t]. Events may be, • # of packets arriving at a router port • # of incoming telephone calls at a switch • # of jobs arriving at file/compute server • Number of component failures • Events occurs successively and that intervals between these successive events are iid rvs, each following EXP( ) • λ: arrival rate (1/ λ: average time between arrivals) • λ: failure rate (1/ λ: average time between failures)

Poisson Process (contd.) • N(t) forms a Poisson process provided: • N(0) = 0 • Events within non-overlapping intervals are independent • In a very small interval h, only one event may occur (prob. p(h)) • Letting, pn(t) = P[N(t)=n], • For a Poisson process, interarrival times follow EXP( ) (memoryless) distribution. • E[N(t)] = Var[N(t)] = λt ; What about E[N(t)/t], as t infinity?

Merged Multiple Poisson Process Streams • Consider the system, • Proof: Using z-transform. Letting, α = λt, +

Decomposing a Poisson Stream • Decompose a Poisson process using a prob. switch • N arrivals decomposed into {N1, N2, .., Nk}; N= N1+N2, ..,+Nk • Cond. pmf • Since, • The uncond. pmf

Generalizing the Poisson Process Poisson Process Non-Homogeneous Poisson Process (NHPP)

If the expected number of events per unit time, l, changes with age (time), we have a non-homogeneous Poisson model. We assume that: 1. If 0  t, the pmf of N(t) is given by: where m(t)  0 is the expected number of events in the time period [0, t] 2. Counts of events in non-overlapping time periods are mutually independent. m(t) : the mean value function. l(x) :the time-dependent rate of occurrence of events or time-dependent failure rate Non-Homogeneous Poisson Process (NHPP)

NHPP(cont.)

Generalizing Poisson Process Poisson Process Renewal Counting Process Non-Homogeneous Poisson Process (NHPP)

Renewal Counting Process • Poisson process  EXP( ) distributed interarrival times. • What if the EXP( ) assumption is removed  renewal proc. • Renewal proc. : {Xi | i=1,2,…} (Xi’s are iid non-EXP rvs) • Xi : time gap between the occurrence of (i-1) st and ith event • Sk = X1 + X2 + .. + Xk time to occurrence of the kth event. • N(t)- Renewal counting process is a discrete-state, continuous-time stochastic process. N(t) denotes no. of renewals in the interval (0, t].

Sn t tn More arrivals possible Renewal Counting Processes (contd.) • For N(t), what is P(N(t) = n)?

Renewal Counting Process Expectation • Let, m(t) = E[N(t)]. Then, m(t) = mean no. of arrivals in time (0,t]. m(t) is called the renewal function.

Renewal Density Function • Renewal density function: • For example, if the renewal interval X is EXP(λ), then • d(t) = λ , t >= 0 and m(t) = λ t , t >= 0. • P[N(t)=n] = • Fn(t) will turn out to be n-stage Erlang e–λ t (λ t)n/n! i.e Poisson pmf

Where: Failure times T1, T2, … are mutually independent with a common distribution function W Restoration times D1, D2, … are mutually independent with a common distribution function G The sequences {Tn} and {Dn} are independent Alternating Renewal Process I(t) 1 Operating Restoration 0 Time

MTBF T1 D1 T2 D2 T3 D3 T4 D4 ……. Availability Analysis • Availability: is defined is the ability of a system to provide the desired service. • If no repair/replacement,Availability(t)=Reliability(t) • If repairs are possible, then above is pessimistic. • MTBF = E[Di+Ti+1] = E[Ti+Di]=E[Xi]=MTTF+MTTR

renewal Repair is completed with in this interval t x Availability Analysis (contd.) • Two mutually exclusive situations: • System does not fail before time t  A(t) = R(t) • System fails, but the repair is completed before time t • Therefore, A(t) = sum of these two probabilities

Repair is completed with in this interval x t x+dx 0 Renewed life time >= (t-x) Availability Expression • dA(x) : Incremental availability • dA(x) = Prob(that after renewal, life time is > (t-x) & that the renewal occurs in the interval (x,x+dx])

Availability Expression (contd.) • A(t) can also be expressed in the Laplace domain. • Since, R(t) = 1-W(t) or LR(s) = 1/s – LW(s) = 1/s –Lw(s)/s • What happens when t becomes very large? • However,

Availability, MTTF and MTTR • Steady state availability A is: • Taking the expression of sLA(s) and taking the limit via L’Hospital rule and using the moment generating property of the LT, we get the required result for the steady-state A=MTTF/(MTTF+MTTR)

Availability Example • Assuming EXP( ) density fn for g(t) and w(t)

Generalizing Poisson Process Bernoulli Process Poisson Process Homogeneous Continuous Time Markov Chain Compound Poisson Process Renewal Counting Process Non-Homogeneous Poisson Process (NHPP) Homogeneous Discrete Time Markov Chain Non-Homogeneous Continuous Time Markov Chain Semi-Markov Process Markov Regenerative Process

Stochastic Processes: Characterization and Classification