Markov Chains and Random Walks

1. Markov Chains and Random Walks

2. Def: • A stochastic process X={X(t),t∈T} is a collection of random variables. • If T is a countable set, say T={0,1,2,…}, we say that X is a discrete time stochastic process. Otherwise it is called continuous time stochastic process. • Here we consider a discrete time stochastic process Xn, n=0,1,2,…

3. If Xn=i, then the process is said to be in state i at time n. • Pr[Xn+1=j|Xn=i,Xn-1=in-1,…,X0=i0]=Pi,j for all states i0,i1,…,in-1,i,j and all n≥0. Xn+1 depends only on Xn . • Such a stochastic process is known as a Markov chain. Pi,1 1 i Pi,2 Pi,j≥0 jPi,j=1 2 Pi,j ⋮ j

4. The n-step transition prob of the Markov chain is defined as the conditional prob., given that the chain is currently in state i, that will be in state j after n additional transitions. I.e, • Chapman-Kolmogorov equation:

5. Proof: • Let P(n) denote the matrix of n-step transition probabilities , then the Chapman-Kolmogorov equationsP(n)=Pn .

6. Eg. 0.7 0 0.5 1 0.4 0 0.3 0.2 0.5 2 0.6 3 0.8

7. Classification of states • Def: • State j is said to be accessible from state i if for some n≥0 . • We say states i and j communicate if they are both accessible from each other. (i↔j) • The Markov chain is said to be “irreducible” if there is only one class, I.e., if all states communicate with each other. • For any state i, let fi denote the prob. that, starting in state i, the process will ever reenter that state. State i is said to be “recurrent” if fi=1, and “transient” if fi<1. • Starting in state i, the prob. that the process will be in state i for exactly n time periods equals fin-1(1-fi), n≥1 .

8. More definitions • A state i has period d if P =0 whenever n is not divisible by d and d is the largest integer with this property. • A state with period 1 is said to be aperiodic. • A state i is said to be positive recurrent, if the expected time to return to state i is finite. Otherwise it is null recurrent. • Positive recurrent, aperiodic states are called ergodic.

9. represents the number of periods that the process is in state i.

10. Prop: State i is recurrent if transient if • Cor: If state i is recurrent, and state i communicates with state j, then state j is recurrent.

11. Proof: • ∵ i communicates with j, ∴ Exist k,m s.t. .For any integer n, Thus, j is also recurrent. m n j i k

12. Stantionary Distributions • Definition: A stationary distribution of a Markov chain is a • probability distribution such that • Thm: Any finite, irreducible and ergodic Markov chain has • the following properties: • the chain has a unique stationary distribution • for all j and i, the limit exists and it is independent of j;

13. Proof of theorem: • Lemma: For any reducible , ergodic Markov chain and for any state i, the limit of We explain instead of proving the lemma as follows. The expected time between visits to i is hi,i and therefore state i is visited 1/hi,i of the time. Thus the limit of , which represents the probability a state chosen far in the future is at state i, must be 1/hi,i. Since the limit exists, we have

14. Proof of theorem: • Recall rjit is the probability that starting at j, the chain first visits i at time t. • By irreducibility, we have

15. Proof of Theorem:

16. Proof of Theorem:

17. Proof of Theorem:

18. Random Walks on Undirected graphs • Lemma: A random walk on an undirected graph G is aperiodic iff G is not bipartite. Pf: A graph is bipartite iff it does not have odd cycle. In an undirected graph, there is always a path of length 2 from a vertex to itself.

19. Thus, if the graph is bipartite then the random walk is periodic (d=2). If not bipartite, then it has an odd cycle and GCD(2,odd-number)=1. Thus, the Markov chain is aperiodic.

20. Thm: G: not bipartite, finite, undirected and connected. A random walk on G converges to a stationary distribution , where v=d(v)/2|E|. Pf: Thus, is a distribution. Let P be the transition probability matrix. Let N(v) be the neighbors of v.

21. Thus, Cor: hv,u: the expected number of steps to reach u from v. For any uG, hu,u=2|E|/d(u).

22. Lemma: If (u,v)E, then hv,u  2|E|. Pf: N(u): the set of neighbors of u. Thus,  hv,u<2|E|.

23. Def: The cover time of a graph G=(V,E) is the maximum over all vV of the expected time to visit all of the nodes in the graph by a random walk starting from v. Lemma: The cover time of G=(V,E) is bounded by 4|V||E|.

24. Pf: Choose a spanning tree of G. Then there exists a cyclic tour on this tree, where each edge is traversed once in each direction, which can be found by doing a DFS. Let v0,v1,…,v2|V|-2=v0 be the sequence of vertices in the tour, starting from v0.

25. Clearly, the expected time to go through the vertices in the tour is an upper bound on the cover time. Hence, the cover time is bounded above by

26. Application: s-t connectivity It can be done with BFS, DFS by using (n) space. The following randomized algorithm works with only O(logn) bits. • s-t Connectivity Algorithm: • Input: G • Start a random walk from s. • If the walk reaches t within 4n3 steps, return that there is a path. Otherwise, return no path.

27. Assume G has no bipartite connected component. (The results can be made to apply to bipartite graphs with some additional work.) Thm: The s-t connectivity algorithm returns the correct answer with probability ½ and it only errs by returning that there is no path from s to t when there is such a path.

28. Why O(logn) bits? Pf: The algorithm gives correct answer, when G has no s-t path. If G has an s-t path, the algorithm errs if it does not find the path in 4n3 steps. The expected time to reach t from s is bounded by the cover time, which is at most 4|V||E|<2n3. By Markov’s inequality,