Markov Models

1. Markov Models CS6800 Advanced Theory of Computation Fall 2012 Vinay B Gavirangaswamy

2. Markov Property Processes future values are conditionally dependent on the present state of the system. Strong Markov Property Similar as Markov Property, where values are conditionally dependent on the stopping time (Markov time) instead of present state. Introduction

3. Markov Process Process that follows Markov Property. e.g. First order Markov Process for probability space (, F, P) P(Xt+1=Sj | Xt=Si, Xt-1=Sk ,...) = P(Xt+1=Sj | qt=Si) , for t = 1, 2, 3, … where,  = {S1, S2,S3, …, Sn}, State at time t is Xt=Si Markov Model Model’s a stochastic system which assumes Markov property. It is formally represented by triplet M = (K, , A), where K is a finite set of states  vector of initial probabilities for each of the states. A is a matrix representing transition probabilities. Markov model is an non deterministic finite state machine. Introduction (Contd.)

4. Stochastic FSM The transition matrix: • Stochastic matrix: • Rows sum up to 1 • Double stochastic matrix: • Rows and columns sum up to 1 Simple Markov Model of Weather

5. Prediction Based on State Transition Probability If we want to know probability of the sequence SUNNY SUNNYSUNNYSUNNYSUNNY Take initial probability of SUNNY day i.e. on a any given day probability that it will be SUNNY is 0.30 And for use to get another SUNNY day after a SUNNY day is 0.42 So, by using Markov Chain we can say probability of getting 5 consecutive SUNNY day is

6. Types of Markov Model

7. Markov Chain • Simplest Markov model • System state at time t+1 depends on state at t. Hidden Markov Model • Similar to Markov chain but system states are unobservable. Markov decision process • Models a system where outcomes are partially random and partially under the control of a decision maker. Types of Markov Model(Contd.)

8. Markov decision process It is formally represented by 4-tuple M = (K, , A, R), where K is a finite set of states  vector of initial probabilities for each of the states. A is a matrix representing transition probabilities. R is reward given for transition from state St to Sp with transition probability given in matrix A. Types of Markov Model(Contd.)

9. Partially observable Markov decision process Similar to Markov decision process where state of system is partially observed. It is formally represented by 4-tuple M = (K, , O , A, ), where K is a finite set of states.  is a set of actions. O is a set of observations. A is a matrix representing transition probabilities.  is a set of conditional observation probabilities. Types of Markov Model(Contd.)

10. System that transitions from one state to another, between a finite number of possible states. • Next state depends only on current state and not its history. • Follows Markov property. • Formally for a stochastic process { Xt} P{ Xt+1 = j | X0 = k0, . . . , Xt-1 = kt-1, Xt = i } = P{ Xt+1 = j |Xt = i } for every i, j, k0, . . . , kt-1 and for every t. • Stationary Markov Chains Pr{ Xt+1 = j |Xt = i } = Pr{ X1 = j |X0 = i } for all t Markov Chain

11. If the state space S = { 0, 1, . . . , m–1} then we have jpij= 1  i and pij 0  i, j where pij = Pr{ X1 = j| X0 = i } (we must go some where) (each transition has probability  0) Transition Matrix

12. 0.95 A B C D 0.95 0 0.05 0 0.2 0.5 A A B 0.2 0.5 0 0.3 B 0.2 0.3 0.05 0.8 0 0.2 0 0.8 D C C 1 0 0 1 0 D Each directed edge AB is associated with the positive transition probability from A to B. Representation of a Markov Chain as a Digraph

13. B A D C • States of Markov chains are classified by the digraph representation (omitting the actual probability values) • A, C and D are recurrent states: they are in strongly connected components which are sinks in the graph. B is not recurrent – it is a transient state Alternative definitions: A state s is recurrent if it can be reached from any state reachable from s; otherwise it is transient. Properties of Markov Chain states

14. A B D C A and B are transient states, C and D are recurrent states. Once the process moves from B to D, it will never come back. Another example of Recurrent and Transient States

15. E A B D C A Markov Chain is irreducible if the corresponding graph is strongly connected (and thus all its states are recurrent). A B Irreducible Markov Chains D C 1

16. E A B D C • A state s has a period k if k is the GCD of the lengths of all the cycles that pass via s. (in the shown graph the period of A is 2). • A Markov Chain is periodic if all the states in it have a period k >1. It is aperiodic otherwise. Periodic States

17. A Markov chain is ergodic if : • the corresponding graph is strongly connected. • It is not peridoic A B D C Ergodic Markov Chains are important since they guarantee the corresponding Markovian process converges to a unique distribution, in which all states have strictly positive probability. Ergodic Markov Chains

18. A Markov chain is reversible if Reversible Markov chain

19. Physics • Chemistry • Testing • Information sciences • Queuing theory • Internet applications • Statistics • Economics and finance • Social science • Mathematical biology • Games • Music Markov Model Applications

20. "Wkipedia-Markov model," [Online]. Available: http://en.wikipedia.org/wiki/Markov_chain. [Accessed 28 10 2012]. • P. Xinhui Zhang, "DTMarkovchains," [Online]. Available: http://www.wright.edu/~xinhui.zhang/. [Accessed 28 10 2012]. References