MDP Problems and Exact Solutions I

1. MDP Problems and Exact Solutions I • Ryan Christiansen • Department of Mechanical Engineering and Materials Science • Rice University • Slides adapted from Mausam and Andrey Kolobov

2. MDP Problems: At a Glance • MDP Definition (2.1) • Solutions of an MDP (2.2) • Solution Existence (2.3) • Stochastic Shortest-Path MDPs (2.4) • Complexity of Solving MDPs (2.6)

3. MDP Definition • MDP: an MDP is a tuple <S, A, D, T, R> • S is a finite state space • A is a finite action set • D is a sequence of discrete decision epochs (time steps) • T: S x A x S x D → [0, 1] is a transition function (probability) • R: S x A x S x D → ℝis a reward function

4. An MDP Problem • How does an MDP problem work? • Initial Conditions: the starting state • Actions: actions are chosen at each decision epoch to traverse the MDP • Termination: reach a terminating state or the final decision epoch • The goal is to end up with the highest net reward at termination

5. The Policy • Policy: a rule for choosing actions • Global/Complete: a policy must always be applicable for the entire MDP • In general, policies will be • Probabilistic: able to choose between multiple actions randomly • History-Dependent: able to utilize the execution history, or the set of state and action pairs previously traversed • π: H x A → [0, 1]

6. Markovian Policy • Markovian Policy: a history-dependent policy that only depends on the current state and time step • For any two histories hs,tand h′s,tboth of which end at the same state s and timestep t, and for any action a, a Markovian policy π will satisfy π(hs,t, a) = π(h′s,t, a) • In practice, it functions as a history-independent policy • π: S x D x A → [0, 1] • For several important types of MDPs, at least one optimal solution is necessarily Markovian

7. Stationary Markovian Policy • Stationary Markovian Policy: a Markovian policy that does not depend on time • For any two timesteps t1and t2and state s, and for any action a, a stationary Markovian policy π will satisfy π(s, t1, a) = π(s, t2, a) • π: S x A → [0, 1]

8. Evaluate a Policy with the Value Function • Value Function: a function mapping the domain of the policy excluding the action set to a scalar value. • History dependent: V: H → [-∞, ∞] • Markovian: V: S x D → [-∞, ∞] • Stationary Markovian: V: S → [-∞, ∞] • Value Function of a Policy: the utility function of the reward sequence returned from executing the policy, or the total utility of the total reward • Vπ(hs,t) = u(Rtπhs,t, Rt+1πhs,t, …)

9. Solutions of an MDP • A solution to an MDP is an optimal policy, or a policy that maximizes utility. • Policy π* is optimal if the value function, V*, is greater than or equal to the value function of any other policy. • V*(h) ≥ Vπ(h) for all h and π • Need to be careful when defining utility u(R1, R2, …) • For the same h, utility can be different across policy executions • Existence and uniqueness are not guaranteed for many types of MDPs.

10. Expected Linear Additive Utility (ELAU) • u(R1, R2, …) = E(R1 + γR2 + γ2R3 …)where γis the discount factor • Assume γ = 1 unless stated otherwise • 0 ≤ γ < 1 : more immediate rewards are more valuable • γ = 1 : rewards are equally valuable, independently of time • γ > 1 : more distant rewards are more valuable

11. The Optimality Principle • The Optimality Principle: if every policy’s quality can be measured by this policy’s ELAU, there exists a policy that is optimal at every timestep • There are some situations where it may not apply: • When stuck in a repeating sequence of states (a loop) • Infinite decision epochs • Infinite utility

12. The Optimality Principle does not Hold • Oscillating Utility • Unbounded Utility

13. Further Utility Considerations • Risk averse, or risk taking (ELAU is risk neutral) • $1 million guaranteed (risk averse) • 50% chance of $2 million, 50% chance of $0 (risk taking) • Expected value is the same, so risk neutral would choose either

14. 3 Models with Well-Defined Policy ELAU • Finite-horizon MDPs • Infinite-horizon discounted-reward MDPs • Stochastic shortest-path MDPs

15. Finite-Horizon MDPs: Motivation • Assume the agent acts for a finite # of time steps, L • Example applications: • Inventory management“How much X to order from the supplier every day ‘til the end of the season?” • Maintenance scheduling“When to schedule disruptive maintenance jobs by their deadline?”

16. Finite-Horizon MDPs: Definition Puterman, 1994 • FH MDP: an FH MDP is a tuple <S, A, D, T, R> • S is a finite state space • A is a finite action set • D is a sequence of discrete decision epochs (time steps) up to a finite horizon L • T: S x A x S x D → [0, 1] is a transition function (probability) • R: S x A x S x D → ℝis a reward function

17. Finite-Horizon MDPs: Optimality Principle • For an FH MDP with horizon |D| = L < ∞, let: • Vπ(hs,t) = Eπhs,t[R1 + … + RL – t] for all 1 ≤ t ≤ L • Vπ(hs,L +1) = 0 • Then • V* exists and is Markovian, π* exists and is det. Markovian • For all sand1 ≤ t ≤ L:V*(s,t) = maxa in A [Σs′ in S T(s, a, s′, t) [ R(s, a, s′, t) + V*(s′, t+1) ] ] π*(s,t) = argmaxa in A [Σs′ in S T(s, a, s′, t) [ R(s, a, s′, t) + V*(s′, t+1) ] ]

18. Finite-Horizon MDPs: Optimality Principle • For an FH MDP with horizon |D| = L < ∞, let: • Vπ(hs,t) = Eπhs,t[R1 + … + RL – t] for all 1 ≤ t ≤ L • Vπ(hs,L +1) = 0 • Then • V* exists and is Markovian, π* exists and is det. Markovian • For all sand1 ≤ t ≤ L: V*(s,t) = maxa in A [Σs′ in S T(s, a, s′, t) [ R(s, a, s′, t) + V*(s′, t+1) ] ] π*(s,t) = argmaxa in A [Σs′ in S T(s, a, s′, t) [ R(s, a, s′, t) + V*(s′, t+1) ] ] ELAU For every history, the value of every policy is well-defined } } Each E[Ri] is finite } # of terms in the series is finite

19. Finite-Horizon MDPs: Optimality Principle • For an FH MDP with horizon |D| = L < ∞, let: • Vπ(hs,t) = Eπhs,t[R1 + … + RL – t] for all 1 ≤ t ≤ L • Vπ(hs,L +1) = 0 • Then • V* exists and is Markovian, π* exists and is det. Markovian • For all sand1 ≤ t ≤ L: V*(s,t) = maxa in A [Σs′ in S T(s, a, s′, t) [ R(s, a, s′, t) + V*(s′, t+1) ] ] π*(s,t) = argmaxa in A [Σs′ in S T(s, a, s′, t) [ R(s, a, s′, t) + V*(s′, t+1) ] ] ELAU For every history, the value of every policy is well-defined } } Each E[Ri] is finite } # of terms in the series is finite Immediate utility of the next action If you act optimally now { { } } } Highest utility derivable from the next state Highest utility derivable from s at time t In expectation

20. Perks of the FH MDP Optimality Principle • If V* and π* are Markovian, then we only need to consider Markovian V and π • Easy to compute π* • For all s, compute V*(s, t) and π*(s, t) for t = L, …, 1

21. To Infinity and Beyond • Why go beyond the finite horizon? • Autonomous agents with long lifespans (elevators, investments, airplanes, etc.) • Infinite Horizon • Known to be infinite (can continue indefinitely) • Indefinite Horizon • Known to be unbounded (finite processes that can be delayed or extended but will eventually reach a terminal state)

22. Analyzing MDPs with In(de)finite Horizon • Due to the infinite nature of D, we must define stationary, or time independent functions: • T: S x A x S → [0, 1] is a transition function (probability) • R: S x A x S → ℝis a reward function • π: S → A (this is also Markovian) • V: S → [-∞, ∞] (this is also Markovian)

23. Infinite-Horizon Discounted-Reward MDPs: Definition • IHDR MDP: an IHDR MDP is a tuple <S, A, T, R, γ> • S is a finite state space • A is a finite action set • T: S x A x S → [0, 1] is a transition function (probability) • R: S x A x S → ℝis a reward function • γ is a discount factor between 0 and 1 (favors immediate rewards) • Policy value = discounted ELAU over infinite time steps

24. Infinite-Horizon Discounted-Reward MDPs: Optimality Principle • For an IHDR MDP, let: • Vπ(h) = Eπh[R1 + γR2 + γ2R3 …] for all h • Then • V* exists and is stationary Markovian, π* exists and is stationary deterministic Markovian • For all s:V*(s) = maxa in A [Σs′ in S T(s, a, s′) [ R(s, a, s′) + γV*(s′) ] ] π*(s) = argmaxa in A [Σs′ in S T(s, a, s′) [ R(s, a, s′) + γV*(s′) ] ]

25. Infinite-Horizon Discounted-Reward MDPs: Optimality Principle • For an IHDR MDP, let: • Vπ(h) = Eπh[R1 + γR2 + γ2R3 …] for all h • Then • V* exists and is stationary Markovian, π* exists and is stationary deterministic Markovian • For all s:V*(s) = maxa in A [Σs′ in S T(s, a, s′) [ R(s, a, s′) + γV*(s′) ] ] π*(s) = argmaxa in A [Σs′ in S T(s, a, s′) [ R(s, a, s′) + γV*(s′) ] ] ELAU For every history, the value of a policy is well-defined thanks to 0 ≤ γ < 1 } } } All γE[Ri] are bounded by some finite K and converge geometrically

26. Infinite-Horizon Discounted-Reward MDPs: Optimality Principle • For an IHDR MDP, let: • Vπ(h) = Eπh[R1 + γR2 + γ2R3 …] for all h • Then • V* exists and is stationary Markovian, π* exists and is stationary deterministic Markovian • For all s:V*(s) = maxa in A [Σs′ in S T(s, a, s′) [ R(s, a, s′) + γV*(s′) ] ] π*(s) = argmaxa in A [Σs′ in S T(s, a, s′) [ R(s, a, s′) + γV*(s′) ] ] ELAU For every history, the value of a policy is well-defined thanks to 0 ≤ γ < 1 } } } All γE[Ri] are bounded by some finite K and converge geometrically Future utility is discounted Optimal utility is time independent { {

27. Perks of the IHDR MDP Optimality Principle • If V* and π* are stationary Markovian, then we only need to consider stationary Markovian V and π

28. The Meaning of γ • γcan affect optimal policy significantly • γ = 0 + ε: yields myopic policies for impatient agents • γ = 1 - ε: yields far-sighted policies, inefficient to compute • How to set it? • Sometimes suggested by data (inflation rate, interest rate, tax rate) • Often set arbitrarily to a value that gives a reasonable policy

29. Stochastic Shortest-Path MDPs: Motivation • Assume the agent pays a cost to achieve a goal • Example applications: • Controlling a Mars rover“How to collect scientific data without damaging the rover?” • Navigation“What’s the fastest way to get to a destination, taking into account the traffic jams?” • Cost is often time or a physical resource

30. Stochastic Shortest-Path MDPs : Definition • SSP MDP: an SSP MDP is a tuple <S, A, T, C, G> • S is a finite state space • A is a finite action set • T: S x A x S → [0, 1] is a stationary transition function (probability) • C: S x A x S → ℝis a stationary cost function (R = -C) • G ⊆ S is a set of absorbing cost-free goal states • Under two conditions: • There is at least one proper policy (reaches goal with P = 1 from all states) • Every improper policy incurs a cost of infinity from every state from which it does not reach the goal with P = 1

31. SSP MDP Details • In SSP, maximizing ELAU = minimizing expected cost • Every cost-minimizing policy is proper • Thus, an optimal policy is the cheapest way to reach the goal • Why are SSP MDPs called “indefinite-horizon?” • If a policy is optimal, it will take a finite, but a priori unknown time to reach the goal (time to goal is dependent on the evaluation of the policy). • At the limit as t approaches infinity, the probability that a goal state has been reached approaches P = 1

32. SSP MDP Example

33. SSP MDP Example

34. SSP MDP Example

35. SSP MDP Example

36. SSP MDP Example

37. SSP MDP Example

38. SSP MDPs: Optimality Principle • For an SSP MDP, let: • Vπ(h) = Eπh[C1 + C2 + …] for all h • Then • V* exists and is stationary Markovian, π* exists and is stationary deterministic Markovian • For all s:V*(s) = maxa in A [Σs′ in S T(s, a, s′) [ C(s, a, s′) + V*(s′) ] ] π*(s) = argmaxa in A [Σs′ in S T(s, a, s′) [ C(s, a, s′) + V*(s′) ] ]

39. SSP MDPs: Optimality Principle • For an SSP MDP, let: • Vπ(h) = Eπh[C1 + C2 + …] for all h • Then • V* exists and is stationary Markovian, π* exists and is stationary deterministic Markovian • For all s:V*(s) = maxa in A [Σs′ in S T(s, a, s′) [ C(s, a, s′) + V*(s′) ] ] π*(s) = argmaxa in A [Σs′ in S T(s, a, s′) [ C(s, a, s′) + V*(s′) ] ] ELAU For every history, the value of a policy is well-defined } Every policy either takes a finite exp. # of steps to reach a goal, or has infinite cost

40. The MDP Hierarchy • FH => SSP: turn all states (s, L) into goals • IHDR => SSP: add (1 – γ)-probability transitions to goal • Focusing on SSP allows us to develop one set of algorithms to solve all three classes of MDPs.

41. Flat vs. Factored Representation of MDPs • We are only concerned with using flat representation • This is the name for the representation already introduced on the definition slides • It is easier to solve MDPs in flat representation, and it is much easier to describe larger MDPs in flat representation • If you are interested in factored representation, read Section 2.5

42. Computational Complexity of MDPs • Solving IHDR, SSP in flat representation is P-complete • Solving FH in flat representation is P-hard • They don’t benefit from parallelization, but are solvable in polynomial time

43. MDP Exact Solutions I: At a Glance • Brute-Force Alogrithm (3.1) • Policy Evaluation (3.2)

44. Brute Force Algorithm • Go over all policies π • How many? |A||S|, a finite amount • Evaluate each policy • Vπ(s), the expected cost of reaching the goal from s • Choose the best, π* • SSP optimality principle tells us that a best exists • Vπ*(s) ≤ Vπ(s)

45. Policy Evaluation • Given a policy π, compute Vπ • To start out, assume that π is proper • Execution of π reaches a goal from any state

46. Deterministic SSPs • Policy graph for π • π(s0) = a0; π(s1) = a1 • Vπ(s1) = 1 • Vπ(s0) = 5 + 1 = 6

47. Acyclic SSPs • Policy graph for π • Vπ(s1) = 1 • Vπ(s2) = 4 • Vπ(s0) = 0.6(5 + 1) + 0.4(2 + 4) = 6

48. Cyclic SSPs • Policy graph for π • Vπ(s1) = 1 • Vπ(s2) = 0.7(4) + 0.3(3 + Vπ(s0)) • Vπ(s0) = 0.6(5 + 1) + 0.4(2 + 0.7(4) + 0.3(3 + Vπ(s0)))

49. Cyclic SSPs • Generalized system of equations • Vπ(sg) = 0 • Vπ(s1) = 1 + Vπ(sg) • Vπ(s2) = 0.7(4 + Vπ(sg)) + 0.3(3 + Vπ(s0)) • Vπ(s0) = 0.6(5 + Vπ(s1)) + 0.4(2 + Vπ(s2))

50. Policy Evaluation with a System of Equations • Constructing the system of equations • Vπ(s) = 0 if s ∈ G • Vπ(s) = ∑s′ ∈ S T(s, π(s), s′) [C(s, π(s), s′) + Vπ(s′)] • |S| variables • O(|S|3) running time