Agenda for This Week

1. Agenda for This Week

2. Chapter 18 Dynamic Programming – Part 1

3. Dynamic Programming • Dynamic Programming (DP) is a model that deals with decisions where several decisions need to be made in sequence usually over time; called dynamic, multistage or sequential decision problems • Also useful for decomposing multi-variable, single-time problems into several single-variable problems at different times

4. Dynamic Programming • DP is a broad problem structure rather than a specific method • The essential characteristic is that there is a series of decisions distributed in time, where there is an interrelationship between decisions; thus, the current decision cannot be made independently of future ones.

5. Dynamic Programming Concepts • Stages: each point in the problem where a decision must be made (can be actual or artificial points in time • States: represents what the status of the system is when entering the next stage (e.g., how many resources left still to allocate) • Policy: decision making rule that transforms the state at a given stage into a state associated with an earlier stage • Optimal Policy: policy that optimizes the value of the objective function

6. Dynamic Programming • General relationship between components: Decision, Di Decision, Di-1 Decision, Di-2 State Si-3 State Si Stage i State Si-1 Stage i-1 State Si-2 Stage i-2 (Ti) (Ti-1) (Ti-2) Return, Ri Return, Ri-1 Return, Ri-2

7. Dynamic Programming • At any stage i, the next state (moving backward), Si-1 is a function only of state Si (entering state) and the decision made, Di, at that stage Decision, D1 State S1 Stage 1 State S0 (T1) Return, R1

8. Dynamic Programming • Ti is state transformation function • Ti transforms state Si to Si-1 given the decision Di (the form of T depends on the specific problem) Si-1 = Ti (Si, Di)

9. DP – Road Trip Example • Suppose you are traveling from the West Coast to the East Coast and you plan on taking 4 days for your trip. Each night you have a choice of cities as to where you will stay, but the decision each night has an impact on the distances that will need to be traveled in the future. • The diagram on the next slide shows the options you have and the distances between cities.

10. DP – Road Trip Example C11 7 4 C31 C21 6 9 4 8 5 7 8 3 C12 12 6 5 3 7 C41 C32 C22 C01 4 9 6 8 10 5 9 3 6 C13 10 6 C33 C23 6 8 C14 Stage 4 Stage 3 Stage 2 Stage 1

11. DP – Road Trip Example Naïve approach would say to look for the shortest route from left to right C11 7 4 C31 C21 6 9 4 8 5 7 8 3 C12 12 6 5 3 7 C41 C32 C22 C01 4 9 6 8 10 5 9 3 6 C13 10 6 C33 C23 6 8 C14 3 + 3 + 6 + 12 = 24

12. DP – Road Trip Example Stage 1 (From C1 to C0) States Total Distance Current city Travel to Distance 9 12 8 10 C11 C12 C13 C14 9 12 8 10 C01 C01 C01 C01

13. DP – Road Trip Example Stage 2 (From C2 to C1) Total Distance Current city Travel to Distance * 7 6 8 8 C21 C22 C11 C12 C13 C14 7 + 6 + 8 + 8 + 9 12 8 10 =16 =18 =16 =18 * * =15 =19 =17 =20 C11 C12 C13 C14 6 7 9 10 6 + 7 + 9 + 10 + 9 12 8 10

14. DP – Road Trip Example Stage 2 (From C2 to C1) Total Distance Current city Travel to Distance 9 6 6 8 C23 C11 C12 C13 C14 9 + 6 + 6 + 8 + 9 12 8 10 =18 =18 =14 =18 *

15. DP – Road Trip Example Stage 3 (From C3 to C2) Total Distance Current city Travel to Distance C31 C32 C21 C22 C23 4 4 7 4 + 4 + 7 + 16 15 14 =20 =19 =21 * 3 + 5 + 4 + 16 15 14 3 5 4 =19 =20 =18 C21 C22 C23 *

16. DP – Road Trip Example Stage 3 (From C3 to C2) Total Distance Current city Travel to Distance 5 3 6 C33 C21 C22 C23 5 + 3 + 6 + 16 15 14 =21 =18 =20 *

17. DP – Road Trip Example Stage 4 (From C4 to C3) Total Distance Current city Travel to Distance 5 3 6 C41 C31 C32 C33 5 + 3 + 6 + 19 18 18 =24 =21 =24 * Optimal Path : C41 C32 C23 C13 C01 21 Days

18. For Next Class • Read Balance of Chapter • Complete HW #1 • Case 3 Due