Game Theoretic Approaches to Solve H  Problems

1. Alex Feng Joint work with Brian D. O. Anderson, Alexander Lanzon and Michael Rotkowitz Game Theoretic Approaches to Solve H Problems

2. Global Motivation • Some H problems give Riccati equations which cause standard solvers to break down • We give a cure • The cure is extendable to linear periodic H problems and nonlinear game theory problems • This may be useful if there are numerical problems • Solution procedures are very few anyway, and numerically not well understood.

3. Outline • Global Motivation • Solving Continuous-Time H Riccati equations • Nonlinear Extension • Solving Discrete-Time H Riccati equations • Conclusions and Future Work

4. Outline • Global Motivation • Solving H Riccati equations • Detailed Motivation • Solving Riccati Equations, Kleinman and its repair • Algorithm Convergence • Game Theoretic Interpretation • Nonlinear Extension • Solving Discrete-Time Riccati Equations • Conclusions and Future Work

5. Detailed Motivation • Software to solve Asymptotic Riccati Equations arising in H2 problems is standard • The software can collapse on certain problems • The Kleinman algorithm will save the day: • Recursive • Requires Lyapunov equation solutions • Requires stabilizing gain to initialize • Converges quadratically (it is a Newton algorithm) • It does not extend to H equations (indefinite quadratic term) • What can we do?

6. Solving Riccati Equations Direct methods The example on the next transparency shows what can go wrong with an H-infinity Riccati equation if we use some direct methods. Computational disadvantages A numerical example

7. Numerical Problem Direct methods Too Large !! Direct methods problems are an old difficulty! It is a motivation for using the Kleinman algorithm in the LQ case.

8. Solving Riccati Equations Direct methods Iterative methods H2 control problem : Definite R Traditional Newton methods Computational disadvantages A numerical example LQ problem Solve AREs with R>0 Difficult to choose an initial condition Kleinman algorithm

9. Kleinman algorithm for H2 Question: Can we use Kleinman algorithm to solve H Riccati equations directly?

10. Diverge !! Kleinman algorithm for H The Kleinman algorithm cannot be used to solve H Riccati equations directly!

11. Solving Riccati Equations Direct methods Iterative methods H2 control problem : Definite R H control problem : Indefinite R Traditional Newton methods Computational disadvantages New algorithm ?? A numerical example LQ problem Solve AREs with R>0 Difficult to choose an initial condition Kleinman algorithm

12. Sign-indefinite Quadratic Term Problem setting

13. CARE Algorithm Sign indefinite quadratic term Simple initial choice Sign definite quadratic term Recursive algorithm using LQ Riccati equations , not Lyapunov equations!

14. Algorithm: A Summary Convert A sequence of H2 CAREs An H CARE Our algorithm Convert A sequence of less difficult problem A more difficult problem

15. Convergence • Global convergence is guaranteed provided the H control problem is solvable A monotone increasing matrix sequence is constructed to approximate the stabilizing solution of an H ARE. • Local quadratic rate of convergence A typical feature of Newton’s method

16. Game theoretic interpretation • Recall • Player u: minimize J; player w: maximize J. is the monotone increasing matrix sequence in our algorithm • Strategies for player u and w: uk+1 solves LQ, not game theory problem, when fixed wk is being used

17. Our algorithm Reduce an ARE with an indefinite quadratic term to a series of AREs with a negative quadratic term; Simple choice of the initial condition; A monotone non-decreasing matrix sequence. Easy Newton method (Kleinman) Reduce an ARE with an positive definite quadratic term to a series of Lyapunov equations; Difficult to choose an initial condition; A monotone non-increasing matrix sequence. Comparison of results For LQ game problems For LQ H2 problems

18. Periodic Equations--H2 • Some problems are periodic, e.g. satellite control • H2 periodic Riccati equations potentially have stabilizing periodic solution • Computational procedures are current research topic • Kleinman-likealgorithm for time-varying Riccati equations over a finite interval predates Kleinman algorithm • Kleinman-like algorithm for periodic time-varying Riccati equations over infinite interval yields stabilizing periodic solution as limit of solution of periodic Lyapunov differential equations

19. Periodic Equations--H • H periodic Riccati equations potentially have stabilizing periodic solution • Solution can be found by solving a sequence of H2 periodic Riccati equations (Kleinman-like algorithm will work for each one of these) • Game theoretic interpretation exists • No surprises in relation to the time-invariant case: • Local Quadratic Rate of Convergence. The algorithm carries over for the periodic case!!

20. Outline • Global Motivation • Solving H Riccati equations • Nonlinear Extension • Isaacs and HJB equations • Recursive solution of Isaacs via HJB equations • Quadratic convergence and game theoretic interpretation • Solving DARE • Conclusions and Future Work

21. Disturbance input HJB Isaacs One player game Nonlinear H-infinity control problem LQ Nonlinear Nonlinear optimal control LQ Game Riccati equations Linear H-infinity control problem LQ problem LQ and generalization

22. Summary of nonlinear result Hproblem solution H2 problem solution Iteration Linear-quadratic to Nonlinear-nonquadratic Linear-quadratic to Nonlinear-nonquadratic Isaacs problem solution HJBproblem solution Iterationcan be found 22

23. Problem Setting Sign indefinite term

24. Isaacs Nice initial condition HJB Nonlinear algorithm Question: How to solve HJB?

25. Solving HJB Hproblem solution H2 problem solution Kleinman iteration Isaacsproblem solution HJBproblem solution Iteration Linear PDE Iteration • Linear PDE iteration to give HJB solution stems from 1967 approx, though not carefully done for infinite time problem. Iteration exists • Other methods to solve HJB do exist

26. Convergence • Local convergence is guaranteed provided the H infinity control problem is locally solvable A monotone non-decreasing function sequence is constructed to approximate the stabilizing solution of an Isaacs equation. • Local quadratic rate of convergence

27. Game theoretic interpretation • Recall • Player u: minimize J; player w: maximize J. • Vk: The monotone increasing functionsequence in algorithm • Strategies for players u and w:

28. Our algorithm • Local quadratic rate of convergence • Simple initial choice • A natural game theoretic interpretation • Supposed to have a higher numerical accuracy and reliability Comparison Results Existing methods • Taylor Expansion Method • Stability not guaranteed • Converges slowly • Galerkin Approximation Method • Difficult to initialize • The Method of Characteristics • Converges slowly • Other Methods

29. Example (van der Schaft) Figure compares exact solution, iterations from method of characteristics, and iterations from our method.

30. A numerical example (continued)

31. Outline • Global Motivation • Solving H Riccati equations • Nonlinear Extension • Solving H Discrete-Time Riccati Equations • Discrete game problem • Mappings between generalized DARE and generalized CARE • The algorithm to solve DARE • Conclusions and Future Work

32. Discrete game problem

33. Discrete game problem Matrix inverse includes unknown variable P !!!

34. Mappings between DARE and CARE Generalized DARE • Why mapping?? • Stem from approximate 1960’s • Mappings between generalizedCARE and generalized DARE Generalized CARE

35. Mappings between DARE and CARE Assumption: Matrix A has no eigenvalue at -1 Other mappings exist !! LHS: parameters and solution of CARE RHS: parameters and solution of DARE

36. A Summary for Mappings Parameters of DARE Parameters of CARE Mapping Solution of DARE Solution of CARE

37. Mappings between DARE and CARE DARE we want to solve recursively Mappings exist !! CARE we have solved recursively

38. Algorithm to Solve DARE Given a DARE Mapping DARE into CARE Obtain the monotone sequence of the CARE Generate the monotone sequence of the DARE Compute the stabilizing solution of the DARE

39. Algorithm to solve DARE Monotone sequence in DARE Monotone sequence in CARE Mapping Game Theory Interpretation Game Theory Interpretation

40. Property of DARE Algorithm • Recursive • Simple initialization • Global convergence • Local quadratic rate of convergence • Game Theoretic Interpretation Properties of CARE algorithm Carry over !!

41. Conclusions and future work Conclusions • We developed a new algorithm to solve Riccati equations(Isaacs equations) arising in H control • We proved the global (local) convergence and local quadratic rate of convergence of our algorithm; • Our algorithm has a natural game theoretic interpretation.

42. Conclusions and future work Future work • More general Isaacs equations • Time-varying, periodic, other system structure • Viscosity case • Nonsmooth solutions • Stochastic case • Random noise in system, modifies Isaacs • Zero(Nonzero)-sum Multi-player game • Coupled HJB/Isaacs, but iteration style similar in existing algorithms • Discrete Isaacs equations

43. Acknowledgement • Andras Varga • Marco Lovera • Matt James • Minyi Huang • Weitian Chen