Predictive data science for physical systems

1. Predictive data sciencefor physical systems From model reduction to scientific machine learning Professor Karen E. Willcox ICIAM 2019 Valencia, Spain July 19, 2019

2. Contributors Dr. Boris Kramer MIT → UCSD Elizabeth Qian MIT Renee Swischuk MIT Prof. Benjamin Peherstorfer Courant Institute Funding sources: US Air Force Computational Math Program (F. Fahroo); US Air Force Center of Excellence on Rocket Combustion (M. Birkan, F. Fahroo, D. Talley);SUTD-MIT International Design Centre

3. 1 Predictive Data Science What & why 2 Lift and Learn Projection-based model reduction as a lens through which to learn predictive models Outline 3 Application example Rocket engine combustion 4 Conclusions & Outlook

4. 1 Predictive Data Science Predictive Data Science 2 Lift & Learn 3 Application Example What and Why 4 Conclusions & Outlook

5. How do we harness the explosion of data to extract knowledge, insight and decisions? • Big decisions need more than just big data… • they need big models too Patient-specific prostate tumor modeling (T. Hughes) Arctic ocean circulation modeling(A. Nguyen & P. Heimbach) Inspired byCoveney, Dougherty, Highfield “Big data need big theory too” Hurricane storm surge modeling(C. Dawson)

6. Big decisions need more than just big data… • Big decisions must incorporate the predictive power, interpretability, and domain knowledge of physics-based models

7. Challenges • high-consequence applications are characterized by complex multiscale multiphysics dynamics • high (and even infinite) dimensional parameters • data are relatively sparse and expensiveto acquire • uncertainty quantification in model inference and certified predictions in regimes beyond training data Predictive Data Science a convergence ofData Science and Computational Science & Engineering

8. Learning from data through the lens of models…

9. Learning from data through the lens of models…

10. 1 Predictive Data Science Lift & Learn 2 Lift & Learn 3 Application Example Projection-based model reduction as a lens through which to learn predictive models 4 Conclusions & Outlook

11. Lift & Learn: Ingredients • A physics-based modelExample: modeling combustion in a rocket engine Conservation of mass (), momentum (), energy (), species (, , , • Lens of projection to define a low-dimensional model • Variable transformations that expose polynomial structure in the model • Non-intrusive learning of the reduced model → work with transformed variables P, kPa T, K YCH4 Q, MW/m3

12. = + = + Projection-based model reduction dimension solution time ~minutes / hours dimension solution time ~seconds 1Label: Solve PDEs to generate training data (snapshots) 2Identify structure: Compute a low-dimensional basis 3Train: Project PDE model onto the low-dimensional subspace

13. Full-order model (FOM) state Reduced models Approximate Residual: eqs ≫ dof 1Label 2Identify structure 3Train Project (Galerkin: ) Reduced-order model (ROM)state

14. Quadratic Model Linear Model FOM: FOM: ROM: ROM: Precompute the ROM matrices: Precompute the ROM matrices and tensor: • , , projection preserves structure ↔ structure embeds physical constraints

15. What is the connection between reduced-order modeling and machine learning? Machine learning “Machine learning is a field of computer science that uses statistical techniques to give computer systems the ability to "learn" with data, without being explicitly programmed.” [Wikipedia] Reduced-order modeling “Model order reduction (MOR) is atechnique for reducing the computational complexity of mathematical models in numerical simulations.” [Wikipedia] Model reduction methods have grown from CSE, with a focus on reducing high-dimensional models that arise from physics-based modeling, whereas machine learning has grown from CS, with a focus on creating low-dimensional models from black-box data streams. Yet recent years have seen an increased blending of the two perspectives and a recognition of the associated opportunities.[Swischuk et al., Computers & Fluids, 2018]

16. Variable Transformations & Lifting The physical governing equations reveal variable transformations and manipulations that expose polynomial structure

17. There are multiple waysto write theEuler equations conservative variablesmass, momentum, energy primitive variablesmass, velocity, pressure • Define specific volume: • Take derivative: Different choices of variables leads to different structure in the discretized system → lifting transformed systemhas linear-quadratic structure cf. specific volume variables

18. Consider the quartic system Simple example Introduce auxiliary variables: Chain rule: Lifting a nonlinear (quartic) ODE to quadratic-bilinear form Can either lift to asystem of ODEs orto a system of DAEs Need additional variable to make auxiliary dynamics quadratic: QB-ODE QB-DAE

19. Many different forms of nonlinear equations can be lifted to polynomial form original equations • quadratic-bilinear lifted equations

20. Operator inference Non-intrusive learning of the reduced models from simulation snapshot data

21. Given state data, learn the system Given state data () and velocity data (): In principle could learn a large, sparse systeme.g., Schaeffer, Tran & Ward, 2017 Find the operators by solving the least squares problem:

22. Given reduced state data,learn thereduced model Given reduced state data () and velocity data (): Operator Inference Peherstorfer & W.Data-driven operator inference for nonintrusive projection-based model reduction, Computer Methods in Applied Mechanics and Engineering, 2016 Find the operators by solving the least squares problem: • Under certain conditions, recovers the intrusive POD reduced model

23. Lift & Learn Variable transformations to expose structure + non-intrusive learning that frees us to choose our variables

24. Lift & Learn [Qian, Kramer, Peherstorfer & W., 2019] • Generate full state trajectories (snapshots)(from high-fidelity simulation) Learning alow-dimensional model Using only snapshot data from thehigh-fidelity model (non-intrusive) but learning the POD reduced model

25. Lift & Learn [Qian, Kramer, Peherstorfer & W., 2019] • Generate full state trajectories (snapshots)(from high-fidelity simulation) • Transform snapshot data to get lifted snapshots(analyze the PDEs to expose system polynomial structure) Learning alow-dimensional model Using only snapshot data from thehigh-fidelity model (non-intrusive) but learning the POD reduced model

26. Lift & Learn [Qian, Kramer, Peherstorfer & W., 2019] • Generate full state trajectories (snapshots)(from high-fidelity simulation) • Transform snapshot data to get lifted snapshots • Compute POD basis from lifted trajectories Learning alow-dimensional model Using only snapshot data from thehigh-fidelity model (non-intrusive) but learning the POD reduced model

27. Lift & Learn [Qian, Kramer, Peherstorfer & W., 2019] • Generate full state trajectories (snapshots)(from high-fidelity simulation) • Transform snapshot data to get lifted snapshots • Compute POD basis from lifted trajectories • Project lifted trajectories onto POD basis, to obtain trajectories in low-dimensional POD coordinate space Learning alow-dimensional model Using only snapshot data from thehigh-fidelity model (non-intrusive) but learning the POD reduced model

28. Lift & Learn [Qian, Kramer, Peherstorfer & W., 2019] • Generate full state trajectories (snapshots)(from high-fidelity simulation) • Transform snapshot data to get lifted snapshots • Compute POD basis from lifted trajectories • Project lifted trajectories onto POD basis, to obtain trajectories in low-dimensional POD coordinate space • Solve least squares minimization problem to infer the low-dimensional model Learning alow-dimensional model Using only snapshot data from thehigh-fidelity model (non-intrusive) but learning the POD reduced model

29. Lift & Learn [Qian, Kramer, Peherstorfer & W., 2019] • Generate full state trajectories (snapshots)(from high-fidelity simulation) • Transform snapshot data to get lifted snapshots • Compute POD basis from lifted trajectories • Project lifted trajectories onto POD basis, to obtain trajectories in low-dimensional POD coordinate space • Solve least squares minimization problem to infer the low-dimensional model Under certain conditions, recovers the intrusive POD reduced model → convenience of black-box learning +rigor of projection-based reduction +structure imposed by physics Learning alow-dimensional model Using only snapshot data from thehigh-fidelity model (non-intrusive) but learning the POD reduced model

30. 1 Predictive Data Science Rocket Engine Combustion 2 Lift & Learn 3 Application Example Lift & Learn reduced models for a complex Air Force combustion problem 4 Conclusions & Outlook

31. Modeling a single injector of a rocket engine combustor Oxidizer Manifold • Spatial domain discretized into cells • Pressure monitored at 4 locations • Oxidizer input: 0.37 of 42% / 58% • Fuel input: 5.0 of • Governing equations: conservation of mass, momentum, energy, species • Forced by a back pressure boundary condition at exit throat Injector Post Injector Element Combustion Chamber Exit Throat

32. Modeling a single injector of a rocket engine combustor Training data • 1ms of full state solutions generated usingAir Force GEMS code (~200 hours CPU time) • Timesteps; total snapshots • Variables used for learning ROMsmakes many (but not all) terms in governing equations quadratic • Snapshot matrix Test data Additional 1 ms of data at monitor locations (10,000 timesteps)

33. Performanceof learned quadratic ROM Pressure time traces at monitor location 1 Basis size

34. Performanceof learned quadratic ROM Pressure time traces at monitor location 1 Basis size

35. Pressure Temperature True Predicted POD modes Relative error

36. CH4 O2 True Predicted POD modes Normalized absolute error

37. 1 Predictive Data Science Conclusions & Outlook 2 Concrete Example 3 Application Example The future of Predictive Data Science 4 Conclusions & Outlook

38. Data Science Predictive Data Science Computational Science & Engineering Revolutionizing decision-making for high-consequence applications inscience, engineering & medicine

39. Predictive Data Science Learning from data through the lens of models is a way to exploit structure in an otherwise intractable problem. … Embed domain knowledge Integrate heterogeneous, noisy & incomplete data Respect physical constraints Bring interpretability to results Get predictions with quantified uncertainties

40. 1 2 Predictive Data Science Embedding domain knowledge Learning from data through the lens of models 3 4 Needs interdisciplinaryresearch & education at the interfaces Principled approximations that exploitlow-dimensional structure Explicit modeling & treatment of uncertainty

41. Data-driven decisions building the mathematical foundations and computational methods to enable design of the next generation of engineered systems KIWI.ODEN.UTEXAS.EDU

42. Our papers on this topic Peherstorfer, B. and Willcox, K., Data-driven operator inference for nonintrusive projection-based model reduction, Computer Methods in Applied Mechanics and Engineering, Vol. 306, pp. 196-215, 2016. Kramer, B. and Willcox, K., Nonlinear model order reduction via lifting transformations and proper orthogonal decomposition, AIAA Journal, Vol. 57 No. 6, pp. 2297-2307, 2019. Qian, E., Kramer, B., Marques, A. and Willcox, K., Transform & Learn: A data-driven approach to nonlinear model reduction. In Proceedings of AIAA Aviation Forum & Exhibition, Dallas, TX, June 2019.