Generation of Pareto Optimal Ensembles of Calibrated Parameter Sets for Climate Models

1. Generation of Pareto Optimal Ensembles of Calibrated Parameter Sets for Climate Models Keith Dalbey, Ph.D. Sandia National Labs, Dept 1441, Optimization and Uncertainty Quantification Michael Levy, Ph.D. Sandia National Labs, Dept 1442, Numerical Analysis and Applications December 12-17, 2010 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under Contract DE-AC04-94AL85000.

2. Outline • Motivation • Approach: Pareto Ensemble • What Does “Pareto Optimal” Mean? • Finding a “Pareto Optimal” Ensemble • Results of Tuning Climate Model • Summary & Future Work References Jackson et al, “Error reduction and convergence in climate prediction,” Journal of Climate, 2008. Eddy & Lewis, “Effective generation of pareto sets using genetic programming,” Proc. of ASME Design Engineering Technical Conference, 2001. Dalbey & Karystinos, “Fast generation of space-filling latin hypercube sample designs,” Proc. of 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2010.

3. Motivation Calibrating (tuning) climate models • choosing values of model parameters to predict well Is difficult because • They have many inputs and outputs • Diverse parameters sets can match observations similarly well • Errors can compensate: “2 wrongs can make a right” under historical conditions • Climate change (new conditions) might expose a previously hidden mis-calibration, so… History matching is necessary but not sufficient for good predictions. • The future is uncertain, but we can quantify the uncertainty (estimate statistics) for possible future climates.

4. Approach: Pareto Ensemble How can we make good statistical predictions? Use a diverse ensemble of “good” parameter sets to determine the range/spread of possible future climates QUESTION:What’s the definition of a “good” parameter set? There are multiple outputs and what’s good for one output can be bad for another. (AN) ANSWER: It’s Pareto optimal. A point (parameter set) is Pareto optimal if there is no other point that is as good or better than it in ALL outputs. What does the “Pareto” mean? It’s just the name of the person who discovered it… Vilfredo Federico Damaso Pareto was an Italian engineer, sociologist, economist, and philosopher.

5. What Does “Pareto Optimal” Mean? 2D Pareto front schematics

6. What Does “Pareto Optimal” Mean? • Usually, the current approx. of the true Pareto front. • ThePareto front defines the “zero sum game” of all optimal compromises you could make. • Unlike a weighted combination of objective functions, it lets you choose a specific compromise/ combination AFTER the optimization is complete. • It does NOT say which compromise/combination is best, just what all the “optimal” choices are. • It says “Don’t choose anything Pareto non-optimal because there’s something better in all criteria.”

7. Finding a “Pareto Optimal” Ensemble • Used the Multi Objective Genetic Algorithm (MOGA) in DAKOTA’s (Design Analysis Kit for Optimization and Terascale Applications) JEGA (John Eddy’s Genetic Algorithm) sub-package • GA’s typically need 1000’s of simulations, I could only afford  1000… • Used test problem (find surface of radius=1 6D hyper-sphere in input space, 10 outputs) to tune MOGA settings and initial population (space-filling, specifically Binning Optimal, Symmetric Latin Hypercube Sampling, or BOSLHS), for: • Large Pareto Ensemble • Mean radius close to 1 • Uniform spread • Small radius variance

8. Finding a “Pareto Optimal” Ensemble Use DAKOTA’s MOGA on a test problem with 6 inputs and 10 outputs; true solution is a radius 1 hypersphere PDF’s of the Pareto Ensemble’s # of points Point spread Mean radius Standard deviation of radius 1 2 3 4 Default Monte Carlo seed

9. Finding a “Pareto Optimal” Ensemble Use DAKOTA’s Multi Objective Genetic Algorithm on a test problem with 6 inputs and 10 outputs true solution is a radius 1 hypersphere Default Monte Carlo seed BOSLHS seed

10. Results of Tuning Climate Model

11. Summary & Future Work • Climate model parameters that match history well might not predict well (climate change might expose a previously hidden mis-calibration of parameters). • Plan: Use a diverse ensemble of “good” (Pareto optimal) parameter sets to determine the range/spread of possible future climates. • Used MOGA to find a (very large) Pareto optimal ensemble of calibrated parameter sets. • Next steps: • down select Pareto optimal ensemble, and • simulate smaller ensemble out to 2100.

12. Some “Good” Parameter Sets