Calibration of Computer Simulators using Emulators

1. Calibration of Computer Simulators using Emulators

2. Recap –Emulators • We are concerned with complex, non-linear simulators • In this session we will look at calibration of such simulators • We will heavily depend on emulators • An emulator is a Gaussian process (or second order process) that interpolates the simulator output • Emulators are fast EGU short course - session 4

3. Calibration • Simulator users often want to tune the simulator using observations of the real system • Adjust the input parameters so that the simulator output matches observations as well as possible • Two very important points • Calibration will reduce uncertainty about x but will not eliminate it • It is necessary to understand how the simulator relates to reality • Model discrepancy EGU short course - session 4

4. Calibration and Assimilation • Calibration is concerned with the values of the inputs that are consistent with the data. • Assimilation is concerned with producing the best forecast/hindcast • Calibration changes the simulator inputs • Assimilation changes the simulator state variables EGU short course - session 4

5. Model discrepancy • Simulator output y = f(x) will not equal the real system valuez • Even with best/correct inputsx • Model discrepancy is the difference z – f(x) • As discussed in Session 1, model discrepancy is due to • Wrong or incomplete science • Programming errors, rounding errors • Inaccuracy in numerically solving systems of equations • Ignoring model discrepancy leads to poor calibration • Over-fitting of parameter estimates • Over-confidence in the fitted values EGU short course - session 4

6. History matching • History matching (a term taken from the petroleum industry) means finding sets of inputs that given simulator outputs that are ‘compatible’ with data • Calibration means finding a best value (or a distribution) for the inputs given the data EGU short course - session 4

7. Implausibility • Define a measure of implausibility (Imp) • If the implausibility is greater then ±3 those values of the inputs are deemed implausible • Because this is a function of the emulator not the original simulator runs we calculate it everywhere in input space EGU short course - session 4

8. Waves of Implausibility • Wave 1: Apply the implausibility measure. Mark part of input space as implausible • Wave 2: Add extra points in the not implausible region and rebuild the emulator. Repeat the implausibility measure • Wave 3+: Repeat until the implausible region ceases to grow EGU short course - session 4

9. A 1-d example EGU short course - session 4

10. EGU short course - session 4

11. EGU short course - session 4

12. EGU short course - session 4

13. EGU short course - session 4

14. EGU short course - session 4

15. EGU short course - session 4

16. Example -Galform EGU short course - session 4

17. Example - Galform • Galform is a simulator of Galaxy formation • It has 17 inputs • The amount of not implausible space in each wave is • None of the original 1000 member LHC was an acceptable fit to the data EGU short course - session 4

18. Calibration • In history matching we were simply looking for regions of input space that were not implausible given the data. • In calibration we want to find the ‘best input’x (and it associated uncertainty) EGU short course - session 4

19. Kennedy and O’Hagan(2001) • ζis the real system • z=ζ+εis data on the real system (ε~N(0,σ2)) • y=f(x) is the simulator output • d=ζ-y is the model discrepancy • ζ=f(x)+d • Build an emulator forfand simultaneously model the discrepancy as a GP • ζ=f*(x)+d* • z=f*(x)+d*+ε EGU short course - session 4

20. Kennedy and O’Hagan (2001) -2 • We can now perform an uncertainty analysis • This shows how much we have learned about the simulator inputs from the data • The mean/mode of the posteriors give us our estimate of the best value for the inputs EGU short course - session 4

21. Model Discrepancy Revisited • We have seen that we can use the model discrepancy to calibrate/history match the simulator • We can also look at the discrepancy between different simulators • This is particularly interesting if we have hierarchies of simulators EGU short course - session 4

22. Hierarchies of Simulators • Often we have hierarchies of simulators • Usually the resolution is increasing but additional processes could be added EGU short course - session 4

23. Hierarchies of Simulators • Rather than emulate each simulator separately • Emulate simulator 1 and then emulate the difference between each level • Need to have some runs at common inputs • Need few runs of expensive complex simulators EGU short course - session 4

24. Reified Simulators EGU short course - session 4

25. Reified Simulators EGU short course - session 4

26. Reified Simulators EGU short course - session 4

27. Reified Simulators • Reified simulators are ‘imaginary’ simulators that we impose between our simulators and reality • They are the ‘best’ simulator we could produce • Model discrepancy is split into two: • The discrepancy between the current simulator and the reified simulator • The discrepancy between the reified simulator and reality • Reification does not reduce the discrepancy. It might make it easier to elicit. EGU short course - session 4

28. Overview • Emulators are useful tools in the calibration of complex simulators • Two methods have been described: • History Matching – ruling out implausible regions of input space • Calibration – Finding ‘best fit’ input values • Reification may be useful in eliciting the relationship between simulators and reality EGU short course - session 4