Aims

1. Aims • Build an emulator for a simple simulator. • Perform sensitivity analysis using this emulator. • Gather more of an understanding of how the simulator works. • What range of output can we expect? • Are the inputs equally influential? • Extension: Deploy and use the emulator.

2. Getting started • Sign up to emulatorization by clicking ‘Get started’ then ‘Sign up here’. • You have two options – build an emulator or perform sensitivity analysis. • We want to build an emulator first. http://uncertws.aston.ac.uk/emulatorization

3. Creating a project • Building an emulator consists of several stages: simulator specification, input screening (optional), design, simulator run, training and validation. • These are all grouped together in a project. • A project represents the building of an emulator for a single simulator. • Click ‘Build a new emulator project’.

4. Creating a project • The service URL is where to find the process you wish to emulate. • As the site is still under development, this is currently restricted to our own processing service. • Selecting a process will display a short description if provided. • Choose ‘SimpleSimulator’ and create the project.

5. The SimpleSimulator • The SimpleSimulator is an imitation crop yield model. • None of the parameters have any relation to their names! • This is a fairly trivial simulator. • Selected for the workshop as it is quick to evaluate. • Has 20 inputs and 3 outputs. • As we can only emulate one output at a time, we will be interested in FinalYield.

6. Specifying input ranges • The first step is to specify ranges on the inputs. • This is so the emulator is trained to cover the whole space. • These ranges could be determined through expert elicitation. • We have another tool for that called ‘The Elicitator’.

7. Specifying input ranges • Click ‘Simulator specification’. • You can see all of the inputs have been automatically set. • This is done using metadata provided by the process. • These ranges are in natural units. • It is also possible to set fixed inputs, which will not be included in the emulator. • This reduces the dimension of the input space, having a positive effect on emulator complexity. • But they are still used to evaluate the simulator.

8. Input screening • We can optionally use screening to identify important inputs. • These are inputs which have significant effect on the model output. • A basic form of sensitivity analysis requiring fewer runs of the simulator. • Once inconsequential inputs are identified, we can fix their value.

9. Input screening • From the menu, click ‘Input screening’. • We can set various parameters (in normalised units), but the defaults are sensible. • Number of trajectories controls the size of the screening design. • Discretisationlevel controls the size of the grid used to split the input space. • Create the input screening.

10. Input screening We need to select the output we are interested in, which is FinalYield. The x axis shows mean*EE: linear effects. The y axis shows stdEE: non-linear and interaction effects. This is a very important input. These are inconsequential inputs, they can be fixed by clicking on them.

11. Create a training design and run • A design is a sample set of inputs used to train and validate the emulator. • We use Latin-hypercube sampling to maximise the space between points in the design. • Takes into account the ranges for each input set in the previous step. • From the menu, click ‘Design’, set the design size, then create. • The size of the design should take into account how much will be used for training and validation. • A typical size for training is 10 * the number of inputs. • A typical size for validation is 5 * the number of inputs.

12. Create a training design and run • Once the design is created, you can view a histogram for each input. • This should show each bin with approximately the same number of points. • A design covering the whole space!

13. Create a training design and run • The next step is to evaluate the design with the simulator. • This is essential so the emulator knows what to fit to. • With more complex models this stage could take some time. • Every action like this is performed in the background, so you can close your browser and come back later. • Click ‘Create Run’ from the design result page.

14. Training the emulator • Once the run is finished, you can view a histogram for each output. • Now we have our design and simulator runs, we can train our emulator. • This is an iterative process: • Train using an initial set of parameters. • Validate against the simulator. • If results are unsatisfactory, re-train using different parameters.

15. Training the emulator • From the menu, click ‘Emulator’. • Choose ‘FinalYield’ as the output you wish to emulate. • Set the size of the design to use for training. • A minimum size is 10 times the number of inputs (of which we have 20).

16. Training the emulator • Keep normalisation enabled. • Means parameters vary over similar ranges. • Helps numerics. • API handles normalisation and de-normalisation for you.

17. Training the emulator • Keep a linear mean function. • Unless you know the simulator behaves differently.

18. Training the emulator • Clockwise (from top-left): zero, constant, linear, quadratic mean functions. • Blue line shows mean simulator output.

19. Training the emulator • Keep the Matern covariance function. • This is suitable for ‘rougher’ functions, the squared exponential for smooth. • Set the length scale multiplier to 0.5. • Controls the scale at which the Gaussian process varies.

20. Training the emulator • Nugget can be left disabled. • This allows for noise in the model. It is often used if the model has a stochastic element. • Can also help numerical stability to have a nugget. • Create the emulator.

21. Training the emulator • Once the emulator is trained, the result screen will show the optimised parameters. • It is possible to export the emulator (as JSON or MATLAB structure) at this stage. • But you might want to validate it first.

22. Validation • Validation is used to assess the accuracy of the emulator. • We evaluate the emulator with the remainder of the design set from training, and compare this with the related simulator run output. • From the menu, select ‘Validation’, set the desired design size and create the validation. • The simulator doesn’t have to be re-run.

23. Validation • Root mean square error (RMSE) shows goodness of fit and should be small. • Standard score (z-score) plot shows goodness of fit, but also accounts for variance. • If the scores are close to 0, this means emulator variance is too big (under confident emulator). • If the scores are higher than 2, this means the emulator variance is too small (over confident emulator).

24. Validation • Output comparison plots the output of the simulator (x) against the emulator (y). • We want the output of the simulator to match the emulator. • This plot has error bars to show how confident the emulator is for each point.

25. Validation • If your validation results are unsatisfactory, you can go back to the training stage to build an emulator: • with different parameters, • with more training points. • You might even go back to the screening stage and remove more inputs from consideration, or add some back. • Otherwise, we can now use the validated emulator for sensitivity analysis.

26. Sensitivity analysis • Sensitivity analysis helps us to understand our model better. • To get started, select the relevant option on the menu.

27. Create the project • As with an emulator, a project is used to group the stages of sensitivity analysis. • Click ‘Build a new sensitivity project’. • The Web application allows you to perform sensitivity analysis on a simulator or emulator. • Select your validated emulator and create the project.

28. Run the analysis • If you had selected a simulator, at this stage you would edit the input specifications. • This will take a lot longer as there are repeated calls to the simulator Web service. • As we are using a validated emulator, everything is ready to go. • Click ‘Analysis’ to begin.

29. Run the analysis • We can set various parameters here, but the defaults are sensible. • In addition to Sobol, the FAST method could also be used. • Don’t be fooled by the name, it’s actually not that fast! • Create the analysis.

30. Run the analysis The main effect shows the relative importance of an individual input variable. The total effect shows input importance both singularly and in combination with others. Both main and total effect are the same: this variable dominates and doesn’t have any interactions.

31. Extension: Deploying your emulator • We can also use the emulator as we would any other Web-accessible model. • The emulator must firstly be uploaded to the emulator running service: • Access the page above in a separate window/tab, and click ‘upload’. http://uncertws.aston.ac.uk:8080/ps-emulatorized

32. Extension: Deploying your emulator • Back on emulation tools, export your emulator as a JSON object. • At the emulator stage, click ‘Export’ then ‘JSON object’. • The object will appear in a text box, where it can be copied. • Switch to the upload page and paste the emulator JSON object into the correct field. • Choose an identifier for your emulator and upload. • Your name could be used here.

33. Extension: Deploying your emulator • On the running page you will now be able to select your emulator. • Once selected, some values based on the ranges will be automatically set. • Click ‘Run’ to evaluate the emulator. http://uncertws.aston.ac.uk:8080/ps-emulatorized/run.jsp

34. Extension: Deploying your emulator • The ‘Run an emulator’ page is basic. • Only for demonstration purposes. • If you wish to create a client, or use elsewhere, the WSDL is located at http://uncertws.aston.ac.uk:8080/ps-emulatorized/service?wsdl. • There is also a JSON description at http://uncertws.aston.ac.uk:8080/ps-emulatorized/service?jsondesc.

35. Further reading • The MUCM Toolkit contains details on the emulation and sensitivity analysis methods: http://mucm.aston.ac.uk/MUCM/MUCMToolkit/