Using uncertainty to test model complexity

1. Using uncertainty to test model complexity Barry Croke

2. Concept • Possible to test whether a model has an appropriate complexity through analysis of scatter in residuals • Take the standard deviation of the uncertainty distribution as a measure of uncertainty • Divide each residual by its uncertainty (need uncertainty in both observed and modelled values)

3. Concept • Result will be a distribution of values: If model has appropriate complexity, then the standard deviation of the distribution of normalised residuals should be near 1. • If standard deviation of normalised residuals >> 1 then model is too simple. • If standard deviation << 1 then model is too complex (has fitted to noise) • Result depends on the accuracy and precision of the uncertainty estimates

4. Complications • assumption is that the estimates of the uncertainties in the observed and modelled values are sufficiently well known. • A systematic over-estimation of the uncertainties will result in a reduction in the standard deviation of the normalised residuals leading to a bias toward simpler models. • an under-estimation of the uncertainties will bias the result to more complex models.

5. Complications • Further, any random noise in the uncertainty estimates will tend to increase the standard deviation of the normalised residuals, biasing the result towards more complex models. • Any structured noise in the uncertainty estimates may lead to an over- or under-estimation in the standard deviation of the normalised residuals. • Must be evaluated based on the confidence in the estimated uncertainties in the observed and modelled values.