Why do models fail? Problems, problems …

1. Why do models fail?Problems, problems … John Rees BGS Head of Policy and Science Co-ordination Andrew Hughes BGS Groundwater Modeller and NMPI Co-organiser

2. Ideal vs reality

3. Generic issues as seen from a government institute - the British Geological Survey

4. BGS and numerical modelling Groundwater - Model Developer and userCoastal change - Poacher and gamekeeperContamination - often the ‘honest broker’

5. Typical reasons for failure • Lack of consistency of approach • Turnover and education of staff in client organisations • Language between user and modellers, and between modellers • Appropriate and sufficient data • Trust between parties • Unrealistic expectations • Honesty about the limitations of models • Cost of taking-up new developments

6. \ \ Policy makers Modellers \ Resource Managers

7. Key organisations • Universities • Consultants • Government Institutes • Regulators • Utilities • Government Ministries • Pan-governmental

8. Regulator issues • Limited number of staff that have modelling skills • Accessibility of models is often poor – they are often not easy to run • The maintenance and updating of models has a high cost (the leaky roof syndrome) • Inadequate data to support development or understanding • Expectations are often not met • Stakeholders are not adequately consulted

9. Resource manager issues • Common uncertainty about the specification and scale of model required • Lack of clarity about the needs of regulators • Find that models have too much uncertainty for detailed use • Inadequate timely stakeholder involvement • Find that models are not as flexible as managers would like

10. Model developer issues • Models are driven by policy and not the other way round e.g. Habitats Directive • Few examples of models being trusted by all parties • Stakeholders are recognised as important, but involvement is very variable • Personalities are important in defining how much stakeholders interact • Managing expectations is very important • Those consultants who deliver to spec and on time may not be the best to drive modelling forwards

11. Broader implications • Lack of efficiency associated with development encourages adoption or tweaking of older ‘industry standard’ models instead of adoption of newer models. • Drive to conservatism encourages usage of ‘tried-and-tested’ consultants who focus on delivery, rather than more innovative scientists who will introduce new concepts and stretching the modelling. • Acceptance of limitations (e.g. empirical constants, black-boxes) instead of driving better modelling.

12. Towards solutions • Need common understanding before attempting to formulate solutions - Climate change models are not taken up by some national governments that do not accept reality! • Language – differs markedly between disciplines • Definition of problem type, organisations involved and geographic extent • Problem is potentially huge, so need boundaries • Guidelines are needed

13. Summary

14. Questions ?