Asking the Oracle: Introducing Forecasting Principles into Agent-Based Modelling

1. Asking the Oracle: Introducing Forecasting Principles into Agent-Based Modelling Samer Hassan Javier Arroyo José M. Galán Luis Antunes Juan Pavón GRASIA, Universidad Complutense de Madrid INSISOC, Universidad de Burgos GUESS, Universidade de Lisboa

2. Contents • A recurrent issue • The Field of Forecasting • Forecasting Principles into ABM • Conclusions WCSS 2010

3. Photo

4. A recurrent issue… in ABM • How many times have we heard… • Are accuratepredictionspossible within ABM? • Should predictions be the main aim of ABM? • Is ABM matureenough to make proper predictions? • Stakeholders want predictions: is ABM an answer? WCSS 2010

5. A recurrent issue… in SIMSOC • “Does anyone know of a correct, model-based forecast of the impact of any social policy?” • Scott Moss, SIMSOC list, April 2009, “any correct policy impact forecasts?” WCSS 2010

6. A recurrent issue… in SIMSOC • “Does anyone know of a correct, model-based forecast of the impact of any social policy?” • Scott Moss, SIMSOC list, April 2009, “any correct policy impact forecasts?” • “The response, once misunderstandings were sorted out, was several accounts of reasons why policy impacts could not be forecast. The criteria I suggested for deeming a forecast to be correct was the correct forecast of the timing and direction of change of specified indicators.” • Scott Moss, SIMSOC list, June 2009, “what is the point?” WCSS 2010

7. A recurrent issue… in JASSS • Joshua Epstein (2008): • Prediction is one possible aim for ABM… among 16 others • Explanation • Guiding data collection • Raise new questions • Challenge theories • … • “Explanation does not imply Prediction” • Tectonics explain earthquakes but cannot predict them WCSS 2010

8. A recurrent issue… in JASSS • Joshua Epstein (2008): • Prediction is one possible aim for ABM… among 15 others • Explanation • Guiding data collection • Raise new questions • Challenge theories • … • “Explanation does not imply Prediction” • Tectonics explain earthquakes but cannot predict them • Thompson & Derr (2009): • “Good explanations predict” • An explanatory model is valid only if it predicts real behaviour WCSS 2010

9. A recurrent issue… in JASSS • Klaus Troitzsch (2009) • Epstein & Thompson discuss different “Prediction levels”: • Prediction of the kind of behaviour of a system, under arbitrary parameter combinations and initial conditions • Earthquakes occur because X and Y • Prediction of the kind of behaviour of a system in the near future • Region R is likely to suffer earthquakes in the following years because X and Y • Prediction of the state a system will reach in the near future • Region R will suffer an earthquake of power P in expected day D with confidence C • “Explanation does not imply 3rd level Prediction” • “Good explanations usually imply 1st or 2nd level” WCSS 2010

10. A recurrent issue • Agent-Based Modelling has multiple aims… • …but still modellers might seek prediction… • How could we help them? WCSS 2010

11. Contents • A recurrent issue • The Field of Forecasting • Forecasting Principles into ABM • Conclusions WCSS 2010

12. The field of Forecasting • Forecasting • A field focused on the study of prediction • Specially aiming 3rd level • 30 years experience • Consolidated (journals, conferences) • Formalised • “Forecasting experiment” WCSS 2010

13. The field of Forecasting • Using ABM as a Forecasting tool • Setting up a forecasting experiment: • Split data in two sets • “Objective” error measures • Compare the model • Fair Comparison WCSS 2010

14. The field of Forecasting • Using ABM as a Forecasting tool • Setting up a forecasting experiment: • Split data in two sets • Training set (in-sample): calibration • Test set (out-of-sample): validation • “Objective” error measures • Compare the model • Fair Comparison WCSS 2010

15. The field of Forecasting • Using ABM as a Forecasting tool • Setting up a forecasting experiment: • Split data in two sets • “Objective” error measures • Error(t)= forecasted(t) - actual_value(t) • Aggregated Error of time series: • Root Mean Square Error • Mean Absolute Error… • Compare the model • Fair Comparison WCSS 2010

16. The field of Forecasting • Using ABM as a Forecasting tool • Setting up a forecasting experiment: • Split data in two sets • “Objective” error measures • Compare the model • Benchmarks: other models, not necessarily ABM • Naïve method (at least): V’(t+1)= V(t) • Fair Comparison WCSS 2010

17. The field of Forecasting • Using ABM as a Forecasting tool • Setting up a forecasting experiment: • Split data in two sets • “Objective” error measures • Compare the model • Fair Comparison • Representative, large sample of forecasts • Ex-ante: forecast of (t+1) uses info available until (t) • Out-of-sample: not include training data in comparison WCSS 2010

18. Contents • A recurrent issue • The Field of Forecasting • Forecasting Principles into ABM • Conclusions WCSS 2010

19. Forecasting Principles into ABM • Principles of Forecasting • Armstrong (2001) with 40 authors • Summarising the best practices • Selection of subset for ABM • Six topics: • Modelling Process • Use of data • Space of solutions • Stake-holders • Validation • Replication WCSS 2010

20. Forecasting Principles into ABM • Modelling Process • Decompose the problem into parts • Bottom-up approach + combination of results • Structure problems that involve causal chains • Results of a (sub)model as input for next one • More accurate than global simulation • Consider the use of adaptive forecasting models • ABM as adaptive systems WCSS 2010

21. Forecasting Principles into ABM • Data-driven modelling • Use theory to guide the search for information on explanatory variables • Reduce complexity pruning design space in advance • Use diverse data sources • Increase of data reliability • Keep forecasting method simple • KISS • Select simple methods unless empirical evidence calls for a more complex approach • KISS + gradual increase of complexity on demand WCSS 2010

22. Forecasting Principles into ABM • Space of solutions • Identify possible outcomesprior to making forecasts • Avoid biases • Design test situations to match the forecasting problem • Put forward scenarios to rehearse policies • Adjust for events expected in the future • Expectability should guide design space exploration and what-if questioning WCSS 2010

23. Forecasting Principles into ABM • Stake-holders and Policy-makers • Obtain decision makers' agreement on methods • Ideally participatory simulation • Ask unbiased experts to ratepotential methods • Emphasising their role • Test the client's understanding of the methods • Including limitations of the model • Establish a formal review process to ensure that forecasts are used properly • Policy deployment should be controlled WCSS 2010

24. Forecasting Principles into ABM • Validation • List all the important selectioncriteriabefore evaluating methods • Temptation of redefining criteria to fit the outcomes • Use “objective” tests of assumptions • Quantitative approach to test assumptions when possible • Use extensions of evaluations to better generalise about what methods are best for what situations • Generalisation leads to applicability; based on what-if scenarios • Use error measures that adjust for scale in the data • Error measuring is as important as accuracy of data • Establish a formal review process for forecasting methods • Ensure verification, replication, trust WCSS 2010

25. Forecasting Principles into ABM • Replication • Compare track records of various forecasting methods • The role of replication for ABM verification • Assess acceptability and understandability of methods to users • Sharing of models & code • Describe potential biases of forecasters • From both modellers and stakeholders • How sensitive is the model to those biases? WCSS 2010

26. Contents • A recurrent issue • The Field of Forecasting • Forecasting Principles into ABM • Conclusions WCSS 2010

27. Conclusions • The choice of Agent-Based Modelling implies • Interest in the “what” is going to happen (Prediction) • Interest in “how” the phenomenon occurs (Understanding) • Prediction (3rd level) is a hard job • Financial crisis • Climate change • Forecasting Principles • Best practices, not a solution • Helpful in seeking the “what” WCSS 2010

28. Thanks for your attention! Samer Hassan samer@fdi.ucm.es Universidad Complutense de Madrid WCSS 2010

29. Contents License • This presentation is licensed under a Creative Commons Attribution 3.0 http://creativecommons.org/licenses/by/3.0/ • You are free to copy, modify and distribute it as long as the original work and author are cited WCSS 2010