Advanced Residual Analysis Techniques for Model Selection

1. 3 2 5 1 Advanced Residual Analysis Techniques for Model Selection 4 University of Rome “Tor Vergata” A.Murari1, D.Mazon2, J.Vega3, P.Gaudio4, M.Gelfusa4, A.Grognu5, I.Lupelli4, M.Odstrcil5

2. The Scientific Method and Models • Model building is a innate faculty of human beings because it allows handling information in a much more economic way. Model • Model validation: process of assessing the quality of your model • Model selection: process of selecting the best model, among many, to interpret the available data. A “subject value” approach is advocated: both model selection and model validation are “utility” based

3. Model selection: introduction • Model selection =No established and universal methodology available • Model Falsification Criterion (MFC) : • Estimates the most appropriate model among a set of competing and independent ones • Based both on the accuracy and the robustness of the candidate models • Implements a form of falsification principle more than the ‘Occam razor’ • A model is not penalised for its complexity but on the basis of its lack of robustness

4. MFC : THE BASIC PHYLOSOPHY • ROBUSTNESS: a model is not penalised for its complexity but more for its lack of robustness, i.e. the fact that its estimates degrade if errors in the parameters are made • small errors introduced on each model parameter • study of the repercussions on the global estimates The repercussions of the parameter errors are quantified with some sort of information theoretic quantity (Shannon entropy) calculated for the residuals Details in the paper A.Murari et al “Preliminary discussion on a new Model Selection Criterion, based on the statistics of the residuals and the falsification principle” Conference FDT2 (Frontiers in Diagnostic Technologies)

5. The correlation tests method Hypothesis: the noise is random and additive Consequence: the residuals of a perfect model should be randomly distributed The model with the distribution of the residuals closer to a random one is to be preferred A random distribution of numbers (residuals) maximizes the Shannon entropy Hypotheses

6. MFC : Matematics an example • Mathematical expression of the Model Falsification Criterion : MFC1 MFC2 • rithe absolute value of the i-th residual • pirthe quantised probability of the i-th residual • rpar,i calculated after varying each parameter one at time (± 10 %) • ppar,i the quantised probability of this new residual • npar the number of model parameters • 1<i<n where n is the total number of experimental points

7. MFC : MATHEMATICS • Mathematical expression of the Model Falsification Criterion : A better model = smaller change in case of small errors introduced on the various parameters A better model = smaller sum of residuals + higher entropy of residuals The best among the candidate models is the one which presents the lowest value of the MFC indicator

8. NUMERICAL TESTS • A purely numerical equation : • exact solution + random noise of ± 10% = synthetic experimental data • Seven models created to fit the data

9. NUMERICAL TESTS

10. NUMERICAL TESTS

11. NUMERICAL TESTS

12. NUMERICAL TESTS

13. NUMERICAL TESTS

14. NUMERICAL TESTS

15. NUMERICAL TESTS

16. NUMERICAL TESTS • INTUITIVE CLASSIFICATION n Model 1 • Model 7 • Model 4 • Model 2 • Model 6 • Model 3 • Model 5 Error of ±10% introduced on each parameter one at time 1 Model classification obtained (classified in order of increasing MFC value) MFC evaluated for each model

17. NUMERICAL TESTS: Results • MFC AIC BIC B • Model 1 41 -105 -549 • Model 7 106 959372 • Model 4 120 889 435 • Model 2 130 955362 • Model 6 309 1071570 • Model 3 493 1231 700 • Model 5 5043 1634 1129 • Various forms of MFC criteria seem to outperform traditional criteria in particular for extrapolation and for high levels of noise

18. ScalingLaws: Numerical tests • Electron temperature required to access the H-mode of confinement in tokamak plasmas : • Variables scanned over their respective interval (using 500 values) • Synthetic experimental data generated by adding a random noise of ±10% • Five models considered to test the indicator • Bt R a n q N • Min 2 0.8 0.2 1 2 • Max 8 2 0.7 10 8

19. SCALING LAWS • INTUITIVE CLASSIFICATION n • Model 1 • Model 2 • Model 5 • Model 3 • Model 4 Error of ±10% introduced on each parameter one at time MFC evaluated for each model Model classification obtained (classified in order of increasing MFC value)

20. MFC Classification • Results of the MFC criterion : • INTUITIVE CLASSIFICATION MFC value n • Model 1 Model 2 = 2.40*106 • Model 2Model 1 = 3.21*106 • Model 5 Model 5 = 4.81*106 • Model 3 Model 3 = 1.12*107 • Model 4Model 4 = 1.22*107 • Results of the MFC criterion: since the exponent of ne is very low, at realistic noise levels the MFC realises that the models containing this quantity are prone to overfitting and that models without this parameter are more robust.

21. Residuals VS VS • small dependence from the density • when affected by an error bigger MFC value • not a fundamental variable • The MFC criterion also automatically penalises the major and minor radii from the scaling laws of individual devices because they do not vary over a significant range

22. ITPA DATABASE theoretical models variables used models generated n ChankinBt, q, Rmodel 1 Kernel collisionless Kernel collisional Bt, q, R, nmodel 2 Rogister ScottBt, nmodel 3 Shaing & Crume q, R, a, nmodel 4 none Bt, q, R, a, n model 5 none Bt, q, n model 6 linear regression • errors introduced = bounds of the 85% confidence interval lower upper • lower bound central value upper bound n • 3.05 3.25 3.44

23. ITPA DATABASE • JET : Linear regression VS well-known theoretical models

24. ITPA DATABASE • JET : 469 shots • MFC n • Model 6 4.49*104 • Model 3 4.49*104 • Model 5 2.08*105 • Model 2 2.18*105 • Model 1 3.27*105 • Model 4 7.88*105

25. ITPA DATABASE • ASDEX : 48 shots • MFC n • Model 6 4265 • Model 3 4664 • Model 5 1.83*105 • Model 1 2.22*105 • Model 2 4.11*105 • Model 4 1.05*106

26. ITPA DATABASE • CMOD : 98 shots • MFC n • Model 6 2.13*105 • Model 3 2.21*105 • Model 2 1.06*109 • Model 1 1.24*109 • Model 4 1.17*1010 • Model 5 1.49*1010

27. ITPA DATABASE • JET + ASDEX + CMOD : 615 shots • MFC n • Model 4 2.20*105 • Model 3 2.25*105 • Model 6 2.37*105 • Model 5 2.68*105 • Model 2 2.72*105 • Model 1 2.93*105

28. ITPA DATABASE: Summary • Summary of the best results obtained : • JET : • ASDEX : • CMOD : • All the database : The MFC determines that Te depends only on Bt, n and q but with not the same exponents at all Results are very different from the ones obtained with each independent tokamak

29. ITPA: Comparison Exponents • Analyse of the best results obtained : • Plasma radius, a and R, not evaluated as fundamental variables • Not the same exponents at all JET ASDEX CMOD

30. Summary and Future developments • The MFC criterion has some potential advantages compared to traditional criteria particularly in the case of scaling laws and extrapolation • The application to the ITPA database has given some interesting results (See also talk by I.Lupelli) • Model selection: various alternative MFC criteria are being applied to the power threshold to reach the H mode of confinement