Verification and validation

1. Verification and validation Presented by: Lars-Göran Eriksson TF Leader : P. Strand Deputies: L-G. Eriksson, M. RomanelliEFDA CSU Contact Person: K. Thomsen

2. Importance of V&V for ITM-TF • It is likely that an important use of the ITM-TF tools/codes will be to simulate ITER scenarios with the aim of demonstrating that they are safe. • The reliability and accuracy of the ITM-TF tools must therefore conform to higher standards than are usual at present in the fusion modelling community. • Thus, two key activities are: Verification and validation • Quality assurance must also be dealt with at some level.

3. Verification: • The process of determining that a model implementation accurately represents the developer's conceptual description of the model and the solution to the model. (AIAA G-077-1998) • Validation: • The process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.(AIAA G-077-1998) American Institute of Aeronautics and Astronautics (AIAA), "Guide for the Verification and Validation of Computational Fluid Dynamics Simulations", AIAA G-077-1998.

4. Thus, the V&Vprocess is mainly a mechanism to help establish confidence in a code or model(s) performance and help assess its predictive capabilities. It should prove useful in guiding further physics exploration. • Many codes might well have been properly verified and validated. • However, without a uniform standard and traceability of code versions and experimental data used in the process, the evidence remains “anecdotic”. • The ITM-TF must therefore have its own well defined standards and procedures. • It must also have the tools to ensure traceability and proper documentation of all the factors that went in to the verification and validation process of a code. • Detailed implementation will depend on each IMP.

5. V&V and QA in General • Level 0– Ad Hoc approach (mainly at individual code level) • Non systematic approach, metrics and reporting left to individuals • No monitoring and forced adherence to any standards • Level 1 – Consistent V&V (codes or packages ) • Following predefined procedures • Detailed consistent reporting • Verified operation • Critical assessment of “performance” • Critical assessment of experimental applicability • Openly reported in standardized formats • Level 2 – Consistent QA (@ Organization or TF level) • A superset of requirements over level 1 relating to • Software Management • Software Engineering • Detailed procedure with checkpoints guaranteeing conformance • to quality and reliability goals • Level 3 – QA procedure under nuclear licensing requirements • ITER requirement for codes within Plant Operating Zone

6. Oversimplified view Qualification: Is the physics description adequate? Plasma Verification: Are the equations implemented and solved for correctly? Ongoingprocess! Qualification Validation Validation: Do we have a reliable and sufficiently accurate description of the plasma? Data Validity Data Validity: Is our measured data a sufficient representation of reality? Computational model Conceptual model Verification Code benchmarking: (C2C) A tool in both V&V and physics exploration TF V&V procedures: EFDA-TF-ITM(04)-8

7. IntegratedProject Timeline Steps in the validation procedure Review of Physics Basis Qualification Theory Limitations and applicability of the theory? Planning To obtain the domain of validity for the model Review of Code Implementation Verification Correctness of the numerical implementation? Coding Algorithms To estimate the accuracy of the calculations Experimentation Data Validity Quality of data? Need for new experiments? Quantify experimental uncertainties and errors Testing Modelling Validation Simulations Validation How to assess the validity of the model? Systematic evaluation of simulation results in terms of quantitative validation metrics Iteration between theorists/modellers/experimentalists will be necessary depending on the outcome of the different V&V steps. Predicting Time

8. From: ITM-TF V&V procedures: EFDA-TF-ITM(04)-8

9. Procedure is very important, but it is important to be aware of the major challenges involved. • Validation of a single code is often difficult. • To simulate most measured quantities, a series of codes must be run, i.e. it is not always easy to identify the weak link in the chain. • Thus, to define detailed procedures and metrics will require much careful consideration by the IMPs.

10. SELFO simulation (Full wave + 3D finite orbit width Fokker-Planck code. • Example of a step towards validation. • Gamma emission due to reactions between fast 3He and Be/C ions during (3He)D ICRF heating. • The calculation depends on an equilibrium; bulk plasma profiles; 3He concentration; antenna data; ICRF full wave power deposition calculation; Fokker-Planck calculation, including plasma response; nuclear data … JET Mantisinen et al. PRL 2002 • This example shows complexity often involved in validation against experimental data. • Although a very nice example of comparison with experimental data, it is not “validation” in the sense of the ITM procedure (traceability, reproducibility, tweaking factors …)

11. Conclusion • V&V activity is crucial for the credibility of the ITM-TF codes/tools. • ITM-TF wide standards and procedures should be applied. • IMPs are responsible for detailed implementation. • Verification is normally easier than validation. • Major difficulties are involved, e.g. it is often difficult to find clear-cut ways of validating a single code (often a series of codes and factors play a role). • V&V involves much work that is not necessarily viewed as “sexy”. • The importance of V&V for fusion plasma modelling is, however, increasingly being recognised. • For example Physics of Plasmas announced a while ago that they encouraged papers on V&V activity (one should also encourage other journals to adopt a similar attitude).