A Technical Validation Module for the offline

1. A Technical Validation Module for the offline Auger-Lecce, 17 September 2009 Design The SValidStore Module Example Scripting Status

2. Technical Validation What we call technical validation is atesting framework of the AUGER offline tomonitor the code development process. What Technical Validation is not! It is not a validation of the goodness of the offline. It doesn’t assert “this result make sense or it does not make sense” What Technical Validation is: It just monitors that a physical quantity(variable) doesn't change during the development process and if it changes sends an alert message to the developers responsible of the change asking information. Report is on BuildBot waterfall page. Massive production/reconstruction validation do not enter in this kind of validation User debug utilities tools do not enter in this.

3. Design The structure of the validation procedure is based on the use of a reference root file containing a library of persistent objects. The reference validation objects, one for each Stage, contain the information that one would like to monitor in order to identify that a certain Stage or Module has undergone some changes between the releases. The whole process is managed by a single module (SValidStore) that is responsible for the access to the reference root file and for the evocation of the validation object (ValidationObj) The single ValidationObj knows the kind of information needed and how a persistent object to be stored in the reference file should be filled DPA Sequence/Module SValidStore reference root file TValidationObj ValidationObj Write or Check TValidationObj TValidationObj

4. The SValidStore module(1) The SValidStore Module is the manager of the validation procedure. His main task is to create the validation object and to perform actions on it, such as storing or checking. A module configuration xml file is used to define the parameters characterizing the algorithm. In the present version of the code this module has three parameters: Filename: name of the file where the reference results are stored. StoreObj: name of the validation object that has to be invoked in the current job Mode: mode of operation. Three mode are possible • recreate write in the reference file, if the file is not present a new file is created otherwise the file is overwritten. • update add a new object to the reference file • check compare the results obtained by the object in memory with the information previously stored in the reference file

5. The SValidStore module(2) SValidStore The base class of the persistent object is TValidationObj. This class encapsulates all the informations to check (Validate) the reconstruction stage. ValidationObj ValidationObj This is the base class of the validation object. This class is responsible for the filling of the persistent information, kept in the corresponding persistent object TValidationObj. Since the filling is strongly dependent from the kind of object, all member functions are made virtual and the implementation is deferred to the derived class. Write or Check TValidationObj TValidationObj TValidationObj

6. Example: FIOValidation- How to … (1) Step 1- Define a sequence to validate A sequence for FD monocular reconstruction contains: 1-Event Extractor 2-Signal Analyzer 3-Pulse Finder 4-SDP Finder 5-Axis Finder 6- Profile Reconstructor 7-Energy Finder Write the Module Sequence you want to Validate, inserting as last the module: <module> SValidStore <module> Step 2-Identify a set of significative variables to monitor SDPtheta: Shower Detector Plane normal-zenith SDPphi: Shower Detector Plane normal–azimuth Axistheta: Shower axis-theta Axisphi: Shower axis-phi Xmax: Depth at the shower max Eem: ElectroMagnetic Energy Etot: Total Energy … try to use variables covering the various Modules in order to better track down problems

7. Example: FIOValidation- How to…(2) class FIOValidationObj : public ValidationObj { ... virtual bool Fill(const evt::Event& event); ... } Step 3- Implement the Fill() and Validate() methods FIOValidationObj.h TFIOValidationObj.h class TFIOValidationObj : public TValidationObj { ... virtual bool Validate(TValidationObj * e); ... } The agreement between the set of variables has to be better than 10 -5 to pass the test- note that the tolerances and the matching are Step 4- Create the dictionary for the persistent object $ROOTSYS/bin rootcint –f (fileDict.cc) –c (file.h)

8. Example: FIOValidation- How to…(3) Step 4- few mods on SValidStore.cc to allow the instance of the new object. #include ‘’FIOValidationObj.h’’ ... ValidationObj* SValidStore::Fill(const string& objName) ... if(objName ==‘’FIOValidation’’|| objName ==‘’FIOValidationObj’’) return new FIOValidationObj(); Step 5- Write the SValidStore.xml Configuration file Step 6- compile and run…

9. Scripting • A set of scripts have been written for the Validation. • They allow to: • Run the validation assuming that all the code is installed in the default mode under the AUGER_BASE directory and assuming that the Validation code and scripting are installed in a ValidationTests sub-directory of AUGER_BASE • All the scripts are written to run in BuildBot environment. • The scripts • create all the need xml files to run the Offline with the validation module (ModuleSequence, bootstrap, EventFileReader, SValidStore, etc.) • Compile and run the Offline and check the exit status. If the status is not equal to 0 a test fault is send to BuildBot and reported in the waterfall web page.

10. Status • The ValidationObj implemented refer to: • FReconstruction ModuleSequence (old-non standard) • SReconstruction ModuleSequence • The tests perform nightly check of: • FRec and SRec stability • I/O (Write data in different format than read and reconstruct) • Automatic creation and use of logfile.xml • Documentation: • GAP first draft in place • Twiki page (first draft) in place: mainly a cut and paste from the GAP

11. to do (? !) • Reconstruction side • Use of standard Module Sequences • Reco for Hybrid • Reco for FD Stereo reconstruction • Introduction of Histograms. • Simulation side • How to compare a reference simulation with a new one? • It is not possible to define few quantities that monitor a simulation chain (like reconstruction). There are many used simulation modules for the same stuff. Therefore implementation of a FSimulation and a SSimulation ValidationObj is not enough. • Irrelevant changes in the random sequence can produce relevant changes in the simulated showers. It is necessary to implement a statistical comparison between reference and under-develop simulation chains-modules. Many showers must be generated in the same condition to create the distribution of values to compare. But this means important amount of CPU time needed to make a test…

12. to do (? !) • Trigger: • Fixed input ad hoc some tracks and check of reproducible modules • IO: • From Tom Paul: • checking that new releases of Offline can read files produced with older versions. • How to approach this : • Trigger the BuildBot build on EventIO change. • Input – A list of reference Events with different versions • A script running a read test • A script running the hybrid Simulation+Reconstruction and writing the Event chained with some reco/sim test. • Input Files selection for specific Module Sequences? • Hybrid Sim • Validatrix .... (not understood fully the architecture behind it! re-iteration with Darko, Tom)

13. to do (? !) • Do we want to set up a BuildBot slave? • If yes, it will be reasonable to have two configurations in place corresponding to the two kinds of software suites we have on the farm: 32/64 machines • Different operating system are of interest? If yes, can these be in place via virtual machines? • Documentation • The idea is to release the GAP in one week in such a way that any following work has a clear reference.