sPHENIX Software & Computing Review

1. sPHENIX Software & Computing Review June 18, 2018 BNL sPHENIX computing framework and software development Chris Pinkenburg, BNL 1 sPHENIX Software & Computing Review

2. The Team - again Offline Computing Chris Pinkenburg Coordinator+Framework Martin Purschke Online Software Jin Huang Calorimeter Software Christof Roland Tracking Software Haiwang Yu Tracking Software Tony Frawley Tracking Simulations Nils Feege ePHENIX Software sPHENIX Software & Computing Review

3. Review Charge • Are the resources required to transmit and store the data adequately understood? • Are the resources required to process the data to a form suitable for physics analyses adequately understood? • Is the plan for developing software processes and framework adequately understood? • I will respond to this question, mainly. 3 sPHENIX Software & Computing Review June 18, 2018

4. 1st sPHENIX workfest, 2011 in Boulder Computing corner sPHENIX Software & Computing Review

5. A week later The first sPHENIX event display (of a pythia event) sPHENIX Software & Computing Review

6. A base class to implement detectors exist A week later G4 integrated into our framework as “just another module” Geant4 Hit Class defined Geant Hits can be saved generically and analyzed from file Pythia events can be simulated Truth Information implemented Detectors can be added and configured in macro sPHENIX Software & Computing Review

7. Software Development process • Small teams, no strict hierarchy • Constant communication • “workfests” – get together for a few days to work exclusively (and undistracted!) on something • 5 minute one on one discussions replace weeklong e-mail exchanges • Also very useful to get new members up to speed • Provide working example code which is kept up to date • Extensive use of 3rd party software (e.g. genfit, rave) • Close cooperation with RACF to make efficient use of hardware sPHENIX Software & Computing Review

8. DST Raw Data (PRDF) HepMC/Oscar Empty Histogram Manager Root File Structure of Fun4All User Fun4AllServer data objects Output Managers Input Managers DST Raw Data (PRDF) Analysis Modules HepMC Calibrations PostGres DB File Lightweight, standard C++, modular, flexible sPHENIX Software & Computing Review

9. Fun4All • Mature reconstruction/analysis framework, no difference between online/offline/simulation reconstruction • Used to reconstruct test beam data • Maintains global objects in tree structure (node tree) which can be navigated by analysis modules • Simple standard C++ and modular, easy to understand by somewhat computer savvy collaborators • Uses ROOT for I/O (to/from DST) and interpreter, no other strong dependency on ROOT • Uniform API’s – analysis modules work within online monitoring, calibration or reconstruction. • Enables Coordinated Analysis (“analysis taxi”) • Users buy into it sPHENIX Software & Computing Review

10. The Node Tree • The Node Tree is at the center of the sPhenix software universe • It is NOT a Root TTree • We have 3 different Types of Nodes: • PHCompositeNode: contains other Nodes • PHDataNode: contains any object • PHIODataNode: contains objects which can be written out to DST • PHCompositeNodes and PHIODataNodes can be saved to a DST and read back • This DST contains ROOT TTrees, the node structure is saved in the branch names. Due to ROOTs limitations not all objects can become PHIODataNodes (e.g. anything containing BOOST). This needs to be re-evaluated with ROOT 6 • We currently save 2 ROOT TTrees in each output file, one which contains the eventwise information, one which contains the runwise information • Input Managers put objects as PHIODataNodes on the node tree, output managers save selected PHIODataNodes to a file. • Fun4All can manage multiple independent node trees sPHENIX Software & Computing Review

11. Node Tree for sPHENIX Node Tree under TopNode TOP TOP (PHCompositeNode)/ DST (PHCompositeNode)/ G4HIT_HCALIN (PHIODataNode) G4HIT_ABSORBER_HCALIN (PHIODataNode) G4HIT_HCALOUT (PHIODataNode) G4HIT_ABSORBER_HCALOUT (PHIODataNode) SVTX (PHCompositeNode)/ SvtxHitMap (PHIODataNode) SvtxClusterMap (PHIODataNode) SVTX_EVAL (PHCompositeNode)/ SvtxClusterMap_G4HIT_SVTX_Links (PHIODataNode) RUN (PHCompositeNode)/ CYLINDERGEOM_SVTX (PHIODataNode) CYLINDERGEOM_SVTXSUPPORT (PHIODataNode) CYLINDERGEOM_EMCELECTRONICS_0 (PHIODataNode) CYLINDERGEOM_HCALIN_SPT (PHIODataNode) PAR (PHCompositeNode)/ SVTX (PHCompositeNode)/ SvtxBeamSpot (PHIODataNode) TOP: Top of Default Node Tree Creation and populating of other node trees is possible (used for embedding) sPHENIX Software & Computing Review

12. Node Tree for sPHENIX Node Tree under TopNode TOP TOP (PHCompositeNode)/ DST (PHCompositeNode)/ G4HIT_HCALIN (PHIODataNode) G4HIT_ABSORBER_HCALIN (PHIODataNode) G4HIT_HCALOUT (PHIODataNode) G4HIT_ABSORBER_HCALOUT (PHIODataNode) SVTX (PHCompositeNode)/ SvtxHitMap (PHIODataNode) SvtxClusterMap (PHIODataNode) SVTX_EVAL (PHCompositeNode)/ SvtxClusterMap_G4HIT_SVTX_Links (PHIODataNode) RUN (PHCompositeNode)/ CYLINDERGEOM_SVTX (PHIODataNode) CYLINDERGEOM_SVTXSUPPORT (PHIODataNode) CYLINDERGEOM_EMCELECTRONICS_0 (PHIODataNode) CYLINDERGEOM_HCALIN_SPT (PHIODataNode) PAR (PHCompositeNode)/ SVTX (PHCompositeNode)/ SvtxBeamSpot (PHIODataNode) • DST and RUN Node: default for I/O • DST – eventwise • RUN - runwise • Objects under the DST node are reset after • every event to prevent event mixing. You • can select the objects to be saved in the • output file. Subnodes like SVTX are saved • and restored as well. DST/RUN nodes can • be restored from file under other TopNodes • ROOT restrictions apply: • Objects cannot be added while running to • avoid event mixing sPHENIX Software & Computing Review

13. Module API • Init(PHCompositeNode *topNode): called once when you register the module with the Fun4AllServer • InitRun(PHCompositeNode *topNode): called whenever data from a new run is encountered • process_event (PHCompositeNode *topNode): called for every event • ResetEvent(PHCompositeNode *topNode): called after each event is processed so you can clean up leftovers of this event in your code • EndRun(const int runnumber): called before the InitRun is called (caveat the Node tree already contains the data from the first event of the new run) • End(PHCompositeNode *topNode): Last call before we quit topNode: Pointer to the Node Tree sPHENIX Software & Computing Review

14. DST structure • Fun4All internal node tree is mapped onto ROOT TTrees – one for eventwise quantities, one for runwise quantities • DST reading is synchronized, DSTs with different content can be read in parallel, without mixing events. Events which don’t exist in all input DSTs are dropped • This enables us to write multiple streams of separate content (e.g. clusters and tracks) and re-combine the content later during analysis • Rather than writing output files with different content for each customers, combine DSTs according to customer needs • Synchronization can be disabled for embedding studies • We can “swap out” a DST containing e.g. calorimeter towers created using a rough online calibration with a DST produced with the final calibration without having to touch or recreate the DST which contains tracks sPHENIX Software & Computing Review

15. Calibrations lookup Conditions DB ODBC – independent of underlying relational DB implementation Calibrations are stored as blobs – fast lookup and retrieval PHENIX needs 2-3 servers, one second granularity Supports tags for reproducibility Bank Id Insert Time Validity range (begin/end) sPHENIX Software & Computing Review

16. Code Quality Assurance • Code management system: git • pull requests, followed by informal code review • Nightly builds • Continuous integration (Jenkins, in beta testing) • State of the art code analysis tools • Valgrind • Cppcheck • Scan-build • Coverity • insure • Written coding conventions sPHENIX Software & Computing Review

17. Summary Is the plan for developing software processes and framework adequately understood? sPHENIX has a fully developed framework and a conditions DB. The framework is flexible and highly modular and fulfills the needs of sPHENIX. The available manpower is focused on the actual reconstruction software. Software development emphasizes the use of existing 3rd party software (e.g. genfit, rave) rather than starting from scratch. Special attention is paid to code verification and checking by using state of the art code analysis tools. Proposed code changes use pull requests followed by an informal code review, depth depending on the nature of the change. Setting up a continuous integration server (Jenkins) is in beta testing. 17 sPHENIX Software & Computing Review June 18, 2018

18. Overall Summary • Are the resources required to transmit and store the data adequately understood? Our data rates and total volumes are adequately understood. They are based on projections of our present run plan, and, where possible, informed by experience from the PHENIX experiment. • Are the resources required to process the data to a form suitable for physics analyses adequately understood? • We do have a good understanding of the required resources. They are based on our current run plan and detailed simulations of the proposed detectors in Geant 4. • Is the plan for developing software processes and framework adequately understood? • sPHENIX has a fully developed framework. Software development is done in small teams using state of the art code analysis tools 18 sPHENIX Software & Computing Review June 18, 2018

19. Back Up 19 sPHENIX Software & Computing Review sPHENIX Software & Computing Review June 18, 2018

20. Geometry, library deployment • Only one source for reconstruction and simulation • Currently afs, move to cvmfs • Use of containers sPHENIX Software & Computing Review