1 / 57

FairRoot framework

FairRoot framework. Mohammad Al-Turany (GSI-Scientific Computing). FairRoot : Timeline. Start testing the VMC concept for CBM. Panda decided to join-> FairRoot: same Base package for different experiments. SOFIA (Studies On Fission with Aladin ). EIC (Electron Ion Collider BNL)

rasia
Download Presentation

FairRoot framework

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. FairRootframework Mohammad Al-Turany (GSI-Scientific Computing)

  2. FairRoot : Timeline Start testing the VMC concept for CBM Panda decided to join-> FairRoot: same Base package for different experiments SOFIA (Studies On Fission with Aladin) EIC (Electron Ion Collider BNL) EICRoot R3B joined 2013 2010 2004 2012 2006 2011 ENSAR-ROOT Collection of modules used by structural nuclear phsyics exp. First Release of CbmRoot MPD (NICA) start also using FairRoot GEM-TPC seperated from PANDA branch (FOPIRoot) ASYEOS joined (ASYEOSRoot) M. Al-Turany, ALICE offline week

  3. CbmRoot R3BRoot SofiaRoot MPDRoot Design PandaRoot AsyEosRoot FopiRoot EICRoot FairRoot Run Manager IO Manager MC Application Event Display Runtime DB DB Interface Task Magnetic Field Module Detector Event Generator … … Root Libraries Cint ROOT IO TTree TGeo TVirtualMC TEve Proof Geant4 Genat4_VMC Geant3 VGM Florian Uhlig ROOT Users Workshop, Saas Fee 13.03.13 … … M. Al-Turany, ALICE offline week

  4. Building & Testing system M. Al-Turany, ALICE offline week

  5. Testing and building system • CMake • Creates Makefiles (and/or project files) for different platforms. • Test support. • Large user base assures support. • CDash to handle data created with CMake • PHP framework • MySQL database • Both tools are open source. M. Al-Turany, ALICE offline week

  6. If someone experiments with new features in his local working copy and wants to test them (experimental build) 2. Configure, build and test on local machine 3. Send results automatically to central web page 1. Update (optional) 4. Dashboard prepares and display results Central SVN repository 5. Developer check results M. Al-Turany, ALICE offline week

  7. Central SVN repository If new code enters the central code base (continuous build) Dedicated test server 2. Basic checks: Style, etc 3. SVN triggers test server 4. Configure, build and test on local machine Pass 5. Send results automatically to central web page Fail Reject commit Developer commit code 7. In case of problems Dashboard sends an E-mail to Developer and Administrator 6. Dashboard prepares and display results M. Al-Turany, ALICE offline week

  8. From time to time a full check on all supported platforms should be done (nightly build) Central SVN repository 5. In the morning Developers and Administrators check their mails and the dashboard. And the development cycle starts again 4. In case of problems Dashboard sends an E-mail to Developer and Administrator 3. Dashboard prepares and display results 1. Update 2. Send results automatically to central web page M. Al-Turany, ALICE offline week

  9. From SVN to Git • With SVN we can only do very fast tests in the pre-commit • Moving to Git will help us to put more tests on the code before it is committed the Master branch M. Al-Turany, ALICE offline week

  10. Time based simulation In FairRoot M. Al-Turany, ALICE offline week

  11. How do events overlap? • In Detectors: • Sensor elements are still blocked from previous hits • Electronic is still busy • Hits too close in time cannot be distinguished • … • Special problem for CBM and PANDA: • Continuous beam with Poisson statistics (?)  many events with short time between them • No hardware trigger • Complex event reconstruction •  Necessary to simulate data stream as realistic as possible M. Al-Turany, ALICE offline week

  12. Time based simulation: Implementation • FairWriteoutBufferis Special buffer to store detector data between different events • You give the data you want to store an absolute time window this data is active in your detector and can influence later events. • If the same detector element is hit a second time the data is modified. • This is an abstract base class where you have to inherit from M. Al-Turany, ALICE offline week

  13. Time based simulation: Reading back data • FairRootManager has new reading algorithms, which make it possible to use the event wise implemented tasks to run on such data streams • Different algorithms available to extract data: • All data up to a given time • All data in a time window • All data between time gaps of a certain size • Other algorithms can be (easily) implemented M. Al-Turany, ALICE offline week

  14. Time-based simulation in FAIRROOT was presented in details with code https://indico.gsi.de/contributionDisplay.py?contribId=9&confId=1810 M. Al-Turany, ALICE offline week

  15. Fast Simulation • The same application, just different configuration: • Event generators just push the event into the stack, no transport is taking place • Detector response is presented as FairTasks (TTask) • The output has the same form as full simulation M. Al-Turany, ALICE offline week

  16. Simulation-Reconstruction Chain Determine particle properties at target vertex Event Generator Transport particles through the detector material Transport SIM Full Simulation Digitizer Determine detector response Fast Simulation RAW Determine physical space point parameters from detector hits Storage Levels Hit Finder Analysis Determine momentum vector and PID for all tracks Reconstruction Physics Analysis Calculate physics observables M. Al-Turany, ALICE offline week

  17. Klaus Götzen Fast Simulation: Concept M. Al-Turany, ALICE offline week

  18. Klaus Götzen ComparedtoFull Sim • ComparisontoFull Sim arereasonable(channel: pp  DsDs0) (10 k Signal events; absolute numbers) M. Al-Turany, ALICE offline week

  19. Next challenge is: Onlinevs. Offline orOnline + Offline ? 300 GB/s 20M Evt/s > 60 000 CPU-core or Equivalent GPU, FPGA, … < 1 GB/s 25K Evt/s How to distribute the processes? How to manage the data flow? How to recover processes when they crash? How to monitor the whole system? …… 1 GB/s 1 TB/s > 60 000 CPU-core or Equivalent GPU, FPGA, … M. Al-Turany, ALICE offline week

  20. FairRoot: Where we are now? • ROOT event loop • User code in Task hierarchy • Task hierarchy runs sequentially in one process • Tasks implement only algorithms (can be exchanged/replaced) T 1 T 5 T 2 Input File(s) Output File T 6 T 3 Parameter File(s) Database T 4 Parameter Manager t0 time t1 M. Al-Turany, ALICE offline week

  21. FairRoot: How to scale • Computer have more and more cores. • Online clusters of CBM and Panda will have about 60.000 cores • One monolithic program only use one of this cores • How we can better use the computing power of the modern computers? • C and C++do not offer any support for concurrency! • Embarrassingly parallel workload (Start as many FairRoot processes as cores are available) • Memory needed for each process expensive • How this scheme should work for the Online cluster? M. Al-Turany, ALICE offline week

  22. Design constrains • Highly flexible: • different data paths should be modeled. • Adaptive: • Sub-systems are continuously under development and improvement • Should work for simulated and real data: • developing and debugging the algorithms • It should support all possible hardware where the algorithms could run (CPU, GPU, FPGA) • It has to scaleto any size! With minimum or ideally no effort. M. Al-Turany, ALICE offline week

  23. Multi-processing vs. Multi-threading • Different processes are insulated from each other by the OS, an error in one process cannot bring down another process. • Inter-process communication can be used across network • Error in one thread can bring down all the threads in the process. • Inter-thread communication is fast M. Al-Turany, ALICE offline week

  24. The best would be to find the correct balance between reliability and performance • Multi-process concept with message queues for data exchange • Each "Task" is a separate process, which can be also multithreaded, and the data exchange between the different tasks is done via messages. • Different topologies of tasks that can be adapted to the problem itself, and the hardware capabilities. M. Al-Turany, ALICE offline week

  25. FairRoot: Where we are going ? (almost there!) • Each Task is a process (can be Multi-threaded) • Message Queues for data exchange • Support multi-coreand multi node t0 time t1 T 1 T 5 Input File(s) Output File T 2 T 4 T 6 T 3 Parameter File(s) Database Publish parameters (when new ones available) Parameter Manager M. Al-Turany, ALICE offline week

  26. Before Re-inventing the Wheel • What is available on the market and in the community? • A very promising package: ZeroMQ is available since 2011 • Do we intend to separate online and offline?NO • Multithreaded concept or a message queue based one? • Message based systems allow us to decouple producers from consumers. • We can spread the work to be done over several processes and machines. • We can manage/upgrade/move around programs (processes) independently of each other. M. Al-Turany, ALICE offline week

  27. ØMQ(zeromq) • A messaging library, which allows you to design a complex communication system without much effort • Abstraction on higher level than MPI (programming model is easier ) • Is suitable for loosely coupled and more general distributed systems • Multiplatform, multi-language (+30) • Small (20K lines of C++ code) • Large and active open source community. • Open source LGPL free software (large community) M. Al-Turany, ALICE offline week

  28. Current Status • The Framework delivers some components which can be connected to each other in order to construct a processing pipeline(s). • All components share a common base called Device (ZeroMQ Class). • Devices are grouped by three categories: • Source: • Data Sampler • Message-based Processor: • Sink, Splitter, Merger, Buffer, Proxy • Content-based Processor: • Processor M. Al-Turany, ALICE offline week

  29. Integrating the existing software: ROOT Files, Lmd Files, Remote event server, … Root (Event loop) ZeroMQ FairRootManager FairRunAna FairTasks Init() Re-Init() Exec() Finish() • FairMQProcessorTask • Init() • Re-Init() • Exec() • Finish() M. Al-Turany, ALICE offline week

  30. FairRoot: Example 3 4 -Tracking stations with a dipole field Simulation: 10k event: 300 Protons/ev Digitization Reconstruction: Hit/Cluster Finder M. Al-Turany, ALICE offline week

  31. From digits to hits with ROOT: Digits Hits TClonesArray TClonesArray M. Al-Turany, ALICE offline week

  32. From digits to hits with : Digits Hits Payload Payload TClonesArray TClonesArray Payload Payload TClonesArray TClonesArray Processor Sampler Sink M. Al-Turany, ALICE offline week

  33. Vs. TClonesArray TClonesArray TClonesArray TClonesArray Payload Payload Payload Payload TClonesArray TClonesArray Sampler Sink Processor M. Al-Turany, ALICE offline week

  34. 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 100 s 263 MB TClonesArray Throughput ~ 1000 ev/s Total Memory 263 Mb TClonesArray M. Al-Turany, ALICE offline week

  35. 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 100 s 2* 263 MB TClonesArray TClonesArray TClonesArray TClonesArray Throughput ~ 2000 ev/s Total Memory 526 Mb M. Al-Turany, ALICE offline week

  36. 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 121 s 4* 263 Mb TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray Throughput ~ 3300 ev/s Total Memory 1052 Mb M. Al-Turany, ALICE offline week

  37. 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 171 s 6 * 263 MB TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray Throughput ~ 3500 ev/s M. Al-Turany, ALICE offline week

  38. 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 300 s 8 * 263 MB TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray TClonesArray Throughput ~ 2660 ev/s M. Al-Turany, ALICE offline week

  39. 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory M. Al-Turany, ALICE offline week

  40. Before we continue: sampler Process that read from ROOT files and send each entry as a massege Proxy Bind on Input and Output sink Get payloads, save/convert to ROOT Objects (TClonesArrays) then to files processor Device class that contains the FairTask M. Al-Turany, ALICE offline week

  41. Wall time: 24 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 14.7 s 75 Mb processor sampler Proxy Proxy sink 15.3 s 76 Mb 14.9 s 167 Mb 2.9 s 114 Mb 3.8 s 71 Mb 24 s 217 Mb Push processor Push Throughput ~ 4166 ev/s Total Memory 720 Mb M. Al-Turany, ALICE offline week

  42. Wall time: 19.25 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 19.25s 35.7 Mb 16.86s 42.8 Mb processor sink sampler Proxy Proxy 19.25 s 35.7 Mb 16.47 s 151 Mb sink Push processor Push 16.75s 42.8 Mb Throughput ~ 5190 ev/s Total Memory 692 Mb M. Al-Turany, ALICE offline week

  43. Wall time: 20.91 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 16.5 s 35 Mb 16.5 s 43 Mb 3x 2x processor 16.9 s 151 Mb sink sampler Proxy processor Proxy sink 12.1 s 33 Mb 13.4 s 33 Mb Push processor Push Throughput ~ 4780 ev/s Total Memory 342 Mb M. Al-Turany, ALICE offline week

  44. Wall time: 25.8 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 20.7 s 78 Mb 25.8 s 215 Mb 3x 2x 16.1 s 167 Mb processor 2x sampler sink Proxy processor Proxy sampler sink 7.18 s 156 Mb 6.8 s 91 Mb Push processor Push Throughput ~ 7320 ev/s Total Memory 1245 Mb M. Al-Turany, ALICE offline week

  45. Wall time: 26.1 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 17.1 s 77 Mb 26.1 s 211 Mb 4x 2x 17.3 s 168 Mb processor 2x sampler processor sink Proxy Proxy processor sampler sink 8.5 s 113 Mb 7,1 s 176 Mb Push processor Push Throughput ~ 7400 ev/s Total Memory 1355 Mb M. Al-Turany, ALICE offline week

  46. Wall time: 30.5 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 27.1 s 35 Mb 26.1 s 44.2 Mb 4x 3x 24 s 151 Mb processor 2x sink Push Push sampler processor Proxy Proxy sink processor sampler 26.1 s 33.5 Mb 25.7 s 34 Mb sink processor Throughput ~ 6560 ev/s Total Memory 643 Mb M. Al-Turany, ALICE offline week

  47. Wall time: 23.74 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 24.6 s 35 Mb 23 s 47 Mb 4x 3x 22 s 151 Mb processor 2x sink Push Push processor sampler Proxy Proxy sink processor sampler 22.7 s 35.5 Mb 25.7 s 34 Mb sink processor Throughput ~ 8425 ev/s Gigabit Ethernet M. Al-Turany, ALICE offline week

  48. Wall time: 36.4 s 2 x 2.4 Xeon Quad core Intel Xeon 16 GB Memory 33 s 86 Mb 36.4 s 228 Mb 4x 3x 22 s 151 Mb 4x sampler processor sink Push Push sampler processor Proxy Proxy sink sampler processor 12.8 s 105 Mb 15.6 s 175Mb sink sampler processor Throughput ~ 10990 ev/s Gigabit Ethernet M. Al-Turany, ALICE offline week

  49. Summary • ZeroMQ communication layer is integrated into our offline framework (FairRoot). • On the short term we will keep both options: ROOT based event loop and concurrent processes communicating with each other via ZeroMQ. • On long term we are moving away from single event loop to distributed processes. M. Al-Turany, ALICE offline week

  50. Next Step: Design and development of a dynamic deployment system (DDS) • STORM is very attractive but no native support for C++ ! • We need to utilize any RMS (Resource Management system) • Support different topologies and process dependencies • Device (process) is a single entity of the system • Each device has its own watchdog process • Devices are defined by a set of props and rules, • All devices are statically inherited (should support) 3 interfaces: IDDSConfig, IDDSStatus, and IDDSLog • ….. Thank you M. Al-Turany, ALICE offline week

More Related