1 / 50

Impact of parallelism on HEP software

Impact of parallelism on HEP software. April 29 th 2013 Ecole Polytechnique /LLR Rene Brun. Software Upgrades. All LHC experiments and groups like CERN/SFT are looking at all possible performance improvements or rethinking their software stack for the post LS2 years.

sorley
Download Presentation

Impact of parallelism on HEP software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Impact of parallelismon HEP software April 29th 2013 EcolePolytechnique/LLR Rene Brun

  2. Software Upgrades • All LHC experiments and groups like CERN/SFT are looking at all possible performance improvements or rethinking their software stack for the post LS2 years. • This effort is driven by the new hardware and also the analysis of the hot spots. • Work is going on in ROOT to support thread safety, parallel buffer merges and parallel Tree I/O. • In the GEANT world, several projects(eg G4MT) investigate multi-core, gpus or like solutions. In this talk I will review the progress with one of these projects.

  3. Hardware From a recent talk by Intel R.Brun : Paralllelism and HEP software

  4. If you trust Intel R.Brun : Paralllelism and HEP software

  5. If you trust Intel 2 R.Brun : Paralllelism and HEP software

  6. Vendors race parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism parallelism R.Brun : Paralllelism and HEP software

  7. Parallelism: manyfailures cray We failed in vectorizing codes like GEANT3 in 1985-1987 on CRAY, Cyber205, ETA10, IBM3090 because our approach was wrong inmos Some successful attempts in online systems in 1983 Are wegoing to take a wrongapproachagain? cm2 We failed too on MPP systems like the Thinking Machines, Elxsi in 1991-1993 because our approach was wrong R.Brun : Paralllelism and HEP software

  8. R.Brun : Paralllelism and HEP software

  9. Parallelism: key points Minimize the sequential/synchronization parts (Amdhal law): Very difficult Run the same code (processes) on all cores to optimize the memory use (code and read-only data sharing) Job-level is better than event-level parallelism for offline systems. Use the good-old principle of data locality to minimize the cache misses. Exploit the vector capabilities but be careful with the new/delete/gather/scatter problem Reorganize your code to reduce tails R.Brun : Paralllelism and HEP software

  10. Data Structures & parallelism C++ pointers specific to a process event event vertices Copying the structure implies a relocation of all pointers I/O is a nightmare tracks Update of the structure from a different thread implies a lock/mutex R.Brun : Paralllelism and HEP software

  11. Data Structures & Locality sparse data structures defeat the system memory caches For example: group the cross-sections for all processes per materialinstead of all materials per process Group objectelements/collections suchthat the storage matches the traversalprocesses R.Brun : Paralllelism and HEP software

  12. Tools & Libs geant5 geant3 geant4 geant1 geant2 bos minuit hbook root paw zbook zebra hydra R.Brun : Paralllelism and HEP software

  13. Detector Simulation tools All based on the same principle: Sequential particle transport

  14. The GEANT versions functionality G4 G3 G2 G1 1995 1975 2010 1980 1990 R.Brun : Paralllelism and HEP software

  15. Conventional Transport o o T2 o Eachparticletrackedstep by stepthroughhundreds of volumes o o o o o o o o o T4 o o o o o o o o o o o o o o o o o o o o o o o o o T1 o o o o o when all hits for all tracks are in memory summable digits are computed T3 R.Brun : Paralllelism and HEP software

  16. Analogywith car traffic 1 2 5 3 4 R.Brun : Paralllelism and HEP software

  17. StartingAssumptions • The LHC experiments use extensively G4 as main simulation engine. They have invested in validation procedures. Any new project must be coherent with their framework. • One of the reasons why the experiments develop their own fast MC solution is the fact that a full simulation is too slow for several physics analysis. These fast MCs are not in the G4 framework (different control, different geometries, etc), but becoming coherent with the experiments frameworks. • Giving the amount of good work with the G4 physics, it is unthinkable to not capitalize on this work. R.Brun : Paralllelism and HEP software

  18. Goals • Design a new detector simulation tool derived from the Geant4 physics , but with a radically new transport engine supporting: • Full and Fast simulation (not exclusive) • Designed to exploit parallel hardware  this talk

  19. Definitions A logical volume has a given shape and material

  20. Steps/lvolume in Atlas Hugedynamic range 7100 lvolumetypes 29 million instances R.Brun : Paralllelism and HEP software

  21. Simple observation: HEP transport is mostly local ! • Locality not exploited by the classical transportation approach • Existing code very inefficient (0.6-0.8 IPC) • Cache misses due to fragmented code 50 per cent of the time spent in 50/7100 lvolumes

  22. Neighbors/lvolume in Atlas Volumeswithtoomanyneighbors R.Brun : Paralllelism and HEP software

  23. Neighbors/lvolume in CMS Sameproblemwith CMS R.Brun : Paralllelism and HEP software

  24. LHCB geometrystatistics 90 per cent of steps in 50/700 volumes Better situation withneighborsbecause of a non cylindricalgeometry R.Brun : Paralllelism and HEP software

  25. New Transport Scheme o o T2 o All particles in the same volume type are transported in parallel. Particles entering new volumes or generated are accumulated in the volume basket. o o o o o o o o o T4 o o o o o o o o o o o o o o o o o o o o o o o o o T1 o o o o o Events for which all hits are available are digitized in parallel T3 R.Brun : Paralllelism and HEP software

  26. Tailsagain A killer if one has to wait the end of col(i) beforeprocessing col(i+1) Averagenumber of objects in memory R.Brun : Paralllelism and HEP software

  27. A better solution Checkpoint Synchronization. Only 1 « gap » every N events Pipeline of objects This type of solution requiredanyhow for pile-up studies R.Brun : Paralllelism and HEP software

  28. Abetterbetter solution checkpoints Ateachcheckpointwe have to keep the non finishedobjects/events. Wecannowdigitizewithparallelism on events, clear and reuse the slots. R.Brun : Paralllelism and HEP software

  29. Benchmarks/lessons from a prototype Benchmarking 10+1 threads on a 12 core Xeon HT mode Excellent CPU usage Locks and waits: some overhead due to transitions coming from exchanging baskets via concurrent queues Event re-injection will improve the speed-up 29

  30. R.Brun : Paralllelism and HEP software

  31. R.Brun : Paralllelism and HEP software

  32. R.Brun : Paralllelism and HEP software Vectorizing the geometry (ex1) Double_tTGeoPara::Safety(Double_t *point, Bool_t in) const { // computes the closest distance from given point to this shape. Double_tsaf[3]; // distance from point to higher Z face saf[0] = fZ-TMath::Abs(point[2]); // Z Double_tyt = point[1]-fTyz*point[2]; saf[1] = fY-TMath::Abs(yt); // Y // cos of angle YZ Double_tcty = 1.0/TMath::Sqrt(1.0+fTyz*fTyz); Double_txt = point[0]-fTxz*point[2]-fTxy*yt; saf[2] = fX-TMath::Abs(xt); // X // cos of angle XZ Double_tctx = 1.0/TMath::Sqrt(1.0+fTxy*fTxy+fTxz*fTxz); saf[2] *= ctx; saf[1] *= cty; if (in) return saf[TMath::LocMin(3,saf)]; for (Int_t i=0; i<3; i++) saf[i]=-saf[i]; return saf[TMath::LocMax(3,saf)]; } Huge performance gain expected in this type of code whereshape constants canbecomputedoutside the loop

  33. Vectorizing the geometry (ex2) G4double G4Cons::DistanceToIn( const G4ThreeVector& p, const G4ThreeVector& v ) const { G4double snxt = kInfinity ; // snxt = default return value const G4double dRmax = 100*std::min(fRmax1,fRmax2); static const G4double halfCarTolerance=kCarTolerance*0.5; static const G4double halfRadTolerance=kRadTolerance*0.5; G4double tanRMax,secRMax,rMaxAv,rMaxOAv ; // Data for cones G4double tanRMin,secRMin,rMinAv,rMinOAv ; G4double rout,rin ; G4double tolORMin,tolORMin2,tolIRMin,tolIRMin2 ; // `generous' radii squared G4double tolORMax2,tolIRMax,tolIRMax2 ; G4double tolODz,tolIDz ; G4double Dist,s,xi,yi,zi,ri=0.,risec,rhoi2,cosPsi ; // Intersection point vars G4double t1,t2,t3,b,c,d ; // Quadratic solver variables G4double nt1,nt2,nt3 ; G4double Comp ; G4ThreeVector Normal; // Cone Precalcs tanRMin = (fRmin2 - fRmin1)*0.5/fDz ; secRMin = std::sqrt(1.0 + tanRMin*tanRMin) ; rMinAv = (fRmin1 + fRmin2)*0.5 ; if (rMinAv > halfRadTolerance) { rMinOAv = rMinAv - halfRadTolerance ; } else { rMinOAv = 0.0 ; } tanRMax = (fRmax2 - fRmax1)*0.5/fDz ; secRMax = std::sqrt(1.0 + tanRMax*tanRMax) ; rMaxAv = (fRmax1 + fRmax2)*0.5 ; rMaxOAv = rMaxAv + halfRadTolerance ; // Intersection with z-surfaces tolIDz = fDz - halfCarTolerance ; tolODz = fDz + halfCarTolerance ; …… //here starts the real algorithm Allthesestatements are independent of the particle !!! Huge performance gain expected in this type of code whereshape constants canbecomputedoutside the loop R.Brun : Paralllelism and HEP software

  34. R.Brun : Paralllelism and HEP software Vectorizing the Physics • This is going to be more difficult when extracting the physics classes from G4. However important gains are expected in the functions computing the distance to the next interaction point for each process. • There is a diversity of interfaces and we have now sub-branches per particle type.

  35. Where are we now? • Present status • Several investigations of possible alternatives for “extremely parallel – no lock” transport • Not much code written, several blackboards full • Some investigation on a simplified but fully vectorized model to prove vectorization gain • New design in preparation

  36. Major points under discussion • How to minimize locks and maximize local handling of particles • How to handle hit and digit structures • How to preserve the history of the particles • This point seems more difficult at the moment and it requires more design • What is the possible speedup obtained by micro-parallelisation • What are the bottlenecks and opportunities with parallel I/O

  37. Current design List of logical Volumes BF: basket status (one char per B) Reused after each transport task Ev build thread Events Logical Volume lv List of baskets for lv p array Input particle list Transport thread p array Flushed at the end of event Output particle list Active event list History List of active events for lv Hits Digitizer thread Event ev Digits for lv and event ev Sensitive volumes

  38. Features • Pros • Excellent potential locality • Easy to introduce hits and digits • Cons • One more copy (but it is done in parallel) • More difficult to preserve particle history (it is non-local!) and introduce particle pruning

  39. Processing flow I • The transport thread takes particles from the input buffer and transports them till they stop, interact or exit from the volume • At this point they are inserted in the output particle buffer for further processing • If the LV is a sensitive detector, hits are generated and stored per LV basket • A LV basked history record is kept (under investigation) • Input and output particle buffers are fixed size structures, which can however evolve (be optimised) during simulation

  40. Design under study List of logical Volumes BF: basket status (one char per B) Logical Volume lv List of baskets for lv p array ✗ empty! Input particle list p array ✔ full! Output particle list Active event list History List of active events for lv Hits Sensitive volumes Event ev Digits for lv and event ev

  41. Note • Containers are “slow growing” contiguous containers • Every time a container has to grow, it is realloc-ated contiguously to the new size • A blocking operation • We expect containers size to converge • If not, there is a design problem

  42. Processing flow II • When an input particle buffer is exhausted • It is marked as such by the transport thread in the LV#BF (Logical Volume # Basket Flag) • Then the transport thread scans the LV#BF (Logical Volume # Basket Flag) data structure to find the next basket to be transported • Used buffers are scanned by the dispatcher thread that updates a global track counter per event • And then they are declared available to be filled (a) to be reused

  43. Important!! • a (available) basket being filled by the dispatcher • f (full) basket ready to be dispatched • r (ready) basket ready to be transported • t (transporting) basket being transported

  44. Current design asynch!! Dispatch thread Transport thread tt0) Initial status: the transport thread is transporting a basket, marked as “transporting” (t) dt1) The dispatch takes the first f basket and dispatches the output particles into the input particle lists of the baskets available to be filled (a) tt1) The transport thread has finished working on the input array dt2) When dispatching is finished the basket is moved from f to a tt2) The transport thread marks the lv#bn from transporting (t) to “to be dispatched” (f) dt3) When a input list is full, the basket is moved from a to r, ready to be transported and it is pushed into the fast selection list tt3) The transport thread gets a basket to be transported (r) from the fast selection list and marks it “transporting” (t) Dispatcher LV#BN Fast LV#BN queue LV#BN Logical Volumes LV LV BF: basket status BN r a t r a List of baskets for lv BF: basket status a a a BN f t p array Input particle list List of baskets for lv p array Output particle list p array Input particle list p array Output particle list

  45. Locks… • The only lock is the push and pop from the fast selection queue • The dispatcher watches continuously the done byte-vector and dispatch every new basket that is ready • Or it can sleep some and then process a number of done baskets in succession • The transport thread marks the done basket (no lock!) • No one touches a (t) basket apart the transport thread that deals with it • The transport thread gets a new basket (lock!) • This is to avoid that two threads get the same basket or that the dispatcher thread is updating the fast selection queue • The dispatcher thread does not need to lock the whole bit-array while dispatching • The basket in f or in a will not be touched by the transport threads • The only doubtful situation is when there are no basket to be transported… • In this case the “global” threshold for transporting should be lowered by a hungry transport thread (lock, but just to update an integer!) • And the dispatcher will mark baskets as ready to be transported (r)

  46. Memory • We hope to have a self-adjusting system that will stabilise with time • In case of an “accident” (an event much larger that any other), we need a way to “quench inflation” • We have identified two methods • Event flushing: do NOT transport particles from a given set of events and move them directly to the output buffer • Energy flushing: transport low energy particles and move the high energy ones to the output buffer • “Untransported” particles are just reinjected into the system, but they do not shower

  47. Processing flow III • Note an important point • The LV basket structure has input and output particle buffers and hits and history buffers • Input and output particle buffers are • Multi-event • Volatile, they get emptied and filled during transport of a single event • Hits and history buffers are • Per event • Permanent during the transport of a single event • A basket of a LV can be handled by different threads successively, each one with a new input and output buffers • …but all these threads will add to the Hits and history data structure till the event is flushed

  48. Processing flow IV • When an event is finished, the digitizer thread kicks in and scans all the hits in all the baskets of all the LVs and digitise them, inserting them in the LV event->digit structure • When this is over, the event is built into the event structure (to be designed!) by the event builder thread • After that, the history for this event is assembled by the same thread • If… • Then the event is output • By an output thread or in parallel?

  49. Questions? • How many dispatcher, digitizer and event-builder threads? • Difficult to say, we need some more quantitative design work • Measurements with G4 simulations could help • Transport thread numbers will have to adapt to the size of simulation and of the detector • In ATLAS for instance 50% of the time is spent in 0.75% of the volumes • Threads could be distributed proportionally to the time spent in the different LVs

  50. Short term tasks • Continue the design work – essential before any more substantial implementation • This is the most important task at the moment • We have to evaluate the potential bottlenecks before starting the implementation • Implement the new design and evaluate it against the first • Demonstrate speedup of some chosen geometry routines • Both on x86 CPUs and GPUs • Demonstrate speedup of some chosen physics methods • Particularly in the EM domain

More Related