GeantV prototype at a glance

1. GeantV prototype at a glance A.Gheata Simulation weekly meeting July 8, 2014

2. Content • Data structures • Baskets and basket management • Basket managers (per LV) • Track and basket lifecycle • Transport (physics and geometry) and track phases • Scheduler workflow

3. GeantTrack • Track identifiers • event, slot (memory management), track ID, PDG, G5 code • Particle identifiers • PDG, GeantV code, charge, mass, species • Kinematics • position, direction, momentum, energy • Status flags • status, N steps, N null steps, boundary flag, pending flag • Geometry/physics context • process, proposed step, current step, edep, distance to boundary, safety, current path, next path

4. GeantTrack TO FIX fBuffer GeantTrack GeantTrack_v 00 00 40 80 C0 vector 1 fEvent fEvslot fParticle fPDG … fXpos fYpos fZpos fXdir fYdir fZdir … Edep Pstep Snext Safety *fEventV *fEvslotV *fParticleV *fPDGV … *fXposV *fYposV *fZposV *fXdirV *fYdirV *fZdirV … *fEdepV *fPstepV *fSnextV *fSafetyV fParticle fEvslotV fEventV V fPDGV … fXPosV fYPosV SOA of fNtracks fZPosV … 192 bytes fNtracks=10 padding=32 fSnextV fSafetyV fPathV fNextpathV vector 2 *fPathV *fPath *fNextpathV *fNextpath GeantTrackPool<VolumePath_t*>

5. Track_v operations (overhead) • Pre-requirement to use vectorized: contiguity at the beginning of the arrays A Use Method(fXposV,…, fNtracks) or Method(GeantTrack_v &) … fEventV fParticleV • During transport, tracks stop leaving holes in the container A Compact A fEventV fParticleV B A Move … fEventV fParticleV Compact fEventV fParticleV fEventV fParticleV

6. Track_v operations (overhead) • Track selection according some criteria A A Reshuffle … fEventV fParticleV … fEventV fParticleV • Tracks have to be copied to a receiver during rescheduling B Copy A fEventV fParticleV Concurrency support C … fEventV fParticleV Copy fEventV fParticleV

7. GeantBasket • Elementary work unit for GeantV • They currently only hold tracks that are physically inside a given logical volume • Input GeantTrack_v array, filled by the scheduler • Output GeantTrack_v array, filled during transport • Baskets have thread local access during transport, but concurrent access during scheduling Transport Scheduler Input Scheduler Output Physics

8. Automatic basket scheduling • Concurrent track addition, garbage collection, collection of tracks from prioritized events • Adjustable threshold • Tvol= Ntracks_in_flight/2Nthreadsrounded to %4 (min 4, max 256) bottleneck Volume GeantScheduler Basket pool empty empty current empty empty BM fThreshold* Transport queue

9. Basket lifecycle Transport queue 1…Nvolumes empty TGeoVolume Push on threshold full Basket pool Basket manager transported current priority recycle Push on garbage collection AddTrack AddTrack Generator Scheduler Propagator 1…Nworkers 1…Nworkers

10. Track lifecycle PostStep (discrete) PhysicsSelect PropagateTracks fProcessV[i] fPstepV[i] kCrossing kNewkKilled(phys) kExiting Input tracks Output tracks kPhysics kKilled (geom) kKilled(phys) PostStep (continuous) fXposV[i], … fXdir[i], …, fPV[i], fEV[i]

11. PropagateTracks • kVector – continue in vector mode • kSingle – call PropagateTracksSingle at the given stage • kPostpone – copy remaining tracks to output • MarkRemoved + Compact – compact holes and copy these tracks to the output PostponedAction stage0 stage1 stage2 kSingle kPostpone kVector Propagate Close to bound. ComputeTransport Length<Single> Propagate Neutrals Propagate Safe<Pstep kCrossing kExiting kPhysics kCrossing kExiting kPhysics vector loop FindNextBoundary AndStep MarkRemoved Compact(output) fSnextV[i], fSafetyV[i] Propagate with safety

12. Propagation to boundaries • Safety-based approach algorithm very slow • What is the step in magnetic field which shifts the final particle position with no more than epsilon with respect to linear propagation? • If proposed step within isotropic safety: use safety • Otherwise take into account only safe_step value in competition with distance to boundary and proposed step ε = 1 micron safe_step = 2√ε/C C =1/R

13. Track stages Basket manager Transport queue Generator Imported Queued for pickup Pending (threshold) Being transported Priority dispatch Basket transport dispatch Scheduled Queued to be dispatched Schedulerqueue Scheduler

14. Scheduler • Pulls transported baskets, dispatches tracks to basket managers per volume • Not anymore! • Applying policies to: • Provide work balancing (concurrency) • Keep memory under control • Keep the vectors up (most of the time)

15. Scheduler workflow Digitize transported events and Inject new events into released slot Recycle transported baskets Check track counters Event done? Y Digitize event ImportTracks Last event done? Y EXIT Priority mode: the scheduler puts all tracks from priority events to special baskets, injected them at every loop regardless the content Priority is ON? Last PE done? Y Stop priority mode Y PE = prioritized event PE range = event number range for priority events Queue flushed? Flush priority baskets Y Collect prioritized tracks (once) Q size<min Y Adjust basket size Priority = ON PE range = (last,last+4) Garbage collect mode when the queue is empty: inject every basket regardless the content Empty Q? Y Garbage collect

16. Monitoring Main bottleneck: GeantObjectPool::Borrow/Return