1 / 22

ALFA - a common concurrency framework  for ALICE and FAIR experiments

ALFA - a common concurrency framework  for ALICE and FAIR experiments. Mohammad Al-Turany GSI-IT/CERN- PH. ALFA. Common layer for parallel processing . Common algorithms for data processing. Common treatment of conditions database. Common deployment and monitoring infrastructure.

eavan
Download Presentation

ALFA - a common concurrency framework  for ALICE and FAIR experiments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ALFA - a common concurrency framework  for ALICE and FAIR experiments Mohammad Al-Turany GSI-IT/CERN-PH

  2. ALFA • Common layer for parallel processing. • Common algorithms for data processing. • Common treatment of conditions database. • Common deployment and monitoring infrastructure. M. Al-Turany, CWG13 meeting 

  3. Current status of FairRoot CbmRoot R3BRoot SofiaRoot MPDRoot PandaRoot AsyEosRoot EICRoot FopiRoot FairRoot Run Manager IO Manager MC Application Event Display Runtime DB DB Interface Task Magnetic Field Module Detector Event Generator … … Root Libraries Cint ROOT IO TTree TGeo TVirtualMC TEve Proof Geant4 Genat4_VMC Geant3 VGM … … 5/18/13 M. Al-Turany, ACAT 2013, Beijing Florian Uhlig ROOT Users Workshop, Saas Fee M. Al-Turany, CWG13 meeting 

  4. What we went to do with FairRoot CbmRoot R3BRoot SofiaRoot MPDRoot PandaRoot AsyEosRoot EICRoot FopiRoot Building configuraion Testing Transport (FairMQ) Runtime DB Fair DB MC Application Module Detector Magnetic Field Event Generator Libraries ROOT CMake Geant4 Genat4_VMC Geant3 VGM ZeroMQ … M. Al-Turany, CWG13 meeting 

  5. How is it with ALFA and FairRoot? • AliRoot6 (O2) CbmRoot R3BRoot SofiaRoot MPDRoot PandaRoot AsyEosRoot EICRoot FopiRoot ALFA FairRoot ???? Building configuraion Testing Fair MQ Fair DB Module Detector DDS MC Application Magnetic Field Event Generator Runtime DB Libraries and Tools ROOT CMake Geant4 Genat4_VMC Geant3 VGM Protocol Buffers ZeroMQ BOOST … M. Al-Turany, CWG13 meeting 

  6. Current Status • The Framework delivers some components which can be connected to each other in order to construct a processing pipeline(s). • All components share a common base called Device • Devices are grouped by three categories: • Source: • Data Readers (Simulated, raw) • Message-based Processor: • Sink, Splitter, Merger, Buffer, Proxy • Content-based Processor: • Processor M. Al-Turany, CWG13 meeting 

  7. The Dynamic Deployment System (DDS) Should: • Deploy task or set of tasks • Use (utilize) any RMS (Slurm, Grid Engine, … ), • Secure execution of nodes (watchdog), • Support different topologies and task dependencies • Support a central log engine • …. See Talk by AnarManafov on Alice Offline week (March 2014) https://indico.cern.ch/event/305441/ M. Al-Turany, CWG13 meeting 

  8. Test Cluster • 8 machines from DAQ lab with SLC6-64 • aidrefma01-08 • FairRoot + FairSoft are installed • /local/home/cwg13/install/FairRoot • Tutorial3 can be used as a demonstrator M. Al-Turany, CWG13 meeting 

  9. Some remarks • Mixture of SL6.5 and SL6.4 • Binary compatible but I had to switch off the test in config.sh • /etc/issue is not available which also require a manual change for FairRoot • Is there any way to export the display from these machines? M. Al-Turany, CWG13 meeting 

  10. First test • Two Samplers • aidrefma03 and aidrefma08 • Six Processors • aidrefma04 and aidrefma05 • One File writer (sink) • aidrefma02 M. Al-Turany, CWG13 meeting 

  11. Processor1startTestDectorProcessor1.sh ID=“301" processorTask="FairTestDetectorMQRecoTask" numIoThreads="1" inputSocketType="pull" inputRcvBufSize=$buffSize inputMethod="connect" inputAddress="tcp://aidrefma08:5565" outputSocketType="push" outputSndBufSize=$buffSize outputMethod="connect" outputAddress="tcp://aidrefma02:5568" /local/home/cwg13/install/FairRoot/build/bin/testDetectorProcessor$dataFormat$ID $processorTask$numIoThreads$inputSocketType $inputRcvBufSize $inputMethod $inputAddress $outputSocketType $outputSndBufSize $outputMethod $outputAddress M. Al-Turany, CWG13 meeting 

  12. Processor2startTestDectorProcessor2.sh ID="401" processorTask="FairTestDetectorMQRecoTask" numIoThreads="1” // How many IO threads ZeroMQ should use. //it is relevant only after 1 GB/s per output. inputSocketType="pull" inputRcvBufSize=$buffSize// Output Buffer size (in # of messages for ZeroMQ, in bytes for nanomsg). inputMethod="connect" inputAddress="tcp://aidrefma08:5565" outputSocketType="push" outputSndBufSize=$buffSize outputMethod="connect" outputAddress="tcp://aidrefma02:5568” /local/home/cwg13/install/FairRoot/build/bin/testDetectorProcessor$dataFormat$ID $processorTask$numIoThreads$inputSocketType $inputRcvBufSize $inputMethod $inputAddress $outputSocketType $outputSndBufSize $outputMethod $outputAddress M. Al-Turany, CWG13 meeting 

  13. SamplerstartTestDetectorSampler.sh ID="101" inputFile=”~/install/FairRoot/example/…/data/testdigi_TGeant3.root" parameterFile=“~/install/FairRoot/example/…/data/testparams_TGeant3.root" branch="FairTestDetectorDigi" eventRate="0" numIoThreads="1" outputSocketType="push" outputBufSize=$buffSize outputMethod="bind" outputAddress="tcp://*:5565" /local/home/cwg13/install/FairRoot/build/bin/testDetectorSampler$dataFormat$ID $inputFile $parameterFile $branch $eventRate $numIoThreads $outputSocketType $outputBufSize $outputMethod $outputAddress M. Al-Turany, CWG13 meeting 

  14. FileSinkstartFileSink.sh ID="601" numIoThreads="1" inputSocketType="pull" inputRcvBufSize=$buffSize inputMethod=“bind" inputAddress="tcp://*:5568" /local/home/cwg13/install/FairRoot/build/bin/fileSin$dataFormat$ID $numIoThreads $inputSocketType $inputRcvBufSize $inputMethod $inputAddress M. Al-Turany, CWG13 meeting 

  15. The Test Processor1 Sampler Processor1 FileSink Sampler Processor1 M. Al-Turany, CWG13 meeting 

  16. From digits to hits with : Digits Hits Payload Payload TClonesArray TClonesArray Payload Payload TClonesArray TClonesArray Processor Sampler Sink M. Al-Turany, CWG13 meeting 

  17. Example: Sampler M. Al-Turany, CWG13 meeting 

  18. Example: Processor M. Al-Turany, CWG13 meeting 

  19. Example: FileSink M. Al-Turany, CWG13 meeting 

  20. The Test The processors are “connecting” on both input and output, thus They can be added (removed) On the fly Processor1 Processor1 Sampler Processor1 FileSink Sampler Processor1 Processor1 Processor1 M. Al-Turany, CWG13 meeting 

  21. M. Al-Turany, CWG13 meeting 

  22. FileSink M. Al-Turany, CWG13 meeting 

More Related