Introduction to the PROOF system root.cern.ch

1. Introduction tothe PROOF systemhttp://root.cern.ch René Brun CERN Do-Son school on Advanced Computing and GRID Technologies for Research Institute of Information Technology, VAST, Hanoi

2. Outline • PROOF essentials • Integrationwith experiment software • ALICE experience at the CAF René Brun

3. Run over data set Implement algorithm Make improvements PROOF essentials • Motivation: provide an alternative, dynamic, approach to end-user HEP analysis on distributed systems • Typical HEP analysis is a continuous refinement cycle • Data sets are collections of independent events • Large (e.g. ALICE ESD+AOD: ~350 TB / year) • Spread over many disks and mass storage systems • Exploiting intrinsic parallelism is the only way to analyze the data in reasonable times René Brun

4. catalog files query jobs data file splitting myAna.C PROOF essentials: classic approach Storage Batch farm queues submit merging final analysis manager outputs • static use of resources • jobs frozen: 1 job / worker node • manual splitting, merging • monitoring requires instrumentation René Brun

5. catalog files Storage PROOF farm scheduler query PROOF query: data file list, myAna.C feedbacks (merged) final outputs MASTER PROOF essentials: an alternative approach • farm perceived as extension of local PC • same syntax as in local session • more dynamic use of resources • real time feedback • automated splitting and merging René Brun

6. PROOF essentials: target Medium term jobs, e.g. analysis design and development using also non-local resources • Short analysis using • local resources, e.g. • end-analysis calculations • visualization Long analysis jobs with well defined algorithms (e.g. production of personal trees) • Optimize response for short / medium jobs • Perceive medium as short René Brun

7. PROOF essentials: design goals • Transparency • minimal impact on ROOT user habits • Scalability • full exploitation of available resources • Adaptability • cope transparently with heterogeneous environments • Real-time interaction and feedback • Addresses the case of • Central or Departmental Analysis Facilities (Tier-2’s) • Multi-core, multi-disks desktops René Brun

8. Trivial Parallelism René Brun

9. Terminology • Client • Your machine running a ROOT session that is connected to a PROOF master • Master • PROOF machine coordinating work between slaves • Slave/Worker • PROOF machine that processes data • Query • A job submitted from the client to the PROOF system.A query consists of a selector and a chain • Selector • A class containing the analysis code (more details later) • Chain • A list of files (trees) to process (more details later) René Brun

10. How to use PROOF • Files to be analyzed are listed in a chain ( TTree/TChain) • Analysis written as a selector ( TSelector) • Input/Output is sent using dedicated lists • If additional libraries are needed, these have to be distributed as packages ("par" = PROOF archive) Input Files (TChain) Analysis (TSelector) Output (TList) Input (TList) René Brun

11. Tree Branch Branch Branch x x x x x x x x x x y y y y y y y y y y z z z z z z z z z z Class TTree • A tree is a container for data storage • It consists of several branches • These can be in one or several files • Branches are stored contiguously (split mode) • When reading a tree, certain branches can be switched off  speed up of analysis when not all data is needed • Set of helper functions to visualize content(e.g. Draw, Scan) • Compressed File Branches René Brun

12. Chain Tree1 (File1) Tree2 (File2) Tree3 (File3) Tree4 (File3) Tree5 (File4) TChain • A chain is a list of trees (in several files) • Normal TTree methods can be used • Draw, Scan  these iterate over all elements of the chain • Selectors can be used with chains • Process(const char* selectorFileName) • After using SetProof() these calls are run in PROOF René Brun

13. PROOF essentials: what can be done? • Ideally everything that can be split in independent tasks • Currently available: • Processing of trees (see next slide) • Processing of independent objects in a file • Tree processing and drawing functionality complete LOCAL PROOF // Create a chain of trees root[0] TChain *c = CreateMyChain.C; // Start PROOF and tell the chain // to use it root[1] TProof::Open(“masterURL”); root[2] c->SetProof() // Process goes via PROOF root[3] c->Process(“MySelec.C+”); // Create a chain of trees root[0] TChain *c = CreateMyChain.C; // MySelec is a TSelector root[1] c->Process(“MySelec.C+”); René Brun

14. output list Selector Process() • Begin() • Create histos, … • Define output list • Terminate() • Final analysis • (fitting, …) OK preselection analysis n event branch read needed parts only branch leaf leaf branch leaf branch leaf leaf leaf leaf Chain n last 1 2 loop over events The ROOT data model: Trees & Selectors René Brun

15. xproofd PROOF essentials: multi-tier architecture One sub-master per geographic domain Structured master - adapt to clusters of clusters - improve scalability Heterogenous hardware / OS Node sessions started by Xrootd René Brun

16. PROOF essentials: connection layer • Sets-up the client session • Authentication, sandbox setup, start sessions on nodes • Based on xrootd • Light weight, industrial strength, networking and protocol handler • New PROOF-related protocol plug-in, xpd • xpd launches and controls PROOF sessions (proofserv) • xrootd act as a coordinator on the farm • Client disconnection / reconnection handled naturally • Can use the same daemon for data and PROOF serving René Brun

17. TSelector • Classes derived from TSelector can run locally and in PROOF • Begin() • SlaveBegin() • Init(TTree* tree) • Process(Long64_t entry) • SlaveTerminate() • Terminate() once on your client once on each slave for each tree for each event René Brun

18. Input / Output • The TSelector class has two members of type TList: • fInput, fOutput • These are used to get input data or put output data • Input list • Before running a query the input list is populatedgProof->AddInput(myObj) • In the selector (Begin, SlaveBegin) the object is retrieved: fInput->FindObject("myObject") René Brun

19. Input / Output (2) • Output list • The output has to be added to the output list on each slave (in SlaveBegin/SlaveTerminate)fOutput->Add(fResult) • PROOF merges the results from each slave automatically (see next slide) • On your client (in Terminate) you retrieve the object and save it, display it, ...fOutput->FindObject("myResult") René Brun

20. Input / Output (3) • Merging • Objects are identified by name • Standard merging implementation for histograms, trees, n-tuples available • Other classes need to implement Merge(TCollection*) • When no merging function is available all the individual objects are returned Result from Slave 1 Result from Slave 2 Merge() Final result René Brun

21. Chain Tree1 (File1) Tree2 (File2) Tree3 (File3) Tree4 (File3) Tree5 (File4) Workflow Summary Analysis (TSelector) Input (TList) proof proof proof René Brun

22. Output (TList) Output (TList) Output (TList) Merged Output Workflow Summary Analysis (TSelector) Input (TList) proof proof proof René Brun

23. PROOF essentials: dynamic load balancing • Pull architecture guarantees scalability • Adapts to definitive / temporary variation in performance Master Worker N Worker 1 René Brun

24. PROOF essentials: intrinsic scalability 1 user • Strictly concurrent user jobs at CAF (100% CPU used) • In-memory data • Dual Xeon, 2.8 GHz • CMS analysis • 1 master, 80 workers • Dual Xeon 3.2 GHz • Local data: 1.4 GB / node • Non-Blocking GB Ethernet 2 users 4 users 8 users I. Gonzales, Cantabria René Brun

25. PROOF essentials: exploiting multi-cores • Alice search for 0’s • 4 GB simulated data • Instantaneous rates (evt/s, MB/s) • Clear advantage of quad core • Additional computing power fully exploited René Brun

26. PROOF essentials: additional remarks • Intrinsic serial overhead small • requires reasonable connection between a (sub-)master and its workers • Hardware considerations • IO bound analysis (frequent in HEP) often limited by hard drive access: N small disks are much better than 1 big one • Good amount of RAM for efficient data caching • Data access is The Issue: • Optimize for data locality, when possible • Efficient access to mass storage (next slide) René Brun

27. PROOF essentials: data access issues • Low latency in data access is essential for high performance • Not only a PROOF issue • File opening overhead • Minimized using asynchronous open techniques • Data retrieval • caching, pre-fetching of data segments to be analyzed • Recently introduced in ROOT for TTree • Techniques improving network performance, e.g. InfiniBand, or file access (e.g. memory-based file serving, PetaCache) should be evaluated René Brun

28. PROOF essentials: scheduling multi-users • Fair resource sharing, enforce priority policies • Priority-based worker level load balancing • Simple and solid implementation, no central unit • Slowdown lower priority sessions • Group priorities defined in the configuration file • Future: central scheduler for per-query decisions based on: • cluster load, resources need by the query, user history and priorities • Generic interface to external schedulers planned • MAUI, LSF, … René Brun

29. PROOF essentials: management tools • Data sets • Optimized distribution of data files on the farm • By direct upload • By staging out from mass storage (e.g. CASTOR) • Query results • Retrieve, archive • Packages • Optimized upload of additional libraries needed the analysis René Brun

30. PROOF essentials: monitoring • Internal • File access rates, packet latencies, processing time, etc. • Basic set of histograms available at tunable frequency • Client temporary output objects can also be retrieved • Possibility of detailed tree for further analysis • MonALISA-based • Each host reports • CPU, memory, swap, network • Each worker reports • CPU, memory, evt/s, IO vs. network rate • pcalimonitor.cern.ch:8889 Network traffic between nodes René Brun

31. PROOF GUI controller • Allows full on-click control • define a new session • submit a query, execute a command • query editor • create / pick up a TChain • choose selectors • online monitoring of feedback histograms • browse folders with results of query • retrieve, delete, archive functionality René Brun

32. PROOF basics (1) • To loop over bigger statistics, PROOF can be used • (Re-)start your ROOT session • Connect to PROOF TProof::Open("<username>@<server>") • Create a chain of files that are stored in the PROOF cluster .L CreateESDChain.C chain = CreateESDChain("input.txt", 1000) René Brun

33. PROOF basics (2) • "Connect" the chain with PROOF chain->SetProof() • Draw something chain->Draw("fZ") chain->Draw("fTPCncls:fTPCnclsF", "", "COLZ") • Local (before): 1 file = 100 events • PROOF (now): 1000 files = 100.000 events René Brun

34. PROOF packages • (Re-)start your ROOT session • Connect to PROOF server TProof::Open("<username>@<server>") • Upload the ESD package (only first time; can be done each time) gProof->UploadPackage("ESD") • Build the package, load the library (each time) gProof->EnablePackage("ESD") • Show available packages gProof->ShowPackages() • Remove a package (don’t do this now, we need it) gProof->ClearPackage("ESD") gProof->ClearPackages() René Brun

35. Run selector locally • (Re-)start ROOT session • Load ESD library (needed for this data) .x loadlibs.C • Create a chain and add a local file chain = new TChain("esdTree") chain->Add("AliESDs.root") • Execute the selector chain->Process("TMySelector.cxx+") • Look at the output René Brun

36. Run selector in PROOF • (Re-)start ROOT session • Connect to PROOF server TProof::Open("<user>@<server>") • Create a (long) chain .L CreateESDChain.C chain = CreateESDChain("input.txt", 100) • Enable the ESD package gProof->EnablePackage("ESD") • "Connect" the chain with PROOF chain->SetProof() • Execute the selector chain->Process("TMySelector.cxx+") René Brun

37. Progress dialog Query statistics Abort query and view results up to now Show processing rate Show log files Abort query anddiscard results René Brun

38. Integration with experiment software: main issues • Finding, using the experiment software • Environment settings, libraries loading • Implementing the analysis algorithms • TSelector strengths • Automatic tree interaction • Structured analysis • TSelector weaknesses • Big macros • New analysis implies new selector • Change in the tree definition implies a new selector • Add layer to improve flexibility and to hide irrelevant details René Brun

39. Integration with experiment software • Experiment software framework available on nodes • Working group dedicated packages uploaded / enabled as PROOF packages (next slide) • Allows user to run her/his own modifications • Minimal ROOT environment set by the daemons before starting proofserv • Setting the experiment environment • Statically, before starting xrootd (inherited by proofserv) • Dynamically, by evaluating a user defined script in front of proofserv • Allows to select different versions at run time René Brun

40. PROOF package management • Allows client to add software to be used in the analysis • Uploaded in the form of PAR files (Proof ARchive) • Simple structure • package/ • Source / binary files • package/PROOF-INF/BUILD.sh • How to build the package (makefile) • package/PROOF-INF/SETUP.C • How to enable the package (load, dependencies) • Versioning support being added • Possibility to modify library / include paths to use public external packages (experiment libraries) René Brun

41. Integration with experiment software: ALICE • AliROOT analysis framework • Deployed on all nodes • Needs to be rebuilt for each new ROOT version • Versioning issue being solved • One additional package (ESD) needed to read Event Summary Data • Uploaded as PAR file • Working group software automatically converted to PROOF packages (‘make’ target added to Makefile) • Generic AliSelector hiding details • User’s selector derives from AliSelector • Access to data by member fESD TSelector AliSelector <UserSelector> René Brun

42. Integration with experiment software: ALICE • Alternative solution: • split analysis in functional modules (tasks) • Each task corresponds to well defined action • Tasks are connected via input/output data containers, defining their inter-dependencies • User creates tasks (derivation of AliAnalysisTask) and registers them to a task manager provide by the framework • The task manager, which derives from TSelector, takes care of the proper execution respecting the dependencies René Brun

43. Integration with experiment software: Phobos • TAM: Tree Analysis Modules solution • Modules structured like TSelector (Begin, Process, …) separating tree structure from analysis • Organization: • TAModule (: public TTask), base class of all modules • ReqBranch (name, pointer) • attach to a branch in Begin() or SlaveBegin() • LoadBranch (name) • Load the branch data in Process() • TAMSelector (: public TSelector) • Module running and management • Handle interaction with tree • TAMOutput: stores module output objects René Brun

44. Integration with experiment software: Phobos Example of user’s module class TMyMod : public TAModule { private: TPhAnTEventInfo* fEvtInfo; // event info TH1F* fEvtNumH; // event num histogram protected: void SlaveBegin(); void Process(); void TMyMod::SlaveBegin() { ReqBranch(“eventInfo”, fEvtInfo); fEvtNumH = new TH1F(“EvtNumH”,”Event Num”,10,0,10); } void TMyMod::Process() { LoadBranch(“eventInfo”); fEvtNumH->Fill(fEvtInfo->fEventNum); } René Brun

45. Integration with experiment software: Phobos Example analysis • Build module hierarchy • No PROOF • PROOF TMyMod* myMod = new TMyMod; TMyOtherMod* subMod = new TMyOtherMod; myMod->Add(subMod); TAMSelector* mySel = new TAMSelector; mySel->AddInput(myMod); tree->Process(mySel); TList* output = mySel->GetModOutput(); dset->AddInput(myMod); dset->Process(“TAMSelector”); TList* output = gProof->GetOutputList(); René Brun

46. Integration with experiment software: CMS • Environment: CMS needs to run SCRAM before proofserv • PROOF_INITCMD contains thepath of a script • The script initializes the CMS environment using SCRAM TProof::AddEnvVar(“PROOF_INITCMD”, “~maartenb/proj/cms/CMSSW_1_1_1/setup_proof.sh”) #!/bin/sh # Export the architecture export SCRAM_ARCH=slc3_ia32_gcc323 # Init CMS defaults cd ~maartenb/proj/cms/CMSSW_1_1_1 . /app/cms/cmsset_default.sh # Init runtime environment scramv1 runtime -sh > /tmp/dummy cat /tmp/dummy René Brun

47. Integration with experiment software: CMS • CMSSW: software framework provides EDAnalyzer technology for analysis purpose • Write algorithms that can be used with both technologies (EDAnalyzer and TSelector) • Possible if well defined interface class MyAnalysisAlgorithm { void process( const edm::Event & ); void postProcess( TList & ); void terminate( TList & ); }; • Used in a TSelector templated framework TFWLiteSelector TFWLiteSelector<MyAnalysisAlgorithm> René Brun

48. Integration with experiment software: CMS • In PROOF selectors libraries distributed as PAR file // Load framework library gSystem->Load(“libFWCoreFWLite”); AutoLibraryLoader::enable(); // Load TSelector library gSystem->Load(“libPhysicsToolsParallelAnalysis”); René Brun

49. ALICE experience at the CAF • Alternative to using the Grid • Massive execution of jobs vs. fast response time • Available to the whole collaboration • number of users will be limited for efficiency reasons • Design goals • 500 CPUs, 200 TB of selected data locally available • CERN Analysis Facility used for short / medium tasks • p-p prompt analysis, Pb-Pb pilot analysis • Calibration & Alignment René Brun

50. ALICE at CAF: example of data distribution Total: 200 TB 20% Last day RAW events 3.2M PbPb or 40M pp 20% Fixed RAW events 1.6M PbPb and 20M pp 20% Fixed ESDs 8M PbPb and 500M pp 40 % Cache for files retrieved from AliEn Grid, Castor Sizes of single events from Computing TDR René Brun