1 / 8

Analysis Framework - status

Analysis Framework - status. Andrei Gheata , Mihaela Gheata , Andreas Morsch ALICE Offline Week – 09 March 2011. Analysis and analyzers. T he framework enters a more mature phase Focusing more on stability and error detection

peigi
Download Presentation

Analysis Framework - status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Analysis Framework - status Andrei Gheata, MihaelaGheata, Andreas Morsch ALICE Offline Week – 09 March 2011

  2. Analysis and analyzers • The frameworkenters a more mature phase • Focusing more on stability and errordetection • Providing more and more generalpurpose utilities (eventtopology, merging, OADB) • Increasing control on EUB (EraticUser Behavior) both in the coreframework and back-ends • User communitydeveloping a solidcommonknowledge base • More and more  experienceduserscontributingdirectly to the frameworkdevelopment (AOD, I/O optimizations, mixing, …) • Manythanks to all ! • A large and proactive user comunityexchangingexperience on the verypopularanalysis mailing list • While the more and more demanding collaboration needs put more and more pressure on the production systems, things are still far frombeingfullyunder control • Central coordination of massive analysis efforts must prevail EUB… • Resourcescannotbestreched, onlybetterused

  3. AliEn handler – many small fixes • Fixes for staged merging • Lots of questions on the mailing listwhilemigrating to the new merging • Still not perfectbut scalingmuchbetterwithnumber of files • Typical setup: plugin->SetMergeViaJDL(kTRUE); plugin->SetMaxMergeFiles(nfiles); // customize to your output plugin->SetMaxMergeStages(nstages); // correlatewithnumber of subjobs • Extra checks for loading all neededlibraries, overwriting files in output directories • Generating on demand a file withstatistics information • mgr->AddStatisticsTask(AliVEvent::kMB) • Publishing in Monalisaeventprocessingstatistics

  4. MyAnalysis_merge(“…/Output, stage=1, chunk) MyAnalysis_merge(“…/Output, stage=2, chunk) MyAnalysis.root wn.xml Output/001/ AnalysisResults.root Output/00n/ AnalysisResults.root Output/006/ AnalysisResults.root Output/005/ AnalysisResults.root Output/004/ AnalysisResults.root Output/003/ AnalysisResults.root Output/002/ AnalysisResults.root Output/ AnalysisResults_ Stage01_002.root Output/ AnalysisResults_ Stage01_001.root Output/ AnalysisResults_ Stage01_000.root MyAnalysis.root wn.xml Output/ AnalysisResults.root plugin->SetMergeViaJDL() plugin->SetMaxMergeFiles(3) MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root 123456.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root 123456.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml MyAnalysis.root wn.xml plugin->SetRunMode(“full”) StartAnalysis(“grid”) plugin->SetRunMode(“terminate”) StartAnalysis(“grid”) plugin->SetRunMode(“terminate”) StartAnalysis(“grid”) mgr->Terminate() (per run) Running again with SetMergeViaJDL(kFALSE) will merge all runs on the client. MyAnalysis.root wn.xml

  5. New warning messages • « FIX YOUR CODE – thiswillproduce a Fatal error in future » • This looks probablyfamiliarlately, make sure youfollow the advice • Call PostData() for all output slots connected to containers of type kOutputContainer • Will prevent un-balanced output files whenhavingchunkswithoutselectedevents • Call fOutputList->SetOwner() whencreating the containers • Will preventunnecessaryleaksduringmerging • Otherdetectableerrorswillbetreated in the samemanner.

  6. Central services • Severalimprovements in the physics and centralityselection • Statisticshistogramsproduced by physicsselection made available on demand on AOD analysis • aodInputHandler->SetCheckStatistics(kTRUE); • Available via: inputHandler->GetStatistics() in FinishTaskOutput() for normalizationpurposes • Cleaned-up filtering (Laurent) + possibility to select useful AOD/MC branches only • OADB setup – seepresentation by Andreas • Multi eventhandlerused for optimizingmixing (M.Vala)

  7. Stability issues • Skipping « bad » events • I/O errors are nowcaught and the eventsskipped, isthisenough ? • Correlatedwith: corrupted files, xrootd issues, temporary SE problems • Affectingefficiency of long jobs • Some fixes or patches alreadyproposed and implemented in root (preventing the nastyeffects of unziperrors) • Investigatingdynamicreplicaswitching

  8. Scalability issues and central trains • Constrainsrelated to the proliferation of analysistasks/jobs of all qualitytrying to process all data as many times as possible • User quotas, queue policy, watchdogs • So far users have higherprioritythan production • Probably not the best approachanymore • All PWG need to provideresultsat due dates • PWG central trains were set up sothat UEB getsstreamed in a wellmonitored, controlled and scheduled effort • This shouldbe the main-stream effort for PWG groups to join • Tools: Savannah for communicating and fixing problemswithin PWG, new mailing list for train operators & central team (to be set up)

More Related