1 / 37

K. Harrison Liverpool, 14th September 2004

K. Harrison Liverpool, 14th September 2004. GANGA DEVELOPMENT Objectives and organisation - Ganga in ATLAS and LHCb - Release status - Work in progress - Conclusions. Main objectives. Aim to help physicists in ATLAS and LHCb carry out studies in the

kaz
Download Presentation

K. Harrison Liverpool, 14th September 2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. K. Harrison Liverpool, 14th September 2004 • GANGA DEVELOPMENT • Objectives and organisation • - Ganga in ATLAS and LHCb • - Release status • - Work in progress • - Conclusions

  2. Main objectives • Aim to help physicists in ATLAS and LHCb carry out studies in the • complex computing environment of LHC  Help configure applications developed within the Gaudi/Athena framework shared by ATLAS and LHCb  Deal with submitting and monitoring jobs to distributed (Grid) and local batch systems  Help users to keep track of what they’ve done  Hide all technicalities: allow user to concentrate on the physics  Provide a single user-friendly environment for all tasks  Ganga: Gaudi/Athena and Grid Alliance • Ganga is the Hindi name for the Ganges, and is the river goddess • who carried lost souls to salvation • Ganga software will carry jobs to the Grid underworld, and hopefully • bring them back

  3. Ganga: first-level decomposition GANGA UI Internal Model MDSvc WLM ProSvc Monitor Grid Services Metadata Service WorkLoad Manager Profile Service GAUDI Program … Instr. SE CE File catalog

  4. People • Ganga is being developed as an ATLAS/LHCb common project, • with support in UK from GridPP • - Current main contributors are: •  Development: K.Harrison, J.Moscicki, A.Soroko, C.L.Tan •  Technical direction: D.L.Adams (ATLAS), U.Egede (LHCb) •  GridPP coordination: R.W.L.Jones (ATLAS), G.N.Patrick (LHCb) • Additional contributions, past and present, from many others: • N.Brook, P.Charpentier, K.Ciba, C.Collins-Tooth, F.Harris, • A.Maier, J.Martyniak, P.Mato, W.Lavrijsen, G.Rybkine, C.E.Tull •  Appologies to anyone left out

  5. Organisation • - Ganga-related information regularly updated on project web site: • http://ganga.web.cern.ch/ganga/ • Software status is tracked using Savannah: • https://savannah.cern.ch/projects/ganga/ • Have two active mailing lists, with archives accessible via web site: • General interest: project-ganga@cern.ch • Technical issues: project-ganga-developers@cern.ch • - Usually have a telephone meeting once a week •  Times, agendas and minutes placed on web site and circulated • to project-ganga@cern.chmailing list • Give status reports at ATLAS Software Workshops, at • LHCb Software Weeks, at GridPP meetings, and at various • other meetings

  6. ATLAS and LHCb: similarities and differences (1) • Two experiments use (almost) the same software framework: • Gaudi/Athena • User chooses alogrithms to run, and sets algorithm • parameters, via job options • LHCb uses text job options •  ATLAS starting to use Python job options

  7. Gaudi architecture Converter Converter Application Manager Converter Event Selector Transient Event Store Data Files Message Service Persistency Service Event Data Service JobOptions Service Algorithm Algorithm Algorithm Data Files Transient Detector Store Particle Prop. Service Persistency Service Detec. Data Service Other Services Data Files Transient Histogram Store Persistency Service Histogram Service

  8. ATLAS and LHCb: similarities and differences (2) • Main applications have similar scope: event generation, • detector simulation, reconstruction, analysis • Both experiments use Configuration Management Tool (CMT) • for code organisation, but with differences in how it’s used • Experiments have similar views on job building, but have • different requirements for implementation: • ATLAS favours use of Analysis Job Description Language (AJDL), • a key ingredient of ATLAS Distributed Analysis (ADA) system • LHCb happy with more-direct mapping to job scripts

  9. Select WorkFlow DaVinci Workflow Edit AlgorithmFlow DaVinci AlgorithmFlow DLLs Prepare AlgFlowOptions and DLLs AlgFlowOptions AlgOptions catalog Edit AlgParamOptions AlgParamOptions Metadata catalog Select Datasets DatasetOptions File catalog Sandbox DLLs JobOptions FileCatalog slice Prepare Sandbox Submit Job Job building: LHCb

  10. Job building: ATLAS

  11. AJDL in outline • AJDL (Analysis Job Description Language) provides for high-level • description of job content • - Basic elements include: • Application: name and version number • Task: user configuration of application •  Dataset: data to be processed •  Preferences: resource requirements • AJDL aims to define way in which job information is transmitted • between software components: •  Allow multiple clients to communicate with independently • implemented services by defining a common interface • AJDL represents job data using XML •  DIAL provides C++ classes that interpret and manipulate the XML

  12. Client-service communication via AJDL Initial ADA

  13. ATLAS and LHCb: similarities and differences (3) • Experiments have different metadata catalogues • ATLAS has AMI • LHCb has bookkeeping database •  Different client tools for extracting information • - Both experiments see Ganga as front-end to distributed analysis • system: •  ATLAS interested in Ganga as front-end to web services with • AJDL interface •  LHCb interested in Ganga as front-end to DIRAC Workload • Management System

  14. Ganga as front-end for ATLAS system From GANGA

  15. Ganga as front-end for LHCb system User interfaces Job monitor Production manager GANGA UI User CLI BK query webpage FileCatalog browser BookkeepingSvc FileCatalogSvc DIRAC Job Management Service DIRAC services JobMonitorSvc InfomarionSvc MonitoringSvc JobAccountingSvc AccountingDB Agent Agent Agent DIRAC resources DIRAC Storage LCG Resource Broker DIRAC Sites CE 3 DIRAC CE gridftp bbftp DIRAC CE DIRAC CE DiskFile CE 2 CE 1 rfio

  16. ATLAS and LHCb: similarities and differences (4) • Similarities between ATLAS and LHCb are motivation for • common project • Differences have led to a component-based design, capable of • satisfying a wide range of requirements • Core components of general applicability supplemented by • experiment-specific components • Approach works well for ATLAS and LHCb • Ganga has also been used successfully in BaBar (J.Martyniak)

  17. GUI CLI Gaudi/Athena Job Definition Job Definition Job Registry Gaudi/Athena Job Options Editor Software Bus Job Handling File Transfer BaBar Job Definition and Splitting Python Native PyDial Py ROOT Gaudi Python PyAMI PyCMT Ganga design: schematic

  18. JobsRegistry Job registry component 1 Job handling component 0… 1 Job 1… Application handling component 1 JobHandler 1 Application 1… 1 0… 1 ApplicationHandler Requirements File Executable 1 Configuration 0, 1 Credentials 1 Parameter FileHandler 1 Options 0… 0, 1 JobAttributes 1 Job definition component ExtraOptions Ganga core classes

  19. Application specific components JobHandler ApplicationHandler LocalJobHandler BaBarApplicationHandler GridJobHandler BatchJobHandler GaudiApplicationHandler EDGJobHandler LSFJobHandler AtlfastApplicationHandler WMSJobHandler PBSJobHandler DaVinchiApplicationHandler Job handling component Example Ganga handler classes

  20. Release history • Ganga 1.0 released in July 2003 • Basic functionality implemented • Tutorial given at US-ATLAS Computing and Physics Meeting, • BNL, 27th August 2003, with about 50 participants • Tutorial provided useful feedback to developers • Major overhaul and upgrade begun in September 2003 • Ganga 2.0 released May 2004, and followed by several minor releases • Improved GUI, revised core classes, better stability, additional • functionality • Ganga 2.3 released 13th September 2004 • - Ganga 3.0 scheduled for October 2004 • Interfaces to experiment tools and services, user-friendly • command-line interface

  21. Ganga 2.x • Ganga 2.x is a working system: download and try! • Installation instructions and user guides available from Ganga web • site: http://ganga.web.cern.ch/ganga/user/v2/QuickStart.html • Main features include: • Graphical User Interface (GUI) • Command echoing • Job-options editor (JOE) • Job submission to different batch systems (localhost, LSF, PBS) • Job submission to Grid via DIRAC, in experimental form • Example of aided application configuration, for LHCb analysis • Interface to LHCb metadata catalogue • Job catalogue • Automatic monitoring

  22. Job creation

  23. Job definition

  24. Examination of job output

  25. Interface to LHCb metadata catalogue

  26. Job-options editor: main features • Standalone Qt-based graphical editor, available from Ganga • Presents hierarchical view of available options, and helps user • with value entry • Multiple job-options files may be edited simultaneously • - Support for both text and Python job-options files • - Files saved with included files expanded, so that result is portable

  27. JOE: main window

  28. JOE: option entry

  29. Work in progress • Work in progress in several areas: • AS: Job submission to DIRAC • KH, CLT: Job submission via AJDL interface • JM: User-friendly command-line interface • CC-T, CLT: Interface to ATLAS metadata catalogue (AMI) • KC: Output merger • GR: Package-management tool • WL: Upgrade of component manager (PyBus) • - Last two aren’t strictly Ganga activities, but will benefit Ganga

  30. Job submission to DIRAC • Job handler for submission to DIRAC is already available • in Ganga 2.x, but testing is still in progress • Handler creates job-description files required by DIRAC • Job submission and monitoring performed making direct use of • DIRAC API • Output files retrieved automatically when ready, in separate thread • Log file is checked for error messages, then job state is updated

  31. Job submission via AJDL interface • Python client for AJDL services developed by producing bindings • for DIAL C++ classes • Generate LCG dictionaries using lcgdict • Compile resulting _dict.cpp files against ReflectionBuilder to • create libraries that can be imported into Python using PyLCGDict2 • Procedure now largely automated • Have added set of Python functions to simplify the use of some of • the classes • For example, can submit job to remove DIAL server • (GSI authentication) and query status with: • import pydial • pydial.submit(“dataset.xml”,”task.xml”,”application.xml”) • pydial.showStatus()

  32. Prototype GUI for AJDL job builder  Working on appropriate handlers for connection to Ganga

  33. Interface to ATLAS metadata catalogue • Python client for ATLAS metadata catalogue (AMI) developed • Allows use of main AMI methods for querying and updating catalogue • Checks for mandatory parameters are made before a command • is sent to AMI • XML results returned by AMI are automatically converted to • appropriate Python object

  34. Prototype GUI for Ganga interface to AMI - Currently very simple - Allows user to select AMI project / processing step - User builds query - User selects LFNs and Ganga adds them to a job script

  35. Output merger • Output merger is being developed as a standalone application, • Josie (more a companion for JOE than an acronym), which will • be available from Ganga • Core classes have been implemented • Currently, can produce macros for merging ROOT ntuples and • histograms, concatenate text files, and perform text filtering • - Functionality available via command line and GUI

  36. User-friendly command-line interface • Ganga echoes commands executed when operations are performed • through the GUI • Echoed commands can be entered directly at Python prompt, but • can be cumbersome • Draft scheme for more-intuitive command-line interface has • been written, and a prototype implementation is available

  37. Conclusions - Differences between ATLAS and LHCb create challenges for Ganga, but these are being met - Significant Ganga-related activity in both experiments GridPP developers have key roles, but also others are making major contributions - Ganga release 2.x is a working system Installation instructions and user guide available from Ganga web site - Ganga 3.0, with submission to DIRAC and via AJDL interface, scheduled for October - LHCb-UK software course to be held at Cambridge in December Ganga will be used as tool for submitting jobs

More Related