1 / 22

ATLAS Distributed Analysis: Overview

ATLAS Distributed Analysis: Overview. Distributed Analysis working group ATLAS software workshop. David Adams BNL December 8, 2004. ADA Architecture Components Datasets Transformations Services Changes Generic dataset schema Hierarchical content DIAL catalog interfaces

urania
Download Presentation

ATLAS Distributed Analysis: Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ATLAS Distributed Analysis:Overview Distributed Analysis working group ATLAS software workshop David Adams BNL December 8, 2004

  2. ADA Architecture Components Datasets Transformations Services Changes Generic dataset schema Hierarchical content DIAL catalog interfaces Goals for this release Current status Contents • Goals for the next release • Transformation interface • Conclusions ATLAS SW Wkshp ADA Overview

  3. ADA Architecture Generalized ATLAS SW Wkshp ADA Overview

  4. Components • ADA model • Data described by a dataset • Location of data, e.g. files • Content, e.g. list of event ID’s and the type of data for each event • Transformation describes an operation that can act on a dataset to produce a new dataset • Application = code shared by multiple transformations • Task = user-supplied configuration (parameters or code) • Job is an instance of a transformation acting on a dataset • User preferences may be provided • Should not affect the essential result • Typically run as a collection of sub-jobs by splitting the input dataset • Each sub-job applies the same xform its sub-dataset • Results (output datasets) must be merged • More generally the transformation might be a DAG (future) ATLAS SW Wkshp ADA Overview

  5. Components (cont) Transformation ATLAS SW Wkshp ADA Overview

  6. Datasets • Datasets enable users to examine and access data • For ATLAS data, we identify • Types of data • Used to define dataset categories • Category is part of the content specification • Types of datasets • Currently C++ classes with XML data representation • Third column indicates if this class exists • Parameter in the new dataset XML • See table on following page for ATLAS examples • There is now a single XML schema for all types of datasets ATLAS SW Wkshp ADA Overview

  7. Datasets ATLAS SW Wkshp ADA Overview

  8. Datasets (cont) Example dataset acas001> dataset_property -i 10003-20151 print AtlasPoolEventDataset 10003-20151 with no parent is locked and not empty   Content includes 1 block:     AtlasPoolEventDataset:AOD       Content ID list has 17 entries:         type MissingET with with key MET_Base         type MissingET with with key MET_Calib         type MissingET with with key MET_Truth         type ParticleBaseContainer with with key BCandidates         type ParticleBaseContainer with with key ElectronCollection         type ParticleBaseContainer with with key MuonCollection         type ParticleBaseContainer with with key ParticleJetContainer         type ParticleBaseContainer with with key PhotonCollection         type ParticleBaseContainer with with key TauJetCollection         type LVL1_ROI with with key LVL1_ROI         type VxContainer with with key VxPrimaryCandidate         type CTP_Decision with with key CTP_Decision         type INavigable4MomentumCollection with with key MuonboyTrackParticles         type INavigable4MomentumCollection with with key TrackParticleCandidate         type Rec::TrackParticleContainer with with key MooreTrackParticles         type Rec::TrackParticleContainer with with key MuidCombnoSeedTrackParticles         type Rec::TrackParticleContainer with with key MuidStandAloneTrackParticles       Event count is 1073   Location has 1 logical file:     Logical file:       Catalog: MagdaFileCatalog:Atlas       ID: AOD_3401_MultiLeptonGamma.AOD.pool.root       State: READONLY Type ID Content type Content Too many events to list Location No sub-datasets ATLAS SW Wkshp ADA Overview

  9. Transformations (cont) Now Soon • For ATLAS we identify the above transformations • Characterized by input and output dataset categories • Most common ones listed above • Others likely • Those available now are highlighted • See talks by F. Fassi and C. Haeberli ATLAS SW Wkshp ADA Overview

  10. Services • Services enable users to find and examine existing data and create new data. • Services include: • Analysis services to submit and monitor jobs • Catalog services to • Select data • Record data, metadata and transformations • Examine and record data provenance • Data management services to access the data (files) • Clients provide the user interface to these services • ROOT command line • Python command line (back soon) • GUI (based on Python) planned ATLAS SW Wkshp ADA Overview

  11. Changes • Move from DIAL 0.92 to 0.94 (almost) • Generic dataset schema (see following) • Hierarchical content (see following) • Unique ID service • Many changes to catalog interface (see following) • Transformations • Integration with production system (C. Haeberli) • Integrate analysis algorithm from the analysis tools group (F. Fassi) • Package management • Define user/application interface (G. Rybkine) • Provide reference implementation (G. Rybkine) ATLAS SW Wkshp ADA Overview

  12. Changes (cont) • Analysis services • Continued integration with GLite (D.Liko) • Begin work on prodsys analysis service (F. Brochu) • Data management • Improved understanding of SRM • Integration of gLite prototype file catalog into DQ (F. Orellana) ATLAS SW Wkshp ADA Overview

  13. Generic dataset schema • Version 0.94 of DIAL include a class GenericDataset • Means to write to and read from an XML description • All ADA datasets inherit from this without adding persistent data • Advantages • Processing system does not need to know the full dataset type • Much easier to make use of datasets outside of DIAL • Including languages other than C++, e.g. python • Other components already have generic schema • I.e., the application, task, job • Schema for the first two need work ATLAS SW Wkshp ADA Overview

  14. Hierarchical content • Each dataset description includes content: • List of event ID’s if relevant and not too large • List of type-keys describing the contained object • For each event in an event dataset • Like the type-keys in StoreGate • New release sorts this list into groups • typically one per processing stage • For ATLAS: RDO, ESD, AOD, … • Dataset can now hold both ESD and AOD with clear distinction ATLAS SW Wkshp ADA Overview

  15. DIAL catalog interface • Much work in DIAL to rationalize the interface through which users interact with catalogs • Class interface for standard catalog types • XyzRepository stores string (XML) descriptions of Xyz objects • XyzSelectionCatalog associates metadata with Xyz ID and name • XyzReplicaCatalog associates replica-logical ID’s for Xyz • Here Xyz = Dataset, Job, Application, Task, … • Generic interface for each of the above • String ID instead Xyz ID • So implementation of GenericRepository interface can be shared by DatasetRepository, JobRepository, … • Generic implementations include • File based (only for GenericRepository) • MySQL table • AMI • Web service (so far only GenericRepository) ATLAS SW Wkshp ADA Overview

  16. Goals for this release • User should be able to • Select dataset from DSC (dataset selection catalog) • Run aodhisto transformation • Input is any AOD (or other event collection) dataset • Output is a dataset containing root histograms • Makes use of the analysis tools algorithm • User can supply their own job options and analysis algorithm • Run atlasreco transformation • Input is any RDO dataset • Output is ESD dataset • Makes use of the production system transform for release 9.0.x • Monitor job status for running jobs • Get description including location of any output dataset • Easily view the histograms in a root histogram dataset ATLAS SW Wkshp ADA Overview

  17. Current status • Releases • DIAL release 0.94 is on hold until everything else needed for the release goals is in place • Dial 0.93 changes often but is now close to what 0.94 will be • Functionality • Root demos 4 and 5 have been added to illustrate use of aodhisto and atlasreco, respectively • aodhisto has only been run with one dataset at one site • atlasreco cannot use 9.0.2 and is flaky with 9.0.1 due to ATLAS SW problems • Magda is being used to catalog and move files • A few demo single-file datasets have been cataloged • See http://www.atlasgrid.bnl.gov/dialds/dlShowMain.pl ATLAS SW Wkshp ADA Overview

  18. Goals for the next release • Transformations • Clarify transformation interface (see following) • So users can add transformations • Continue development of aodhisto (F. Fassi) • Complete suite of prodsys transformations (C. Haeberli) • Catalogs • Build catalog of datasets from existing production and user data • Add transformation catalogs • Add local (to server) and global job catalogs • Provide catalog interface integrated with job submission client(s) • Analysis services • Enable ADA production with DC2 production system (F. Brochu) • Enable ADA production and analysis with gLite WMS (D. Liko) ATLAS SW Wkshp ADA Overview

  19. Goals for the next release (cont) • Data management • User anywhere can put and get data from a storage element (SE) • SE can retrieve requested data from other SE’s • Integrate DIAL with DQ and SRM (F. Orellana) • Package management • Continue development of ADA package management interface and implementations (G. Rybkine) • Integrate DIAL with ADA PM system • Deploy PM at processing sites, i.e. integrate with existing systems • AJDL • Revisit transformation specification • Integrate with GANGA and DC2 production system • Better error reporting ATLAS SW Wkshp ADA Overview

  20. Transformation interface • Clarify and document transformation interface • How xform is packaged and released • How analysis service finds xform • Runtime environment that a xform can expect • How xform is called • How xform finds input dataset and extracts it files • How transform locates software (including itself and its task) • How transform stores output files and creates output dataset • How transform indicates job status (running, failed, done, …) • How task (user code) is built and accessed • Make it easy for users to add their own transformations • E.g. run my athena algorithm • Keep task mechanism for runtime configuration ATLAS SW Wkshp ADA Overview

  21. Conclusions • Status • Much progress since last meeting but more to do • Still in demo mode • Releases • Expect DIAL 0.94 soon • When other pieces are in place • Then like to get feedback on interface and functionality • Aim for ADA/DIAL 1.0 in February • Useful system: more than demo • Meeting the short-term goals outlined earlier • Need more people • Within ADA • More attention from external providers (DQ, AMI, prodsys) • Physics contributions of data and algorithms ATLAS SW Wkshp ADA Overview

  22. More information • For more information on ADA, see the home page • http://www.usatlas.bnl.gov/ADA • Includes status of subprojects, relevant talks and documents, and links to associated projects • DIAL release 0.94 is described at • http://www.usatlas.bnl.gov/~dladams/dial/releases/0.94/index.html • To try it out, run DIAL root demos 4 and 5 in that release • Comments and questions • ADA mailing list • ADA Savannah coming soon • DIAL Savannah (with bug reporting) linked from DIAL page ATLAS SW Wkshp ADA Overview

More Related