1 / 10

ByteStream Components

ByteStream Components. Jack Cranshaw February 20, 2012. ATLAS Code Base. TDAQ releases Contain the basic structure of blocks in the ByteStream format. Sets the content. Currently on tdaq-common-01-09-00

dahlia
Download Presentation

ByteStream Components

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ByteStreamComponents Jack Cranshaw February 20, 2012

  2. ATLAS Code Base • TDAQ releases • Contain the basic structure of blocks in the ByteStream format. Sets the content. • Currently on tdaq-common-01-09-00 • Releases occur roughly every 6 months and must be approved by trigger group and online. • Attached to the ATLAS Offline code in external package DataCollection. I/O and Persistence Evolution Workshop

  3. ATLAS Code Base • Event/ByteStreamData • Classes for data storage (ROB, RawEvent, metadata) • Event/ByteStreamCnvSvcBase • Utility and base classes used by detector Cnv, address providers and data providers • Converters in detector specific packages like for normal T/P. • Event/ByteStreamCnvSvc • Input and output services. • Utility programs such as AtlCopyBSEvent, AtlListBSEvent. • Event/ByteStreamStoragePlugins • Plugins to map different storage technologies onto the same basic converters: castor, xrootd, dcache, gfal. I/O and Persistence Evolution Workshop

  4. Dependencies Control/DataModel Event/ByteSreamData External/DataCollection Database/AthenaPOOL External/AtlasROOT Event/ByteStreamCnvSvcBase Event/ByteStreamStoragePlugins Event/ByteStreamCnvSvc I/O and Persistence Evolution Workshop

  5. ByteStream Input in Athena • Handled by changing EventSelector used by serviceMgr. • Mostly handled by job options in ByteStreamCnvSvc/ReadByteStream.py. User just sets file name to EventSelector and ByteStreamInputSvc. • ReadByteStream.py • Adds ByteStreamEventStorageInputSvc, ROBDataProviderSvc, EventSelectorByteStream, ByteStreamCnvSvc, ByteStreamAddressProviderSvc, and ProxyProviderSvc to svcMgr. • Sets EventSelector property ByteStreamInputSvc. • Sets EventPersistencySvc property CnvServices. • TAG input setup different and handled directly in RecExCommon_topOptions. I/O and Persistence Evolution Workshop

  6. ByteStream Output in Athena • Somewhat more sophisticated python contained in ByteStreamCnvSvc/WriteByteStream.py. • WriteByteStream.py • getStream method returns an AthenaOutputStream. • Two types of output: EVENTSTORAGE, TRANSIENT. • EVENTSTORAGE (write out) • Adds ByteStreamEventStorageOutputSvc to svcMgr. • Sets some properties by parsing output name. • TRANSIENT (multi-stage) • Adds ROBDataProviderSvc and RDP_OutputSvc to svcMgr. • Sets EventPersistencySvc property CnvServices since it is faking input. I/O and Persistence Evolution Workshop

  7. Comparison to POOL/APR/ROOT • BS Input • ByteStreamCnvSvc. ReadByteStream • BS Output • ByteStreamCnvSvc.WriteByteStream • Use C++ class AthenaOutputStream from AthenaServices • Only handling event data. • Itemlist handled in RecExCommon_topOptions • POOL Input • AthenaPoolCnvSvc. ReadAthenaPool • POOL Output • OutputStreamAthenaPool. MutipleStreamManager • Use python class AugmentedStreamBase • Integrated stream handling event data and metadata. • Itemlist handled in files in RecExPers package. Differences on output complicate maintenance I/O and Persistence Evolution Workshop

  8. Job Options and Packages • As noted in the POOL-ByteStream comparison there are differences, primarily on output. • There are also a variety of hacks (for most output streams) in RecExCommon_topOptions which could benefit from cleanup. • Some pieces of code which seem general are placed in packages which are meant to be technology specific. I/O and Persistence Evolution Workshop

  9. Testing • ByteStreamCnvSvc • Tests of utilities AtlCopyBSEvent, AtlListBSEvent • AthenaPoolExampleAlgorithms • Tests of input and output using ReadByteStream and WriteByteStream. • Also tests I/O with TAG as input. • RecExAnaTest • Tests of usage in RecExCommon using Reco_trf. I/O and Persistence Evolution Workshop

  10. Summary • Concerns • There are a variety of hacks which need to be cleaned up if possible. • Some of the processes, such as parsing filenames seem brittle. • The differences in output methodologies used in RecExCommon complicate maintenance. • The package structure (or placement of code) is not logically consistent everywhere. • The treatment of metadata is becoming more complex. • The system is currently working other than a few metadata issues. I/O and Persistence Evolution Workshop

More Related