1 / 11

Data Distribution

Data Distribution. Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26 th September 2000. KanGA Exports Current exports Developments Obectivity Data Exports Objectivity SP Imports Objectivity Data Distribution Tools Plans Parallel Session on Thursday morning.

jeri
Download Presentation

Data Distribution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26th September 2000 Tim Adye

  2. KanGA Exports • Current exports • Developments • Obectivity Data Exports • Objectivity SP Imports • Objectivity Data Distribution Tools • Plans • Parallel Session on Thursday morning... Tim Adye

  3. KanGA Exports • Currently 1.8 TB on disk at SLAC • 8 sites take everything, 6 a subset [CCG Survey] • European sites copy from each other to reduce transatlantic traffic • Sites can elect to just take the skims of local interest • Data copied using rsync/syncslac mirroring tools • Easy to set up and use • Various improvements in tools • But does not scale so well • Takes ~3 hours just to scan disk for updates (130k files) • Difficult to specify subset of data to copy (uses file path) Tim Adye

  4. KanGA Tools • Want to control export using using a database • Use existing collection catalogue (skimData) • Catalogue mirrored at remote sites Don’t have to log onto SLAC to use skimData  Generate TCL files just for files already local • Outside SLAC, Use mySQL, so don’t have to buy/install Oracle • Which skims/runs/processings to copy selected using skimData Tim Adye

  5. KanGA Export Tools Status • skimData catalogue mirrored SLAC->RAL->UK Universities (under test) [Alessandra Forti] • Export control being commissioned [AF/TA] • A similar approach is in use at CASPUR • Includes local backup [Alvise Dorigo, Emanuele Leonardi] • Investigate alternative file copy tools [TA et al] • rsync vs. multi-threaded tools: bbftp (IN2P3), sfcp (SLAC), gsiftp (GLOBUS) Tim Adye

  6. Objectivity Export • All micro data exported to CC-IN2P3 [A-M Lutz] • Rome and RAL now take just KanGA • All exports go via the network • bbftp to optimise bulk data transport [Gilles Farrache] • Various optimisations implemented • Data/conditions export now integrated into OPR sweeps [Adil Hasan, Artem Trunov] • Skim exports still come from analboot2 (more tricky) • Fewer local file copies on export (SLAC) and import (CC-IN2P3) [J-N Albert, Dominique Boutigny] Tim Adye

  7. SP Imports to SLAC • MC Imports being automated • Production Centres responsible for export and copying to SLAC • Automatic procedure being developed [Daniele Andreotti, Emanuele Leonardi] • Automatic procedure for SLAC import [Cristina Bulfon et al] • Starts when files arrive on disk • Check files, import to SP3 federation, update bfreport • See Jim Smith’s talk… Tim Adye

  8. Objectivity Data Distribution Tools(Recent improvements -- only a very brief summary) • New “collection metadata” information [Moreno Marzolla] • Stores collection<->DB file correspondences • Bypass slow scan to find which files to export • Error checking and reporting improvements [TA] • Objectivity Catalogue browser [Yemi Adesanya] • Java GUI and “batch mode” selection of DB files to export Tim Adye

  9. Objectivity Data Distribution Tools(Current work -- only a very brief summary) • Single event / run / skim export [MM] • Creates and exports “All-In-One” database files, with just the requested data • Can be used for raw/reco/micro export of small samples of events • Redesign of Data Distribution Tools in Java[JNA] • Uses existing (C++) Objectivity access code • Optimised for specific requirements • Eg. Bulk import over the net (CC-IN2P3) Tim Adye

  10. Plans • Wait for Computing Model report (tomorrow ) • Whichever way we go, we need to continue to support • Objectivity exports (SP, access to samples of reco) • KanGA exports (at least until Bdb/Objectivity meets requirements) • Automatic bulk exports via the net • Exports of small samples for individual users Tim Adye

  11. Parallel Session • Data Distribution / Simulation Production meeting on Thursday morning • 8:30, Orange Room • See HN for agenda and phone-in details • datadist / simuprod HN groups Tim Adye

More Related