1 / 7

Data Management Overview and Site Configurations in DE Cloud

An in-depth look at the data management issues faced in the DE cloud, including site configurations, dark data, xrootd, WebDAV services, and Squid status. Updates on efficiency, throughput, and recent errors with relevant solutions and proposals.

barnes
Download Presentation

Data Management Overview and Site Configurations in DE Cloud

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. General Overview • Functional tests failing due to low res CRL on CERN FTS (fixed) • Rqueued errors due to high load at FZK, DESY-ZN (recently disappeared) • Chksum errors due to duplicated file production in analysis (fixed in new rucio?) • Missing production files (Freiburg), production subscription issue (few) • DE as destination • Total efficiency: 94% (DE 94%) • 939 MB/s throughput • Sites with lower efficiency: FMPHi (56%), IEPSAS (35%), BONN (78%) • DE as source • Total efficiency: 95% (DE 94%) • 717 MB/s throughput • Sites with lower efficiency: + GOEGRID (61%), PSNC (50%) Andrzej Olszewski

  2. General Overview DDM downtimes Andrzej Olszewski

  3. DDM Site Configuration • ATLASGROUPDISK migration and removal • Done at all sites • ATLASHOTDISK: T2s asked to clean space and assign to ATLASDATADISK • Done at all sites • ATLASPRODDISK cleanup after migration to Rucio naming convention • PRODDISK: T2s are asked to remove files and subdirectories of .../atlasproddisk/ and keep only those in .../atlasproddisk/rucio/ and .../atlasproddisk/SAM/ • Remaining: GOEGRID, PSNC Andrzej Olszewski

  4. Dark Data in DE Cloud • Sites with considerable dark data • Just from looking at burricot accounting plots (to be confirmed) • DATADISK: CSCS, PSNC • LOCALGROUPDISK: LRZ, BONN • PRODDISK: GOEGRID (not cleaned), DESY-HH • SCRATCHDISK: most sites • Cleaning • Usually requires consistency checks (srm vs Atlas catalogs) • Currently official scripts not working (incompatiblilities) • Asked Kai to provide compatible srm dump script • Cleaning on SCRATCHDISK • Proposal to clean all files with mod time older than 30 days • Yesterday confirmed by central ddm ops team • FZK, LRZ going to implement this soon • What about other sites? Andrzej Olszewski

  5. XROOTD • FAX: certification phase 1 • https://twiki.cern.ch/twiki/bin/view/AtlasComputing/FAXSiteCertificationP1 • Monitoring testing Rucio file access mode • Sites working fine • CYFRONET, DESY-HH, DESY-ZN, FZK, LRZ, MPPMU, PSNC, FREIBURG • Sites installed but failing tests • GOEGRID, PRAGUE, WUPPERTAL (New N2N needed) • Sites that not installed yet • CSCS, FMPH, IEPSAS, HEPHY Andrzej Olszewski

  6. WebDAV • WebDAV services needed at sites for renaming files to Rucio path convention • ATLAS specific instructions for sites • https://twiki.cern.ch/twiki/bin/viewauth/Atlas/WebDAVSetupForATLAS • Rucio renaming status • Deadline: February 1st • All sites renamed • BONN: manual renaming finishing • WUPPERTAL renaming on LOCALGROUPDISK running Andrzej Olszewski

  7. SQUID • SQUID status • CYFRONET, no backup to Prague (to be fixed soon) • DRESDEN, no backup to CYFRONET • DORTMUND, no squid (WUPPERTAL), no backup (FREIBURG) • GOEGRID, no backup to DESY • HEPHY, no backup to CSCS • PSNC, no squid, only backup to Prague • No tests: LRZ, BONN, MAINZGRID • Red means site goes directly to Frontier and cloud gets a ticket • Solution • Proposal to deploy central Squid backup service at FZK • This would make it easier to keep sites’ access to backup Squid up to date • Avoids negotiations between sites, loss of access after host site reconfiguration Andrzej Olszewski

More Related