1 / 12

The LCG File Catalog (LFC)

The LCG File Catalog (LFC). Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005. LCG File Catalog. Based on lessons learned in DC’s (2004) Fixes performance and scalability problems seen in EDG Catalogs Cursors for large queries Timeouts and retries from the client

pekelo
Download Presentation

The LCG File Catalog (LFC)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005

  2. LCG File Catalog • Based on lessons learned in DC’s (2004) • Fixes performance and scalability problems seen in EDG Catalogs • Cursors for large queries • Timeouts and retries from the client • Provides more features than the EDG Catalogs • User exposed transaction API • Hierarchical namespace and namespace operations • Integrated GSI Authentication + Authorization • Access Control Lists (Unix Permissions and POSIX ACLs) • Checksums • Based on existing code base • Supports Oracle and MySQL database backends CERN IT-GD

  3. User Metadata Replica Replica Replica Replica User Defined Metadata srm://host.example.com/foo/bar host.example.com srm://host.example.com/foo/bar host.example.com srm://host.example.com/foo/bar host.example.com srm://host.example.com/foo/bar host.example.com LFN Symlink Symlink GUID Symlink /grid/dteam/dir1/dir2/file1.root /grid/dteam/mydir/mylink /grid/dteam/mydir/mylink /grid/dteam/mydir/mylink Xxxxxx-xxxx-xxx-xxx- System Metadata “size” => 10234 “cksum_type” => “MD5” “cksum” => “yy-yy-yy” Relationships in the Catalog CERN IT-GD

  4. Features • Namespace operations • All names are in a hierarchical namespace • mkdir(), opendir(), etc… • Also chdir() • GUID attached to every directory and file • Security – GSI Authentication and Authorization • Mapping done from Client DN to uid/gid pair • Authorization done in terms of uid/gid • VOMS will be integrated (collaboration with INFN/NIKHEF) • VOMS roles appear as a list of gids • Ownership of files is stored in catalog • Permissions implemented • Unix (user, group, all) permissions • POSIX ACLs (group and users) CERN IT-GD

  5. LFC interfaces LFC SERVER LCG UTIL GFAL Python LFC CLIENT C API DLI WMS CLI lfc-ls, lfc-mkdir, lfc-setacl, … CERN IT-GD

  6. LFC Tests • LFC has been tested and shown to be scalable to at least: • 40 million entries • 100 client threads • Performance improved with comparison to RLSs • Stable : • Continuous running at high load for extended periods of time with no crashes • Based on code which has been in production for > 4 years • Tuning required to improve bulk performance CERN IT-GD

  7. FiReMan Performance - Insert • Comparison with LFC: 350 Fireman - Single Entry Fireman - Bulk 100 300 LFC 250 200 Inserts / Second 150 100 50 0 1 2 5 10 20 50 100 Number of Threads CERN IT-GD

  8. FiReMan Performance - Queries • Comparison with LFC: 1200 Fireman - Single Entry Fireman - Bulk 100 LFC 1000 800 Entries Returned / Second 600 400 200 0 1 2 5 10 20 50 100 Number Of Threads CERN IT-GD

  9. Tests Conclusion • Both LFC and FiReMan offer large improvements over RLS • Still some issues remaining: • Scalability of FiReMan • Bulk Operations for LFC • More work needed to understand performance and bottlenecks • Need to test some real Use Cases CERN IT-GD

  10. LFC Deployment • Catalog called “LCG File Catalog”, but not HEP specific ! • LFC is Open Source • Secure LFC is part of LCG-2_4_0 release • Easy installation/configuration : RPMs or tarballs • YAIM components provided (for MySQL only at the moment) • LFC Administrator’s Guide http://goc.grid.sinica.edu.tw/gocwiki/How_to_set_up_an_LFC_service • In pre-production at : CERN/DESY/HG-01-GRNET/LIP/Bari/Pisa • Script for automatic migration from RLS to LFC is provided : http://goc.grid.sinica.edu.tw/gocwiki/How_to_migrate_the_RLS_entries_into_the_LCG_File_Catalog_%28LFC%29 CERN IT-GD

  11. LFC usage example CERN IT-GD

  12. LFC future developments • Bulk operations • Sessions • Integration with the ARDA Metadata Catalog • Integration with VOMS/LCAS/LCMAPS • Integration with AUTHZ • Contacts : • Jean-Philippe.Baud@cern.ch • Sophie.Lemaitre@cern.ch CERN IT-GD

More Related