Storage Resource Broker Tutorial

1. Storage Resource Broker Tutorial Reagan W. Moore Wayne Schroeder Arcot Rajasekar George Kremenek Lucas Gilbert Arun Jagatheesan San Diego Supercomputer Center {moore, schroeder, sekar, kremenek, iktome, arun} @ sdsc.edu

2. SDSC SRB Team • Reagan Moore • Michael Wan • Arcot Rajasekar • Wayne Schroeder • Arun Jagatheesan • Charlie Cowart • Lucas Gilbert • George Kremenek • Sheau-Yen Chen • Bing Zhu • Roman Olschanowsky • Vicky Rowley (BIRN) • Marcio Faerman (SCEC) • Antoine De Torcy (IN2P3) • Students & emeritus • Erik Vandekieft • Reena Mathew • Xi (Cynthia) Sheng • Allen Ding • Grace Lin • Qiao Xin • Daniel Moore • Ethan Chen • Jon Weinburg

3. SRB Homepage: http://www.npaci.edu/DICE/SRB/ inQ Homepage http://www.npaci.edu/dice/srb/inQ/inQ.html mySRB URL https://srb.npaci.edu/mySRB2v7.shtml Grid Port Toolkit https://gridport.npaci.edu/ SRB Chat srb-chat@sdsc.edu SRB bug list http://www.npaci.edu/dice/srb/bugs.html SRB Info Resources

4. SRB overview Concepts behind data management Production data grid examples Integration of data grids with digital libraries and persistent archives SRB installation Java administration tool SRB APIs Shell commands Java Matrix dataflow Topics

5. A distributed file system (Data Grid), based on a client-server architecture. It’s also more: It provides a way to access files and computers based on their attributes rather than just their names or physical locations. It replicates, syncs, archives, and connects heterogeneous resources in a logical manner using abstraction mechanisms. Introduction to SRB

6. Data collecting Sensor systems, object ring buffers and portals Data organization Collections, manage data context Data sharing Data grids, manage heterogeneity Data publication Digital libraries, support discovery Data preservation Persistent archives, manage technology evolution Data analysis Processing pipelines, manage knowledge extraction Information Technologies

7. Collection The organization of digital entities to simplify management and access. Context The information that describes the digital entities in a collection. Content The digital entities in a collection Data Management Concepts(Elements)

8. Curation The process of creating the context Closure Assertion that the collection has global properties, including completeness and homogeneity under specified operations Consistency Assertion that the context represents the content Data Management Concepts(Mechanisms)

9. Distributed data sources Management across administrative domains Heterogeneity Multiple types of storage repositories Scalability Support for billions of digital entities, PBs of data Preservation Management of technology evolution Data Management Challenges

10. Distributed data sources Inter-realm authentication and authorization Heterogeneity Storage repository abstraction Scalability Differentiation between context and content management Preservation Support for automated processing (migration, archival processes) Data Grids

11. Data Grids provide the underlying abstractions required to support Digital libraries Curation processes Distributed collections Discovery and presentation services Persistent archives Management of technology evolution Preservation of authenticity Assertion

12. Descriptive Provenance information, discovery attributes Administrative Location, ownership, size, time stamps Structural Data model, internal components Behavioral Display and manipulation operations Authenticity Audit trails, checksums, access controls Types of Context Metadata

13. SRB Collections at SDSC

14. Digital libraries and persistent archives can be built on data grids Common capabilities are needed for each environment Multiple examples of production systems across scientific disciplines and federal agencies Common Infrastructure

15. Federated client-server architecture Servers can talk to each other independently of the client Infrastructure independent naming Logical names for users, resources, files, applications Collective ownership of data Collection-owned data, with infrastructure independent access control lists Context management Record state information in a metadata catalog from data grid services such as replication Abstractions for dealing with heterogeneity Data Grid Components

16. Logical name space for files Global persistent identifier Storage repository abstraction Standard operations supported on storage systems Information repository abstraction Standard operations to manage collections in databases Access abstraction Standard interface to support alternate APIs Latency management mechanisms Aggregation, parallel I/O, replication, caching Security interoperability GSSAPI, inter-realm authentication, collection-based authorization Data Grid Abstractions

17. C, C++, Libraries Unix Shell Databases DB2, Oracle, Postgres Archives HPSS, ADSM, UniTree, DMF File Systems Unix, NT, Mac OSX SDSC Storage Resource Broker & Meta-data Catalog Application Linux I/O OAI WSDL Access APIs DLL / Python Java, NT Browsers GridFTP Federation Consistency Management / Authorization-Authentication SRB Server Logical Name Space Latency Management Data Transport Metadata Transport Catalog Abstraction Storage Abstraction Databases DB2, Oracle, Sybase, SQLServer Drivers HRM

18. SDSC Storage Resource Broker Federated client-server system, managing Over 90 TBs of data at SDSC Over 16 million files Manages data collections stored in Archives (HPSS, UniTree, ADSM, DMF) Hierarchical Resource Managers Tapes, tape robots File systems (Unix, Linux, Mac OS X, Windows) FTP sites Databases (Oracle, DB2, Postgres, SQLserver, Sybase, Informix) Virtual Object Ring Buffers Production Data Grid

19. Federated SRB server model Peer-to-peer Brokering Read Application Parallel Data Access Logical Name Or Attribute Condition 1 6 5/6 SRB server SRB server 3 4 5 SRB agent SRB agent 2 Server(s) Spawning R1 MCAT 1.Logical-to-Physical mapping 2.Identification of Replicas 3.Access & Audit Control R2 Data Access

20. Global, location-independent identifiers for digital entities Organized as collection hierarchy Attributes mapped to logical name space Attributed managed in a database Types of administrative metadata Physical location of file Owner, size, creation time, update time Access controls Logical Name Space

21. Logical file name Infrastructure independent Used to organize files into a collection hierarchy Globally unique identifier GUID for asserting equivalence across collections Descriptive metadata Support discovery Physical file name Location of file File Identifiers

22. Define logical resource name List of physical resources Replication Write to logical resource completes when all physical resources have a copy Load balancing Write to a logical resource completes when copy exist on next physical resource in the list Fault tolerance Write to a logical resource completes when copies exist on “k” of “n” physical resources Mappings on Name Space

23. Integrate data management system, data processing system, and data storage system into a modular unit Commodity based disk systems (1 TB) Memory (1 GB) CPU (1.7 Ghz) Network connection (Gig-E) Linux operating system Data Grid technology to manage name spaces User names (authentication, authorization) File names Collection hierarchy Grid Bricks

24. Hardware components Intel Celeron 1.7 GHz CPU SuperMicro P4SGA PCI Local bus ATX mainboard 1 GB memory (266 MHz DDR DRAM) 3Ware Escalade 7500-12 port PCI bus IDE RAID 10 Western Digital Caviar 200-GB IDE disk drives 3Com Etherlink 3C996B-T PCI bus 1000Base-T Redstone RMC-4F2-7 4U ten bay ATX chassis Linux operating system Cost is $2,200 per Tbyte plus tax Gig-E network switch costs $500 per brick Effective cost is about $2,700 per TByte Data Grid Brick

25. Used to implement “picking” environments for 10-TB collections Web-based access Web services (WSDL/SOAP) for data subsetting Implemented 15-TBs of storage Astronomy sky surveys, NARA prototype persistent archive, NSDL web crawls Must still apply Linux security patches to each Grid Brick Grid Bricks at SDSC

26. Logical name space for files Common collection hierarchy across modules Collection owned data Data grid manages access control lists for files Data grid manages authentication Logical resource name Aggregate modules under a single logical name Support load leveling across modules Grid Brick Logical Names

27. Bulk register Create a logical name for a file Load context (metadata) Bulk load Create a copy of the file on a data grid storage repository Bulk unload Provide containers to hold small files and pointers to each file location Requests for bulk operations for delete, access control, … Latency Management -Bulk Operations

28. SRB Latency Management Remote Proxies, Staging Data Aggregation Containers Prefetch Network Destination Network Destination Source Caching Client-initiated I/O Streaming Parallel I/O Replication Server-initiated I/O

29. 2MASS (2 Micron All Sky Survey): Bruce Berriman, IPAC, Caltech; John Good, IPAC, Caltech, Wen-Piao Lee, IPAC, Caltech NVO (National Virtual Observatory): Tom Prince, Caltech, Roy Williams CACR, Caltech, John Good, IPAC, Caltech SDSC – SRB : Arcot Rajasekar, Mike Wan, George Kremenek, Reagan Moore Latency ManagementExample - Digital Sky Project

30. Digital Sky Data Ingestion SRB SUN E10K Data Cache star catalog Informix SUN HPSS 800 GB …. input tapes from telescopes 10 TB SDSC IPAC CALTECH

31. http://www.ipac.caltech.edu/2mass The input data was originally written to DLT tapes in the order seen by the telescope 10 TBytes of data, 5 million files Ingestion took nearly 1.5 years - manual loading of tapes Images aggregated into 147,000 containers by SRB Digital Sky - 2MASS

32. Extract image cutout from Digital Palomar Sky Survey Image size 1 Gbyte Shipped image to server for extracting cutout took 2-4 minutes (5-10 Mbytes/sec) Remote proxy performed cutout directly on storage repository Extracted cutout by partial file reads Image cutouts returned in 1-2 seconds Remote proxies are a mechanism to aggregate I/O commands Remote Proxies

33. SRB Installation Wayne Schroeder San Diego Supercomputer Center Schroede@sdsc.edu

34. Installation procedure written by Michael Doherty SRB_Install_Notes.doc Perl install script for Mac OS X and Linux written by Wayne Schroeder http://www.npaci.edu/DICE/SRB/tarfiles/install.pl Installs PostgreSQL, MCAT, SRB server, SRB clients Installation takes 18 minutes on a Mac G4 SRB Installation

35. SRB Federation Wayne Schroeder Arcot Rajasekar Charlie Cowart Reagan Moore San Diego Supercomputer Center {schroede, sekar, charliec, moore}@sdsc.edu

36. Data grids provide the ability to name, organize, and manage data on distributed storage resources Federation provides a way to name, organize, and manage data on multiple data grids. Data Grid Federation

37. Consistency constraints in federations Cross-register a digital entity from one collection into another Who manages the access control lists? Who maintains consistency between context and content? How can federation systems be characterized? Peer-to-Peer Federated Systems

38. Each SRB zone uses a metadata catalog (MCAT) to manage the context associated with digital content Context includes: Administrative, descriptive, authenticity attributes Users Resources Applications SRB Zones

39. Mechanisms to impose consistency and access constraints on: Resources Controls on which zones may use a resource User names (user-name / domain / SRB-zone) Users may be registered into another domain, but retain their home zone, similar to Shibboleth Data files Controls on who specifies replication of data Context metadata Controls on who manages updates to metadata SRB Peer-to-Peer Federation

40. Occasional Interchange - for specified users Replicated Catalogs- entire state information replication Resource Interaction- data replication Replicated Data Zones- no user interactions between zones Master-Slave Zones- slaves replicate data from master zone Snow-Flake Zones- hierarchy of data replication zones User / Data Replica Zones- user access from remote to home zone Nomadic Zones “SRB in a Box”- synchronize local zone to parent zone Free-floating “myZone” - synchronize without a parent zone Archival “BackUp Zone”- synchronize to an archive SRB Version 3.0.1 released December 19, 2003 Peer-to-Peer Federation

41. Principle peer-to-peer federation approaches (1536 possible combinations)

42. Federation Approaches Peer-to-Peer Zones Free Floating Partial User-ID Sharing Occasional Interchange Partial Resource Sharing Replicated Data Hierarchical Zone Organization One Shared User-ID System Set Access Controls System Controlled Complete Synch Complete User-ID Sharing No Metadata Synch Nomadic Resource Interaction User and Data Replica System Managed Replication System Set Access Controls System Controlled Partial Metadata Synch No Resource Sharing System Managed Replication Connection From Any Zone Complete Resource Sharing Replicated Catalog Snow Flake Replication Zones Super Administrator Zone Control Master Slave System Controlled Complete Metadata Synch Complete User-ID Sharing Archival Hierarchical Zones

43. Demo Federations

44. SRB APIs George Kremenek Reagan Moore San Diego Supercomputer Center {kremenek, moore} @sdsc.edu

45. C library calls Provide access to all SRB functions Shell commands Provide access to all SRB functions mySRB web browser Provides hierarchical collection view inQ Windows browser Provides Windows style directory view Java Supports grid portals, WSDL interface SRB APIs

46. Command line access to the SRB Login to machine with Scommand binaries Verify/Create ~/.srb/.MdasEnv Verify/Create ~/.srb/.MdasAuth Scommands

47. mdasCollectionHome '/home/kremenek.npaci’ Logical path name for collection mdasDomainHome ‘npaci' srbUser 'kremenek’ The combination DomainHome/srbUser defines a user srbHost ’srb.sdsc.edu‘ Location of MCAT catalog srbPort ‘5615’ Port for accessing MCAT catalog The combination srbHost/srbPort defines the catalog ~/.srb/.MdasEnv file

48. valid authorization schemes are : 'PASSWD_AUTH', ‘ENCRYPT1','GSI_AUTH’ ENCRYPT1 is a challenge/response mechanism GSI-AUTH is Grid certificate mechanism defaultResource 'dl1-unix-sdsc’ Default location for storage repository File ~/.srb/.MdasAuth contains the SRB password .MdasEnv, .MdasAuth

49. Command line interface SCRIPTING BATCH and Workflow upload/download Flexibility Power Complexity Installed man pages via “man [Scommand]” Scommand Features

50. Shelp Gives list of commands with brief summary “[Scommand] <return>” gives usage info (usually) Sinit – establishes connection Senv – displays connection information Sexit – ends connection Scommand Features (cont’d)