1 / 26

An Introduction to the

An Introduction to the. By: Roman Olschanowsky roman2u@sdsc.edu. Outline. SDSC and History of SRB Example Project Introduction to SRB Discussion on SRB basics SRB Clients Overview of a Data Grid Infrastructure Topology Teragrid Demo How to use your TG SRB account

dugan
Download Presentation

An Introduction to the

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Introduction to the By: Roman Olschanowsky roman2u@sdsc.edu

  2. Outline • SDSC and History of SRB • Example Project • Introduction to SRB • Discussion on SRB basics • SRB Clients • Overview of a Data Grid • Infrastructure • Topology • Teragrid Demo • How to use your TG SRB account • How to access Digital Data Collections

  3. TeraGrid Linux Cluster (IA64) 4.4 TF Sun F15K Disk Server Storage Area Network Disk 1.4 PB www.sdsc.edu Human infrastructure:Experienced multi-disciplinary staff support a broad spectrum of national science, engineeringand technology projects Archival Systems Blue Gene/L (Due 12/04) DataStar IBM Power4 18 PB Networking Visualization Storage and Compute Resources 2.8/5.7 TF 15.6 TF

  4. Sites Using the SRB

  5. SDSC SRB Projects (60 million, .5 PB ) • Digital Libraries • UCB, Umich, UCSB, Stanford,CDL • NSF NSDL - UCAR / DLESE • NASA Information Power Grid • Astronomy • National Virtual Observatory • 2MASS Project (2 Micron All Sky Survey) • Particle Physics • Particle Physics Data Grid (DOE) • GriPhyN • SLAC Synchrotron Data Repository • Medicine • Digital Embryo (NLM) • Earth Systems Sciences • ESIPS • LTER • Persistent Archives • NARA • LOC • Neuro Science & Molecular Science • TeleScience/NCMIR, BIRN • SLAC, AfCS, …

  6. Storage Resource Broker (SRB) • A distributed file system (Data Grid) • Client-Server, Server-Server architecture. • Abstracts physical • SRB provides the ability to transparently share data across remote sites. • Heterogeneous Resources • Single sign on • Single logical file hierarchy

  7. What we are familiar with

  8. What we are not familiar with, yet

  9. How do the file systems differ? • Logical Abstraction • Folders are NOT physical • Files do NOT inherit physical location • Everything is potentially distributed • Access Control • Permissions are NOT rwxrwxrwx • Permissions ARE on a object by object basis • Groups and permissions ARE more similar to NTFS • Domains • Geographical / logical grouping of users • Namespace scalability: john@harvard john@mit • Also doubles as groups

  10. Interfaces to theStorage Resource Broker • inQ – Windows Client • Scommands – UNIX, DOS Command line Client • Jargon – Java API and GUI components • mySRB – Web Client • Matrix – WSDL, Data Grid Workflows • C, C++ – C and C++ API • Python – Python API • Perl – Perl API

  11. Sinit Senv Spwd Sls Scd Sget Sput Ssh Scp Smv (logical) Sphymove (physical) Srm Smkdir Srmdir Serror Schmod Sexit Common Scommands (75 total)

  12. mySRB

  13. BIRN Portal (perl based)

  14. NEEScentral Portal (php based)

  15. Biomedical Informatics Research Network (BIRN) • Major collaboration with SDSC, several of the projects’ Co-Investigators and Co-PIs are at SDSC. • BIRN’s purpose is to provide it’s consortium of neuroscience laboratories the ability to share, compute, and collaborate. • The Storage Resource Broker provides the ability to transparently share data across remote sites.

  16. The BIRN SRB Data Grid

  17. Doing this “Manually”

  18. The BIRN Data Grid

  19. The grid is in the details

  20. File Replication • Sls • /home/Demo/SRB-Tutorial/files-2: • Doc.txt • Sls -l • /home/Demo/SRB-Tutorial/files-2: • romanoly 0 z-ucsd-ncmir-nas1 15 2003-07-09-05.15 Doc.txt • romanoly 1 z-jhu-cis-nas0 15 2003-07-09-05.16 Doc.txt • romanoly 2 z-stanford-lucas-nas 15 2003-07-09-05.16 Doc.txt • romanoly 3 z-umn-cmrr-nas0 15 2003-07-09-05.16 Doc.txt • romanoly 4 z-uci-bic-nas0 15 2003-07-09-05.17 Doc.txt

  21. Teragrid SRB • All Teragrid accounts are given a SDSC SRB Teragrid account • The ‘username’ is the same as your SDSC UNIX account name • Your SRB ‘domain’ is ‘teragrid’ • You must register your DN string with SDSC’s grid-mapfile or request a SRB password to activate your SRB account. Instructions to do so are here: http://www.sdsc.edu/user_services/srb/account_form.html • Your ~/.srb/.MdasEnv file OR env variables mdasCollectionHome '/home/<user-name>.teragrid' mdasDomainName 'teragrid' srbUser '<user-name>' #AUTH_SCHEME 'ENCRYPT1' AUTH_SCHEME 'GSI_AUTH' srbHost 'srb.sdsc.edu' srbPort '7321' defaultResource 'sfs-tape-tgd' SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb'

  22. Scommand Features • Command line interface -> SCRIPTING • Available for all of the most popular UNIX flavors and DOS • S-commands are the most flexible and powerful of the clients • They are the fastest, and most reliable • They are multithreaded for big gains in data flow • They are great for scripts, perl wrappers, batch jobs, etc… • Installed man pages via “man [Scommand]” • man Sput

  23. Scommand Notes • Shelp • Gives list of commands with brief summary • “[Scommand] <return>” gives usage info (usually) or try –h flag • Sinit – establishes a session • Senv – displays connection information • Spwd – display current working directory • Sexit – ends session

  24. Some Public SRB Collections • Southern California Earthquake Center /home/public.teragrid/SCEC • Two Micron All Sky Survey /home/public.teragrid/2MASS • The Palomar Digital Sky Survey /home/public.teragrid/DPOSS

  25. Watch me do a SRB demo

  26. Thanks! • SRB handles large data and provides the ability to share and collaborate on distributed heterogeneous resources. • Questions? • Teragrid SRB userguide: http://www.sdsc.edu/user_services/srb • SRB website: • www.sdsc.edu/srb • SRB email: • srb@sdsc.edu

More Related