1 / 44

Managing Simulation Output

Managing Simulation Output. Storage Resource Broker. Reagan W. Moore moore@sdsc.edu http://www.sdsc.edu/srb. Topics. Data management evolution Shared collections Digital Libraries Persistent Archives Managing simulation output Project level / National level / International

gloria-wood
Download Presentation

Managing Simulation Output

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Managing Simulation Output Storage Resource Broker Reagan W. Moore moore@sdsc.edu http://www.sdsc.edu/srb

  2. Topics • Data management evolution • Shared collections • Digital Libraries • Persistent Archives • Managing simulation output • Project level / National level / International • Demonstration of shared collections • Access to collections at SDSC

  3. Using a Data Grid – in Abstract Data delivered Ask for data • The data is found and returned • Where & how details are hidden Data Grid • User asks for data from the data grid

  4. Using a Data Grid - Details DB Storage Resource Broker Metadata Catalog Storage Resource Broker • User asks for data • Data request goes to SRB Server • Server looks up data in catalog • Catalog tells which SRB server has data • 1st server asks 2nd for data • The data is found and returned

  5. Using a Data Grid - Details DB MCAT SRB SRB SRB SRB SRB SRB • Data Grid has arbitrary number of servers • Heterogeneity is hidden from users

  6. Storage Resource Broker 3.3.1 C Library, Java Unix Shell Databases - DB2, Oracle, Sybase, Postgres, mySQL, Informix File Systems Unix, NT, Mac OSX Archives - Tape, Sam-QFS, DMF, HPSS, ADSM, UniTree, ADS Application DSpace, OpenDAP, GridFTP, Fedora http, Portlet, WSDL, OAI-PMH) DLL / Python, Perl, Windows Linux I/O C++ NT Browser, Kepler Actors, HDF5 Library Federation Management Consistency & Metadata Management / Authorization, Authentication, Audit Latency Management Metadata Transport Logical Name Space Data Transport Database Abstraction Storage Repository Abstraction Databases - DB2, Oracle, Sybase, Postgres, mySQL, Informix ORB

  7. Storage Resource Broker 3.3.1 C Library, Java Unix Shell Databases - DB2, Oracle, Sybase, Postgres, mySQL, Informix File Systems Unix, NT, Mac OSX Archives - Tape, Sam-QFS, DMF, HPSS, ADSM, UniTree, ADS Application DSpace, OpenDAP, GridFTP, Fedora http, Portlet, WSDL, OAI-PMH) DLL / Python, Perl, Windows Linux I/O C++ NT Browser, Kepler Actors, HDF5 Library Federation Management Consistency & Metadata Management / Authorization, Authentication, Audit Latency Management Metadata Transport Logical Name Space Data Transport Database Abstraction Storage Repository Abstraction Databases - DB2,Oracle, Sybase, Postgres, mySQL, Informix ORB

  8. Logical Name Spaces Data Access Methods (C library, Unix, Web Browser) • Storage Repository • Storage location • User name • File name • File context (creation date,…) • Access constraints Data access directly between application and storage repository using names required by the local repository

  9. Logical Name Spaces Data Access Methods (C library, Unix, Web Browser) Data Collection • Storage Repository • Storage location • User name • File name • File context (creation date,…) • Access constraints • Data Grid • Logical resource name space • Logical user name space • Logical file name space • Logical context (metadata) • Control/consistency constraints Data is organized as a shared collection

  10. Federation Between Data Grids Data Access Methods (Web Browser, DSpace, OAI-PMH) Data Collection A Data Collection B • Data Grid • Logical resource name space • Logical user name space • Logical file name space • Logical context (metadata) • Control/consistency constraints • Data Grid • Logical resource name space • Logical user name space • Logical file name space • Logical context (metadata) • Control/consistency constraints Access controls and consistency constraints on cross registration of digital entities

  11. NARA Persistent Archive NARA U Md SDSC MCAT MCAT MCAT Original data at NARA, data replicated to U Md & SDSC Replicated copy at U Md for improved access, load balancing and disaster recovery Active archive at SDSC, user access Federation of Three Independent Data Grids • Demonstrate preservation environment • Authenticity • Integrity • Management of • technology evolution • Mitigation of risk of data loss • Replication of data • Federation of catalogs • Management of preservation • metadata • Scalability • Types of data collections • Size of data collections

  12. Shared Collections • Data grids support the creation of shared collections that may be distributed across multiple institutions, sites, and storage systems. • Digital libraries publish data, and provide services for discovery and display • Persistent archives preserve data, managing the migration to new technology

  13. Shared Collections • Purpose of SRB data grid is to enable the creation of a collection that is shared between academic institutions • Register digital entity into the shared collection • Assign owner, access controls • Assign descriptive, provenance metadata • Manage state information • Audit trails, versions, replicas, backups, locks • Size, checksum, validation date, synchronization date, … • Manage interactions with storage systems • Unix file systems, Windows file systems, tape archives, … • Manage interactions with preferred access mechanisms • Web browser, Java, WSDL, C library, …

  14. Federated Server Architecture Peer-to-peer Brokering Read Application Parallel Data Access Logical Name Or Attribute Condition 1 6 5/6 SRB server SRB server 3 4 5 SRB agent SRB agent 2 Server(s) Spawning R1 MCAT 1.Logical-to-Physical mapping 2.Identification of Replicas 3.Access & Audit Control R2 Data Access

  15. Generic Infrastructure • Digital libraries now build upon data grids to manage distributed collections • DSpace digital library - MIT and Hewlitt Packard • Fedora digitial library - Cornell University and University of Virginia • Persistent archives build upon data grids to manage technology evolution • NARA research prototype persistent archive • California Digital Library - Digital Preservation Repository • NSF National Science Digital Library persistent archive

  16. Southern California Earthquake Center Select Receiver (Lat/Lon) Output Time History Seismograms Select Scenario Fault Model Source Model SCEC Community Library • Intuitive User Interface • Pull-Down Query Menus • Graphical Selection of Source Model • Clickable LA Basin Map (Olsen) • Seismogram/History extraction (Olsen) • Access SCEC Digital Library • Data stored in a data grid • Annotated by modelers • Standard naming convention • Automated extraction of selected data and metadata • Management of visualizations SCEC Digital Library

  17. Terashake Data Handling • Simulate 7.7 magnitude earthquake on San Andreas fault • 50 Terabytes in a simulation • Move 10 Terabytes per day • Post-Processing of wave field • Movies of seismic wave propagation • Seismogram formatting for interactive on-line analysis • Velocity magnitude • Displacement vector field • Cumulative peak maps • Statistics used in visualizations • Register derived data products into SCEC digital library

  18. Humidity Climate Ecological Wireless Oceanography Wind Speed Climate Ecological Wireless Oceanography ROADNet Sensor Network Data Integration Seismic Geophysics Rain start Fire start Frank Vernon - UCSD/SIO

  19. National Science Digital Library • URLs for educational material for all grade levels registered into repository at Cornell • SDSC crawls the URLs, registers the web pages into a SRB data grid, builds a persistent archive • 750,000 URLs • 13 million web pages • About 3 TBs of data

  20. Astronomy Data Grid • Chile • Tucson, Arizona • NCSA, Illinois • A functioning international Data Grid for Astronomy Manchester-SDSC mirror Moved over 400,000 images

  21. BaBar High-energy Physics • Stanford Linear Accelerator • Lyon, France • Rome, Italy • San Diego • RAL, UK • A functioning international Data Grid for high-energy physics Manchester-SDSC mirror Moved over 100 TBs of data

  22. Worldwide University Network Data Grid • SDSC • Manchester • Southampton • White Rose • NCSA • U. Bergen • A functioning, general purpose international Data Grid for academic collaborations Manchester-SDSC mirror

  23. KEK Data Grid • Japan • Taiwan • South Korea • Australia • Poland • US • A functioning, general purpose international Data Grid for high-energy physics Manchester-SDSC mirror

  24. SRB Objectives • Automate all aspects of data discovery, access, management, analysis, preservation • Security paramount • Distributed data • Provide distributed data support for • Data sharing - data grids • Data publication - digital libraries • Data preservation - persistent archives • Data collections - Real time sensor data

  25. SRB Developers Reagan Moore - PI Michael Wan - SRB Architect Arcot Rajasekar - SRB Manager Wayne Schroeder - SRB Productization Charlie Cowart - inQ Lucas Gilbert - Jargon Bing Zhu - Perl, Python, Windows Antoine de Torcy - mySRB web browser Sheau-Yen Chen - SRB Administration George Kremenek - SRB Collections Arun Jagatheesan - Matrix workflow Marcio Faerman - SCEC Application Sifang Lu - ROADnet Application Richard Marciano - SALT persistent archives Contributors from UK e-Science, Academia Sinica, Ohio State University, Aerospace Corporation, … 75 FTE-years of support About 300,000 lines of C

  26. History • 1995 - DARPA Massive Data Analysis Systems • 1997 - DARPA/USPTO Distributed Object Computation Testbed • 1998 - NSF National Partnership for Advanced Computational Infrastructure • 1998 - DOE Accelerated Strategic Computing Initiative data grid • 1999 - NARA persistent archive • 2000 - NASA Information Power Grid • 2001 - NLM Digital Embryo digital library • 2001 - DOE Particle Physics data grid • 2001 - NSF Grid Physics Network data grid • 2001 - NSF National Virtual Observatory data grid • 2002 - NSF National Science Digital Library persistent archive • 2003 - NSF Southern California Earthquake Center digital library • 2003 - NIH Biomedical Informatics Research Network data grid • 2003 - NSF Real-time Observatories, Applications, and Data management Network • 2004 - NSF ITR, Constraint based data systems • 2005 - LC Digital Preservation Lifecycle Management • 2005 - LC National Digital Information Infrastructure and Preservation program

  27. Development • SRB 1.1.8 - December 15, 2000 • Basic distributed data management system • Metadata Catalog • SRB 2.0 - February 18, 2003 • Parallel I/O support • Bulk operations • SRB 3.0 - August 30, 2003 • Federation of data grids • SRB 3.4 - October 31, 2005 • Feature requests (extensible schema)

  28. Separation of Access Method from Storage Protocols Access Method Map from the operations used by the access method to a standard set of operations used to interact with the storage system Access Operations Data Grid Storage Operations Storage Protocol Storage System

  29. Data Grid Operations • File access • Open, close, read, write, seek, stat, synch, … • Audit, versions, pinning, checksums, synchronize, … • Parallel I/O and firewall interactions • Versions, backups, replicas • Latency management • Bulk operations • Register, load, unload, delete, … • Remote procedures • HDFv5, data filtering, file parsing, replicate, aggregate • Metadata management • SQL generation, schema extension, XML import and export, browsing, queries, • GGF, “Operations for Access, Management, and Transport at Remote Sites”

  30. Examples of Extensibility • Storage Repository Driver evolution • Initially supported Unix file system • Added archival access - UniTree, HPSS • Added FTP/HTTP • Added database blob access • Added database table interface • Added Windows file system • Added project archives - Dcache, Castor, ADS • Added Object Ring Buffer, Datascope • Added GridFTP version 3.3 • Database management evolution • Postgres • DB2 • Oracle • Informix • Sybase • mySQL (most difficult port - no locks, no views, limited SQL)

  31. Examples of Extensibility • The 3 fundamental APIs are C library, shell commands, Java • Other access mechanisms are ported on top of these interfaces • API evolution • Initial access through C library, Unix shell command • Added iNQ Windows browser (C++ library) • Added mySRB Web browser (C library and shell commands) • Added Java (Jargon) • Added Perl/Python load libraries (shell command) • Added WSDL (Java) • Added OAI-PMH, OpenDAP, DSpace digital library (Java) • Added Kepler actors for dataflow access (Java) • Added GridFTP version 3.3 (C library) • Added Fedora

  32. Types of Risk • Media failure • Replicate data onto multiple media • Vendor specific systemic errors • Replicate data onto multiple vendor products • Operational error • Replicate data onto a second administrative domain • Natural disaster • Replicate data to a geographically remote site • Malicious user • Replicate data to a deep archive

  33. How Many Replicas • Three sites minimize risk • Primary site • Supports interactive user access to data • Secondary site • Supports interactive user access when first site is down • Provides 2nd media copy, located at a remote site, uses different vendor product, independent administrative procedures • Deep archive • Provides 3rd media copy, staging environment for data ingestion, no user access

  34. Deep Archive Firewall Deep Archive Staging Zone Remote Zone Server initiated I/O Pull Pull Z2 Z1 Z3 PVN Register Register No access by Remote zones Z3:D3:U3 Z2:D2:U2

  35. iRODS Design Intelligent Rule-Oriented Data System • Traditional shared collection • Metadata catalog manages state information • Add rule engine • Allow dynamic rule changes • Track version of rule, date version was applied and the level of granularity (item, sub-collection) • Manage state information needed for rule execution • Manage state information resulting from rule application • Validate consistency of rules after a change • Validate order of application of rules • Deadlock detection

  36. Rule Engine • Rule is an Event-Condition-Action • An action can be another rule or a micro-service or a set of rules and micro-services • Rules are organized • Named rule sets • Separate list for each rule type • Rules incorporate: • $ - temporary state information • Recovery procedure for each action

  37. Rule Engine Design Issues • What are the types of rules? • Deferred rules (consistency assertions) • Periodic rules (maintenance assertions) • Atomic rules (micro-service composition) • Dynamic rule definition (state information) • Rule types also characterized as: • Consistency rules • Procedural rules • Periodic consistency rules • Event triggered consistency rules • Scheduled versus delayed versus triggered

  38. Implementation • Open-source software • Requires re-write of SRB software system • 30,000 lines of “C” written to date • Rebuilt: • Data management micro-services for remote data manipulation (M. Wan) • Metadata catalog management system (W. Schroeder) • Generic client API generation (standardized options) • Adding • Rule engine (A. Rajasekar) • Rule consistency checking (UCSD CSE)

  39. iRODS - intelligent Rule-Oriented Data System Resource-based Services Micro Service Modules Client Interface Admin Interface Rule Invoker Resources Service Manager Rule Modifier Module Config Modifier Module Metadata Modifier Module Rule Consistency Check Module Consistency Check Module Consistency Check Module Engine Current State Confs Metadata-based Services Rule Base Meta Data Base Micro Service Modules

  40. Rule List 0 ON register_data IF $objPath like /home/collections.nvo/2mass/fits-images/* DO cut [nop] AND check_data_type(fits image) [nop] AND get_resource(nvo-image-resource) [nop] AND registerData [recover_registerData] AND addACLForDataToUser(2massusers.nvo,write) [recover_addACLForDataToUser] AND extractMetadataForFitsImage [recover_extractMetadataForFitsImage] 1 ON register_data IF $objPath like /home/collections.nvo/2mass/* DO get_resource(2mass-other-resource) [nop] AND registerData [recover_registerData] AND addACLForDataToUser(2massusers.nvo,write) [recover_addACLForDataToUser] 2 ON register_data DO get_resource(null) [nop] AND registerData [recover_registerData]

  41. Demonstration • URL • https://srb.npaci.edu/iRods/ruleAdmin.html • https://srb.npaci.edu/iRods/ruleAdminRaja.html • Rule example: 0ON register_data IF $dataType like *image* DO get_resource(null) [nop] AND registerData [recover_registerData] AND extract_metadata_for_image [nop]

  42. For More Information Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.sdsc.edu/srb/

More Related