1 / 16

Data Management for Internet Backplane Protocol

Data Management for Internet Backplane Protocol. by Tang Ming Assoc/Prof. Francis Lee. School of Computer Engineering, Nanyang Technological University, Singapore. 0. Outline. Introduction The Logistical Networking & Globus Replica Catalog

yostm
Download Presentation

Data Management for Internet Backplane Protocol

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Management for Internet Backplane Protocol by Tang Ming Assoc/Prof. Francis Lee School of Computer Engineering,Nanyang Technological University, Singapore

  2. 0. Outline • Introduction • The Logistical Networking & Globus Replica Catalog • Data Management for IBP

  3. 1. Introduction • Backgrounds of IBP • Univ. of Tennessee • Base layer of the Logistical Networking • A technology to share network storages distributed across the globe • Our work • Integrate IBP and Globus Replica Catalog, develop the ExNode File Warehouse, and implement a data management system for IBP.

  4. 2. Logistical Networking • IBP: a middleware to allow the sharing of distributed storage resources over the network • L-Bone: a resource discovery service that maintains a list of public depots and metadata about those depots • exNode: a data structure for aggregation storage allocations • LoRS: provide API and command line tools

  5. 2.1 Examples: Upload / Download a File to IBP with LoRS Tools • $ lors_upload blues.mp3 –f –c 1 –F 1 • File “blues.mp3” is uploaded to IBP and the exNode file “blues.mp3.xnd” is generated. • $ lors_download blues.mp3.xnd –f • File “blues.mp3” is downloaded • The data resources are not shareable to public • Need a data management service for data sharing!

  6. 3. Globus Replica Catalog • Provide the replica location information for Data Grid • Logical File Name  Physical File Name LFN PFN File1 gsiftp://ssim1.sce.ntu.edu.sg/dir/ File2 ftp://ssim4.sce.ntu.edu.sg/dir2/ File3 irm://vista.sas.ntu.edu.sg/tmp/

  7. 3.1 Motivation of using GRC • Globus is the de facto standard of the Grid system • The GRC has been used by many Data Grid projects • It’s a way to integrate IBP and the existed Data Grids

  8. 4. Data Management for IBP • System architecture • Components • The Logistical Networking • Globus Replica Catalog (GRC) • ExNode File Warehouse (EFW) • IBP Data Management (IDM) • Workflow examples (upload & download) • Application

  9. 4.1 System Architecture

  10. Main thread EFW Expiration checking thread Communication thread Communication thread Communication thread Globus RC exNode info. (in memory) Storage client client client … 4.2 ExNode File Warehouse (EFW) • EFW stores and maintains the exNode files for the data in IBP • Also keeps the metadata about the data files, e.g., the expiration dates and time stamps • EFW interacts with IDM and GRC

  11. 4.3 IBP Data Management (IDM) • IDM interacts with the Logistical Networking, GRC, and EFW, and it provides APIs to applications. • Functions: upload, download, refresh, delete, query a file or a file set

  12. 4.4 Workflow – Uploading a File IBP Data Management (IDM) 6 1 7 8 Globus Replica Catalog(GRC) 4 User 5 2 3 IBP exNode file warehouse(EFW)

  13. 4.5 Workflow – Downloading a File IBP Data Management (IDM) 2 1 3 8 Globus Replica Catalog(GRC) User 4 5 6 7 IBP exNode file warehouse(EFW)

  14. 4.6 Support Grid Computing • The system provides a global data cache to support Grid computing

  15. 5. Future Works • Develop data discovery and replica selection service • Support Globus new service: Replica Location Service (RLS) For further information: http://www.ntu.edu.sg/home5/p148943492/idm.htm Email: mtang@pmail.ntu.edu.sg ebslee@ntu.edu.sg

  16. Thanks!

More Related