230 likes | 348 Views
Corollary. System Overview. Second Key Idea: Specialization. Think GoogleFS. Third idea: Enable cross-layer optimizations Layered Architectures: High benefits, but …. TCP/IP File System Benefits, but… … limits information flow across layers. API. Cross-Layer Optimizations. Examples
E N D
Second Key Idea: Specialization • Think GoogleFS
Third idea: Enable cross-layer optimizationsLayered Architectures: High benefits, but … • TCP/IP • File System • Benefits, but… • … limits information flow across layers. API http://netsyslab.ece.ubc.ca
Cross-Layer Optimizations • Examples • IP • Storage systems • …. • Applications Storage System • Performance • QoS requirements • Consistency requirements • Applications Storage System • Provide storage-level information to applications Data Intensive Applications: Co-usage of files Data Intensive Schedulers: Notification about data movements What’s missing? A vehicle to pass information across layers http://netsyslab.ece.ubc.ca
Traditional Use of Custom Metadata Author=Smith input.dat File Browser Application Layer POSIX API Metadata Manager File System Layer File Organization Module Basic File System Storage System Layer http://netsyslab.ece.ubc.ca
Cross-Layer Communication OK. Schedule Task on node3 Replicate input.dat 3x Application Layer POSIX API Metadata Manager File System Layer File Organization Module Basic File System Storage System Layer input.dat moved from node1 to node3 http://netsyslab.ece.ubc.ca
Recap • Object-based storage • Enable specialization --> performance • Enable cross-layer optimization --> genrality
Workflow Example - ModFTDock • Protein docking application • Simulates the creation of a complex protein from two known proteins • Applications • Drugs design • Protein interaction prediction
Platform Example – Argonne BlueGene/P 2.5K IO Nodes 160K cores GPFS IO rate: 8GBps = 51KBps / core !! 10 Gb/s Switch Complex Torus Network 24 I/O servers 2.5 GBps per node 3D Torus 850 MBps per 64 nodes Tree The central storage is a potential bottleneck Underused resources
App. task App. task App. task App. task App. task Local storage Local storage Local storage Local storage Local storage Background – ModFTDock in Argonne BG/P 1.2 M Docking Tasks Workflow Runtime Engine File based communication Large IO volume Scale: 40960 Compute nodes IO rate : 8GBps = 51KBps / core Backend file system (e.g., GPFS, NFS)
App. task App. task App. task Local storage Local storage Local storage Intermediate Storage Approach Workflow Runtime Engine Scale: 40960 Compute nodes … POSIX API Stage Out Intermediate Storage Stage In Backend file system (e.g., GPFS, NFS)
Usage scenario II: • Support for deduplication
Stakeholders • The final clients • Financing agencies ($) • DoE • NSERC • Science teams • Development team • Graduate students (6+) • Undergraduate students, visitors (10+) • Me
Stakeholders – and their goals • The final clients • Financing agencies ($) • DoE • NSERC • Science teams • Development team • Graduate students (6+) • Undergraduate students, visitors (10+) • Me
Requirements • Easy to deploy • Easy to integrate with applications • Versatility and ability to configure • Efficiency / high-performance /scalability • Ability to support versioning and partially similar data. All have big architectural implications
Early architectural decisions • Object-based storage • system structure 2.) Network/protocol stack: uniform - Stateless to the degree possible
Early architectural decisions 3.) FUSE-based implementation • Impact: structure, deployability 4.) Policy to manage tension between code maturity and need to experiment
Mid-way architectural decisions 5.) GeneralIO hack 6.) Test-driven design - integrate 3month projects
Implicit architectural policies 7.) Personnel management: • prioritize ‘fun’ • Flat Team structure • Bottom-up decision making / prioritization: • ‘campaigns’ 8.) Align ‘values’
Key architectural decisions • Object-based storage 2.) Uniform protocol stack 3.) POSIX, FUSE-based implementation, 4.) Policy to manage tension between code maturity and need to experiment 5.) GeneralIO hack 6.) Test-driven design 7.) Personnel management: prioritize ‘fun’ 8.) Align values