50 likes | 155 Views
The Armada framework for parallel I/O on computational grids. Ron Oldfield and David Kotz Department of Computer Science Dartmouth College. Must use large remote datasets Often computationally intensive Datasets often need pre- and/or post-processing Examples Climate modeling (EOS-DAS)
E N D
The Armada framework for parallel I/O on computational grids Ron Oldfield and David Kotz Department of Computer Science Dartmouth College FAST 2002 WIP Presentation
Must use large remote datasets Often computationally intensive Datasets often need pre- and/or post-processing Examples Climate modeling (EOS-DAS) Astronomy (Digital Sky Surveys) Comp. Biology (Computed MicroTomography) Computational physics Flexibility Application control of the interface Application control of system policies (caching, data-dist., …) Performance Parallel data transfers. Remote execution of user code (e.g., filtering, transforms, compression, encryption) Introduction Computational Grids: geographically distributed networks of heterogeneous computer systems and devices. Data-intensive grid applications I/O system requirements FAST 2002 WIP Presentation
Flexible design, based on stackable file systems. Applications access data through a graph of “ships” called an “armada”. Requests travel toward data servers. Data is pushed toward clients for reads, pulled toward servers for writes. The armada abstracts details of the I/O system Caching, filtering, data distribution clients on site A API data flow added by application filter replica from data provider dist dist file file file file file data segments on site B data segments from site C Armada An I/O framework for data-intensive grid applications FAST 2002 WIP Presentation
clients on site A clients on site A API API API API combine filter dist dist replica replica replica filter filter filter filter filter dist dist file file file file file file file file file file data segments on site B data segments from site C data segments on site B data segments from site C Improving Performance FAST 2002 WIP Presentation
In progress… • Automate graph restructuring. • Formalize rules and algorithms • Develop placement algorithms. • Requires detailed information on ship requirements and available resources. • Performance monitoring and analysis. Contact Information Ron Oldfield (raoldfi@cs.dartmouth.edu) David Kotz (dfk@cs.dartmouth.edu) http://www.cs.dartmouth.edu/~dfk/armada FAST 2002 WIP Presentation