1 / 17

VGrADS: Virtual Grid Development Software Project

Explore the VGrADS project by Ken Kennedy Center for High Performance Software at Rice University, focusing on enhancing grid computing with abstract views and efficient execution. Learn about virtual grids, programming tools, applications, and educational initiatives.

peggymartin
Download Presentation

VGrADS: Virtual Grid Development Software Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Virtual Grid Application Development Software (VGrADS) Project Ken Kennedy Center for High Performance Software Rice University http://www.hipersoft.rice.edu/vgrads/

  2. Fran Berman Andrew Chien Henri Casanova Rich Wolski Carl Kesselman Keith Cooper Ken Kennedy Charles Koelbel Linda Torczon Dan Reed Jack Dongarra Lennart Johnsson The VGrADS Team • VGrADS is an NSF-funded Information Technology Research project • Plus many graduate students, postdocs, and technical staff!

  3. Database Supercomputer Database Supercomputer The VGrADS Vision:National Distributed Problem Solving • Where We Want To Be • Transparent Grid computing • Submit job • Find & schedule resources • Execute efficiently • Where We Are • Low-level hand programming • What Do We Need? • A more abstract view of the Grid • Each developer sees a specialized “virtual grid” • Simplified programming models built on the abstract view • Permit the application developer to focus on the problem

  4. Program Preparation System Execution Environment Performance Feedback Real-time Performance Performance Problem Software Monitor Components Resource Config- Whole- Source Grid Negotiator urable Appli- Program Negotiation Runtime Object Compiler cation System Scheduler Program Binder Libraries The Original GrADS Vision

  5. Lessons from GrADS • Mapping and Scheduling for MPI Jobs is Hard • Although we were able to do some interesting experiments • Performance Model Construction is Hard • Hybrid static/dynamic schemes are best • Difficult for application developers to do by hand • Heterogeneity is Hard • We completely revised the launching mechanisms to support this • Good scheduling is critical • Rescheduling/Migration is Hard • Requires application collaboration (generalized checkpointing) • Requires performance modeling to determine profitability • Scaling to Large Grids is Hard • Scheduling becomes expensive

  6. VGrADS Virtual Grid Hierarchy

  7. Virtual Grids and Tools • Abstract Resource Request • Permits true scalability by mapping from requirements to set of resources • Scalable search produces manageable resource set • Virtual Grid services permit effective scheduling • Fault tolerance, performance stability • Look-Ahead Scheduling • Applications map to directed graphs • Vertices are computations, edges are data transfers • Scheduling done on entire graph • Using automatically-constructed performance models for computations • Depends on load prediction (Network Weather Service) • Abstract Programming Interfaces • Application graphs constructed from scripts • Written in standard scripting languages (Python,Perl,Matlab)

  8. Virtual Grids • Goal: Provide abstract view of grid resources for application use • Will need to experiment to get the right abstractions • Assumptions: • Underlying scalable information service • Shared, widely distributed, heterogeneous resources • Scaling and robustness for high load factors on Grid • Separation of the application and resource management system • Basic Approach: • Specify vgrid as a hierarchy of … • Aggregation operators(ClusterOf, LooseBagOf, etc.) with … • Constraints(type of processor, installed software, etc.) and … • Application-basedrankings(e.g. predicted execution time) • Execution system returns (candidate) vgrid, structured as request • Application can use as it sees fit, make further requests

  9. Programming Tools • Collaborating on definition of the Virtual Grids interface • Initial experiments based on GrADS infrastructure • Focus: Automating critical application-development steps • Building workflow graphs • From Python scripts used by EMAN • Scheduling workflow graphs • Heuristics required (problems are NP-complete at best) • Good initial results if accurate predictions of resource performance are available (see EMAN demo) • Constructing of performance models • Based on loop-level performance models of the application • Requires benchmarking with (relatively) small data sets, extrapolating to larger cases • Initiating application execution • Optimize and launch application on heterogeneous resources

  10. VGrADS Applications • EMAN • Contruction of 3-D models from 2-D electron micrographs • Talk by Charles Koelbel, Posters by Anirban Mandal and Bo Liu and by Gabriel Marin • LEAD • Collection of applications for real-time weather prediction, sponsored by NSF ITR Large • Talk by Dan Reed, Poster? • GridSAT • Satisfiability on the Grid • Talk by Rich Wolski? Poster? • EOL (Encyclopedia of Life) • Reduced Emphasis (see notes)

  11. VGrADS Education • Graduate Education • Shared students • New courses • Underrepresented Minorities • Support for AGEP program • Activities at Tapia and Hopper Conferences • Others? • Courses • Andrew’s Grad Course • New Internet Architectures Course

  12. Leverage • TIGRE and LEARN • State of Texas • 500K for two years • Application driven software stack • NMI • Community infrastructure that can be counted on • Vehicle for deployment of successful research • Outlet for technology

  13. Management • Management • Executive Committee • Technical Working Group • Strategy for Collaboration • Workshops, Telecons • Grad student exchange • Value of Virtual Organization • No one site could take on topic of this breadth • Address KDI report

  14. Outreach • Web Site • Participation at SC Conferences • Two application demonstrations • Conferences and Talks • Application Collaborations

  15. EMAN - Electron Microscopy Analysis [Rice, Houston] 3D reconstruction of particles from electron micrographs Workflow scheduling and performance prediction to optimize mapping GridSAT - Boolean Satisfiability [UCSB] Classic NP-complete problem useful in circuit design and verification Performance-based dynamic resource allocation and scheduling EMAN Refinement Process VGrADS Demos at SC04

  16. Research vs Development • Original Proposal • Research • Significant investment in software infrastructure development • Application collaborations (funded staff members) • Revised Statement of Work • Focus on research (guidance from NSF) • Situation Today • To conduct research we are building prototype software • Leveraging NMI and Texas • Application collaborations still drive research

  17. Summary • What Justifies a Large ITR? • Community: No one site covers everything • Project Vision • Shared Infrastructure • Integration: Would not happen without the unified project • VGrADS • Built on GrADS • Community of leading researchers who work together effectively • Broad coverage topics • Vision for extremely simple application development interface • Extensive shared testbed and shared software stack • Many interrelated layers require integrated effort

More Related