Nut Taesombut and Andrew A. Chien Department of Computer Science and Engineering

1. Distributed Virtual Computer (DVC):Simplifying the Development of High-Performance Grid Applications Nut Taesombut and Andrew A. Chien Department of Computer Science and Engineering University of California, San Diego Workshop on Grids and Advanced Networks (GAN’04) Chicago, Illinois April 22, 2004

2. Outline • Background and Motivation • Distributed Virtual Computer (DVC) • Example Application • Related Work • Summary and Future Work

3. Emerging Opportunity of Lambda Grids • Network Advances and Trends • DWDM optical paths (or lambdas) enable • Dedicated High Bandwidth • Dynamic Configuration • Lambda Grid • Distributed, shared resources interconnected by plentiful lambdas • Configurable Connections and Capacity • Deterministic Communication Performance • Novel Communication Capabilities (e.g. optical multicast)

4. OptIPuter Project • OptIPuter – Large-scale Research Project on Impact of Lambdas on System Software and next-generation E-science • International Testbed for Experimentation (UCSD, UIC, UCI, Amsterdam, etc.) • Leading E-science Drivers (Neuroscience, Geophysical/Earth Sciences) • 3-D Data Analysis, Visualization and Collaboration Applications • Data-intensive and Real-time, Distributed data sources/sinks • Wealth of Innovative System Software Research (protocols, DVC, storage, etc.) Smarr, Papadopoulos, Ellisman, Orcutt, Chien – UCSD DeFanti, Leigh - UIC http://www.optiputer.net

5. Motivation • Building Grid Applications is Difficult! • Applications must deal with complexity of resource environment • Resource Heterogeneity, Performance, Communication • Multi-Organization Security, Resource Management • Shared and Untrusted Resource Environments • Low-Quality Networks • Adding Low-level Management of Network Complicates the Task • No Uniform Interfaces (routers, switches, end nodes) • Wildly Different Semantic Level (BELOW IP!) • Novel Capabilities (e.g. multicast, RDMA, etc.) • Key Requirements • A new abstraction which simplifies Grid environments • A view which integrates communication as first class

6. Example of Grid Complexity • Access to Resources Across Multiple Namespaces • User-Controlled Configuration of Dynamic Lambda Network • Heterogeneous Communication http://dream.uci.edu:4010 http://intania.uic.edu:4566 Internet http://zebra.ucsd.edu:5220 10.1.3.34 10.1.2.61 10.1.2.68

7. Distributed Virtual Computer (DVC) • A Simple Execution Environment for Grid Applications • Set of LambdaGrid resources (connections, resources) • Naming, access, and management services • Transparently shared amongst Applications • Simplify Use of Network and Grid Capabilities • Automate compute/data resource binding • Automate dynamic λ-configuration; expose novel λ-capabilities • Leverage existing Grid Technologies (Globus, NWS etc.) DVC

8. DVC Design Principles • Separate Resource Config/Mgmt and Application Programming • Resource Environments Configured to Spec • Applications simply use them • Aggregate and Bind Grid Resources; Present as Workgroup • Central resource control • Single namespace • Unified resource access mechanisms • Trusted and secure environment • Controllable performance • Enable Collective Resource Views • Unified naming structures (e.g. collective names) • Collective properties (e.g. group communication, trust, access control)

9. Example: Locally Simplified Grid Programming • Single Control Domain • Unified Naming Mechanism • Uniform Use of Different Communication Mechanisms (e.g. protocols) DVC Domain comp1 comp2 http://dream.uic.edu:4010 Internet http://intania.uic.edu:4566 comp3 http://zebra.ucsd.edu:5220 10.1.3.34 10.1.2.61 10.1.2.68 str1 Simple View of DVC

10. How DVC’s Simplify Application Grid Programming • Automate Resource Binding and Configuration • Reduce user interaction through resource broker and manager • Unify Resource Naming and Access Mechanisms • Hide heterogeneity through simple abstractions • Transparently Enable Security Protection • Implement cryptographic operations at the middleware layer • Monitor and Control Resource and Communication Performance • Detect asynchronous events and notify application based on subscription

11. Realizing Simplified Application Grid Programming DVC Data Flow Control Flow DVC Manager Ghost Manager • DVC Manager • Single master controller • Resource selector/negotiator/scheduler • Trust mediator and security authority • Synchronizer of global state information • Ghost Managers • Slave managers, each running at each bound resource • Job process controller at remote resource • Communication mediator • Resource status monitor and reporter

12. Example: Dynamic Configuration of Lambda Grid Duke Harvard NCMIR/UCSD BIRN DVC SDSC UNC UCLA UCI • Example Application: • BIRN (Biomedical Information Research Network) • DVC Advantages • On-demand creation of a private Grid resource workgroup • Transparent use of novel communication capabilities • high-speed multi-point communication • SAN-like storage access across geographically distributed resources

13. Example: Dynamic Configuration of Lambda Grid harvard duke ucsd Grp1 sdsc unc uci uci Grp2 • Sequence to Create a BIRN DVC • Create a resource configuration specification and send a request to bind resources • Create resource groups (i.e. for collective data source and sink) • Create multipoint-to-point and point-to-multipoint communication sessions • Define the properties of communication sessions (e.g. security and communication mechanisms) Duke Harvard NCMIR/UCSD GTP + enc + auth TCP + Optical Multicast SDSC UNC UCLA UCI Physical-Level View of BIRN DVC Application-Level View of BIRN DVC

14. Related Work • Abstractions of Distributed Resources • PVM [Geist94] • Grid Middleware • Globus System • Grid Programming Tools • GridRPC [Nakada02], MPICH-G2 [Karonis03], Condor-G [Frey01] • GrADS [Berman01], GridLab [Allen03], • Federation of Resources • Virtual Organization (VO) [Foster01] • Distributed Virtual Computer • Provide an Application-Focused Dynamic Resource Container • Dynamic resource configuration and sharing policies

15. Summary and Future Work • Summary • DVC’s provide simple computing environments for Grid applications • Locally simplified resource workgroup • DVC’s allow on-demand instantiation and dynamic configuration of Lambda-Grid • DVC’s enable simple use of novel communication capabilities • Future Work • Develop the full implementation of the DVC Prototype • Implement as Web Services (i.e. WS-RF specification) • Deploy the prototype on the OptIPuter Testbed • Demonstrate with OptIPuter applications (Bioinformatics and Geophysical) • Explore other system technologies that can be integrated into the DVC framework • Real-time System • High-Performance Distributed File System

16. Thank You • Questions and Remarks? • Contact Information: • Nut Taesombut (nut@cs.ucsd.edu) • OptIPuter Website: • www.optiputer.net