Virtualization: Towards More Flexible and Efficient Grids

1. Virtualization: Towards More Flexible and Efficient Grids Kate Keahey keahey@mcs.anl.gov Argonne National Laboratory

2. The Grid Metaphor What happens if a power station fails? How do we store energy? How do we charge for energy? What elements make for a safe and efficient power Grid? How do we ensure quality of service? How do we reliably deliver energy? How do we make sure that supply meets demand? Kate Keahey, PPAM 2005

3. What happens if a power station fails? How do we store computing? Tera Grid How do we charge for computing? Caltech ANL What elements make for a safe and efficient power Grid? SDSC NCSA How do we ensure quality of service? How do we reliably deliver cycles? Grid Middleware How do we make sure that supply meets demand? Computational Grids How can we manage different computing environments? What is the “unit” of resource usage? We need a “computon” that will combine environment and enforcement aspects as well as a way of managing the multi-dimensional nature of the Grid How can we negotiate for computation? How can we ensure that disk, CPUs, network are all available? Kate Keahey, PPAM 2005

4. Grids Today • Grid Middleware Tools • Security, Data Management, Resource Management & Scheduling, Monitoring • Standards: GGF, OASIS • Implementations: Globus Toolkit, Condor and others • Many new services are being developed • Significant deployments and use of Grid infrastructure • TeraGrid, Open Science Grid (OSG), Grid 3, many European deployments • Multiple projects making production use of Grid infrastructure. • Still issues: heterogeneity, lack of satisfactory control and accounting, no on-demand computing Kate Keahey, PPAM 2005

5. The Virtualization Layer • Virtual Grids: virtualize computers, networks, disks, memory • Overlay networks, virtual storage… • Use middleware to map the virtualized constructs onto physical hardware • Trust middleware to map and remap the virtual environment as needed • Trust market forces to ensure that physical resources are plentiful when you need them Kate Keahey, PPAM 2005

6. Virtual Workspace • For now: focus on “virtual workspace” • Unit of enforcement, a “computon” for the Grid • Representation of a desired environment • Later: put all elements of the system together into a virtual Grid • We need progress in the following areas: • Protocols to dynamically negotiate and describe a workspace • Ongoing work at GGF: WS-Agreement, JSDL spec • A unit of enforcement • A “critical mass” implementation • Recent revival in virtual machine technologies provides potential for such an implementation Kate Keahey, PPAM 2005

7. VM VM VM Virtual Machine Basics • A VM can serialize all of its state (including RAM) • A VM image is simply a collection of files • Disk partitions, RAM, configuration file • Such image can be easily moved (migrated) between hypervisors of the same type • Such image can also be saved and used for rollbacks App App App App App Guest OS (Linux) Guest OS (NetBSD) Guest OS (Windows) Virtual Machine Monitor (VMM) / Hypervisor Hardware Kate Keahey, PPAM 2005

8. Different Hypervisor Implementations • Depending on the layer you virtualize you will end up with a different VM • API: language VMs (JVM) • ISA: system VMs (VMware) • Different types of system virtual machines • Full virtualization (VMware) • Run multiple unmodified guest OSs • Para-virtualization (Xen, UML, Denali) • Run multiple guest OSs ported to a special architecture • Single OS image (Vserver) • What is the cost of using VMs? • Paper: “From Sandbox to Playground: Dynamic Virtual Environments in the Grid”, Grid 2004 Kate Keahey, PPAM 2005

9. 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 L X V U L X V U L X V U L X V U SPEC INT2000 (score) Linux build time (s) OSDB-OLTP (tup/s) SPEC WEB99 (score) Benchmark suite running on Linux (L), Xen (X), VMware Workstation (V), and UML (U) The Need for Speed Paper: “Xen and the Art of Virtualization”, SOSP 2003 Kate Keahey, PPAM 2005

10. Licensing and Distribution • License • Open source (Xen, UML) • Visible effects of open source community at work • Commercial (VMware) • Also, XenSource • Distribution/Installation • Para-virtualization requires kernel modifications • Yes, but … everything else stays the same • Xen is (or soon to be) part of multiple distributions: Fedora Core 4, Debian, inofficial: Gentoo, Mandrake and SUSE distributions • Work on making Xen part of the Linux kernel • Privilege • Xen (root, patch kernel, domain 0 privileges setup) • VMware Workstation (root, installation only) • UML: user-level Kate Keahey, PPAM 2005

11. What Makes VMs Great • Summary of VM properties: • Good isolation properties • Generally enhanced security, audit forensics • Excellent enforcement potential • Details depend on implementation • Customizable software configuration • Library signature, OS, maybe even 64/32-bit architectures • Serialization property • VM images (include RAM), can be copied • The ability to pause and resume computations • Allow migration • How do we make VMs available over the network and manage them so as to leverage this potential? • Challenges: security, enforcement, protocols Kate Keahey, PPAM 2005

12. Grid Services • Web Service Resource Framework • An extension of Web Services • Provides standard mechanisms for • Creation • Lifetime Management • State management, inspection (notification) • Globus Toolkit 4 • Implementation of the WSRF framework • Available since April 2005 • Provides secure authentication, authorization as well as tools for fast transfer, replica management, monitoring, and others. Kate Keahey, PPAM 2005

13. What are Virtual Workspaces? • Virtual Workspaces: environments that can be made available dynamically the Grid • well-defined properties in terms of environment definition and resource usage enforcement • Examples: • A physical cluster booted to a desired configuration (e.g. Cluster on Demand) • A Grid3 node dynamically configured using Pacman • A cluster partition configured with a hypervisor • A VM representing an OSG configuration enforcing memory and CPU usage • Workspaces can be implemented using a variety of technologies • VMs are the most promising Kate Keahey, PPAM 2005

14. Virtual Workspace • Environment Aspect (workspace meta-data) • Information/state that outlives its deployment • Generic information (name, time to live) • Attested software partition information: OS, “OSG configuration”, “application installation”, etc. • Services: ssh, GRAM, pre-configured job • Resource allocation request (deployment time) • Flexibly negotiated within desired constraints • See GGF WS-Agreement standard • Memory, disk, networking, etc. • See GGF JSDL standard • On deployment the actual resource allocation information becomes available for inspection • Atomic workspaces and virtual clusters • Clusters are simply aggregate workspaces Kate Keahey, PPAM 2005

15. Define workspace environment Manage workspace Negotiate workspace deployment characteristic request a workspace workspace meta-data negotiate workspace deployment manage/monitor/renegotiate workspace deployment Workspace Deploying Workspaces in the Grid Workspace Wizard (VW Factory) manage workspace environment Workspace Management Service (VW Repository) workspace metadata Workspace Service (VW Manager) terminate workspace deployment manage activities within the workspace Kate Keahey, PPAM 2005

16. Current Implementation • Current prototype using Globus Toolkit 4 • Leveraging standard Grid Service features • Workspace Wizard • Returns workspace meta-data • Very rudimentary implementation • Workspace Service • Create: takes workspace meta-data and a deployment descriptor • Manage: • renegotiate resource allocation • Also traditional Grid Service management: TTL, etc. • Destroy • Different options: pause, shutdown or destroy • First tech preview release expected later this month Kate Keahey, PPAM 2005

17. How dynamic is the deployment? • Automatic • Protocol-based • Moving towards better articulation of migration • Renegotiation of resource allocation • How fast is this deployment? • Deployment of workspace for EMBOSS suite: • Manual: ~45 minutes • Based on pre-configured Vmware VMs: ~6 minutes • Based on pre-configured Xen VM: < 1 second • How much overhead does workspace deployment add over what we have today? Kate Keahey, PPAM 2005

18. How much deployment overhead are we adding? • Using a paused VM allows us to “save” on initiation time GRAM job execution GRAM job execution in a paused Xen VM job execution in a booted Xen VM (pre-configured job) Kate Keahey, PPAM 2005

19. Workspace Service: Virtual Clusters Kate Keahey, PPAM 2005

20. Workspace Deployment Across Technologies • Basic node configuration (+/-boot from image) • Cluster on Demand, PXE, bcfg • On the order of many minutes (~30 minutes) • Refining configuration, creating access • Dynamic account with workspace service: < 1s (mostly GT4 request processing time) • Refining Installation: ~2 hours to configure an ATLAS node using Pacman • Virtual machines • Deploying images • Xen: ~100 ms • VMware Workstation: ~ several seconds Kate Keahey, PPAM 2005

21. program program program … VM deploy VM workspace (with hypervisor/OS) VM VM deploy hypervisor/OS workspace Hypervisor/OS procure hardware Physical machine Nested Workspaces It is easier to maintain a few hypervisor configurations than thousands of user configurations. Those can be deployed in virtual machines. Kate Keahey, PPAM 2005

22. Virtual Grid Virtual Playgrounds Application Kate Keahey, PPAM 2005

23. Ongoing Work on Workspaces • Dynamic resource management with VMs • Virtual clusters, fine-grained resource mangement, migration, moving towards economic management • X. Zhang, T. Freeman • IP overlay network for virtual machines • Management infrastructure for VM IP addresses • T. Freeman & L. Chen • Secure management of VM images • Image attestation and verification • Handling image distribution • Managing workspace identity • W. Lu, T. Freeman, F. Siebenlist • Deployment • Edge Services for OSG: with F. Wuertherwein & A. Rana Kate Keahey, PPAM 2005

24. Related Work • In-Vigo • VM-based infrastructure for the Grids • VM deployment, virtual storage, virtual networks • Renato Figueiredo, Jose Fortes • Virtuoso • VNET: virtualizing networks • Peter Dinda & lab • VIOLIN • Isolated, virtual networks for VMs • Dongyan Xu & lab • Cluster on Demand • Clusters of VMs on demand, also networking, resource management • Jeff Chase and lab Kate Keahey, PPAM 2005

25. The Challenges that Lie Ahead • Deployment • How do I prepare a cluster for VM execution? • Reserve and publish • Site-specific versus Grid-specific • What security trade-offs are acceptable? • How will VM usage change site configuration? • And many, many others • Environment configuration management • How to configure and manage a VM? • GGF CDDLM working group • Packaging infrastructures • Security • Huge potential: how are we going to leverage it? • Economics, Grid markets, and many others Kate Keahey, PPAM 2005

26. Conclusions • Virtual is the new real! • Virtualization is emerging as an important abstraction layer in the Grids • Virtual workspaces are cornerstone of this new abstraction layer • Rapidly developing VM technology has the potential to implement a “computon” for the Grids • Fast, accessible VMs • “critical mass” implementation for virtual workspaces • Two sides to providing “computation on tap” • Abstractions and enforcement mechanisms • Protocols • There is much ongoing work in VMs but even more challenges still like ahead Kate Keahey, PPAM 2005

27. If you like a challenge, give us a call keahey@mcs.anl.gov