Symbiotic Virtualization

1. Symbiotic Virtualization John R. Lange Thesis Proposal Department of Electrical Engineering and Computer Science Northwestern University June 2009

2. Introduction • VMs are traditionally Black boxes • Separated from the VMM by a semantic gap • Does provide a clean interface • Does that make sense in today’s environment? • Cloud computing, live migration, differing architectures • Guests should know they are in a VM • Many reasons to bridge the gap • Performance, Security, Monitoring, etc… • Existing approaches don’t allow this • Symbiotic Virtualization is an alternative to black box design

3. Symbiotic Virtualization • Novel approach to designing VMMs and operating systems • OS compatible with native hardware interface • BUT also optionally exposes a software interface that can be used by a VMM • Essentially, the VMM can easily inspect and modify the guest OS • Optional and Incremental

4. Outline • The Semantic Gap • Thesis Statement • Palacios and Kitten • Symbiotic Virtualization • Schedule • Contributions

5. Semantic Gap • VMM architectures are designed as black boxes • Explicit OS interface (hardware or paravirtual) • Internal OS state is not exposed to the VMM • Many uses for internal state • Performance, security, etc... • VMM must recreate that state • “Bridging the Semantic Gap” • Many examples • Virtuoso Project • Lycosid, Antfarm, Geiger, IBMon, many others

6. Virtuoso Project • Bridged the semantic gap for virtual networking • Examine physical network traffic to model application behavior • Provide virtual services to unmodified OSes and Applications • Virtuoso Project Components • VNET • Sundararaj, A., and Dinda, P. Towards virtual networks for virtual machine grid computing.In Proceedings of the 3rd USENIX Virtual Machine Research And Technology Symposium (VM 2004) • VTTIF • Gupta, A., and Dinda, P. Inferring the topology and traffic load of parallel programs running in a virtual machine environment. In Proceedings of the 10th Workshop on Job Scheduling Strategies for Parallel Processing • VADAPT • Sundararaj, A., Gupta, A., and Dinda, P. Increasing application performance in virtual environments through run-time inference and adaptation. In Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing • VRESERVE • Lange, J., Sundararaj, A., and Dinda, P. A. Automatic dynamic run-time optical network reservations. In Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing • VTL • Lange, J. and Dinda, P. Transparent network services via a virtual traffic layer for virtual machines. In Proceedings of the 16th International Symposium on High Performance Distributed Computing

7. VNET • Overlay network for virtual machines • Remotely distributed VMs appear connected to a LAN • Layer 2 overlay, operates on ethernet frames • Supports arbitrary overlay topologies, routing, and link types • Provides mechanisms to maximize network performance

8. VTTIF and VADAPT • Virtual Topology and Traffic Inference Framework • Infers communication topology and traffic load matrix for a VM • VADAPT • Uses information from VTTIF • Adaptively optimizes VNET overlay topology

9. VRESERVE • Automatic and dynamic network reservations • Allows unmodified applications to use circuit switched optical networks • Added optical network reservation interface to VNET • Automatically reserves network link when VTTIF detects traffic between two connected hosts

10. VTL: Transparent Network Services • Manipulate data and signaling of connections to add services to existing unmodified applications and OSes • High Level transformations of Low Level traffic • Transparency: Manipulations invisible to guest environment (Black Box approach) • VTL (Virtual Traffic Layer) • A framework for creating Transparent Network Services • Can transform TCP connections into different protocols

11. Bridging the Semantic Gap • Enables many useful features and optimizations • However… • Current approaches are labor intensive • Reverse engineering an OS • Highly specific to OS implementation • Collected information not always accurate

12. Symbiotic Virtualization • Bridging the semantic gap is hard • Can we design a virtual environment with no gap? • Symbiotic Virtualization • Design both guest OS and VMM to minimize semantic gap • 2 components • Guest OS provides internal state to VMM • Guest OS services requests from VMM • Interfaces are optional • Not required for correct operation

13. Thesis Statement • I propose symbiotic virtualization, an approach to OS design that preserves the benefits of full system virtualization, while enabling performance and functionality benefits. • In symbiotic virtualization, an OS targets the native hardware interface and can run unmodified on raw hardware. However, it also exposes a software interface that can be leveraged by a symbiotic virtualization-aware VMM. • Both the interface and its use by the VMM are optional, but if it exists, and the VMM uses it, the VMM and the OS can mutually benefit. • Symbiotic virtualization is markedly different from the current virtualization approaches, and is best considered as being on a continuum between full system virtualization and paravirtualization.

14. Thesis Goals • Define and formalize Symbiotic Virtualization • Develop formal symbiotic interfaces • Implement symbiotic interfaces inside an OS • Implement set of symbiotic extensions • Use examples to evaluate the symbiotic approach

15. Palacios • OS independent embeddable VMM • Written from scratch at NU and UNM • Designed to be modularly linked into existing kernels • Minimal host OS interface • Compiles into static library • Currently embedded: Kitten and GeekOS • Open Source (BSD License) • Downloaded ~1000 times • Lead developer

16. Palacios Details • Supports 32 and 64 bit environments • Host and Guest • Full hardware virtualization • Currently only supports AMD extensions • Intel VMX in process • Supports Linux and HPC guest OSes • Relatively small: ~28K lines

17. Architecture Palacios

18. Kitten • Lightweight HPC OS from Sandia National Labs • Designed for large scale HPC systems (Cray XT) • Successor to Catamount and earlier lightweight kernels • Based on Linux • Only the necessary components • Limited Linux ABI compatibility • Uses Palacios for virtualization • Embedded as a library • VMs launch as part of job submission • Contributing developer

19. Palacios as an HPC VMM • Minimalist interface: • Does not require extensive host OS features • Easily embedded into even small kernels • Full system virtualization: • Does not require guest OS changes • Runs existing kernels without any porting • Kitten, Catamount, Cray CNL, and IBM’s CNK • Contiguous memory preallocation: • Preallocates guest memory as a physically contiguous region • Vastly simplifies the virtualized memory implementation • Deterministic performance for most memory operations • Passthrough resources and resource partitioning: • Host resources are easily mapped directly into a guest environment • Provides access to high performance devices, with existing device drivers, with no virtualization overhead. • Low noise: • Minimizes the amount of OS noise injected by the VMM layer. • No internal timers and no accumulated deferred work.

20. Symbiotic Virtualization in HPC • HPC environments are well suited to symbiotic techniques • Full trust of the software stack • Fewer security concerns • Specific hardware configurations • Limited number of devices • Constrained problem space • Small number of applications • Implementations can be very specific • Environments are much smaller • Internal OS state is simpler than a general purpose OS • At large scale performance impact is dramatic • Large impetus to optimize VMM and OS

21. HPC Performance Example • Guest OS behavior can differ widely • Must optimize for specific OSes and applications • Example: • Catamount and Compute Node Linux • 2 HPC OSes • Process switching implementation • CNL swaps page tables • Catamount does not • Nested and shadow page tables have very different performance characteristics • Evaluated with 2 HPC benchmarks • HPCCG and CTH • 3 configurations (Native, Shadow Paging, Nested Paging) • Running on RedStorm Development Cages (Cray XT)

22. HPCCG Benchmark Compute Node Linux Catamount

23. CTH Benchmark Compute Node Linux Catamount

24. Takeaway • At large scale minor performance problems become large • Very important to minimize any performance overhead introduced • VMM needs to know about guest internals • Should modify behavior for each guest environment • Which paging method to use depends on guest • Inference is not desirable in HPC environment • Unacceptable performance overhead • Convergence time • Mistakes have large consequences • Symbiotic approach is very appealing

25. Symbiotic Virtualization • Definition based on formalization • Formalized interfaces • Two types of interfaces • Passive information interface • VMM can read guest OS state • Functional interface • VMM can send requests to guest OS • Neither required for OS to function correctly • Symbiotic OS can run on hardware • Non-symbiotic OS can run on symbiotic VMM • Can be implemented incrementally

26. Passive Interface • Formalize the interface for bridging the semantic gap • Ideally removes the gap • Internal state already exists but it is hidden • Existing tools try to recreate this data in the VMM • Symbiotic Interface: • Structure internal OS state in a way that is easily parsed • Semantically rich • Expose OS state to the VMM • Easily accessible

27. Example interface • Linux process list • Organized in a series of lists • Scattered throughout kernel address space • Lots of information included inside • Priority, memory map, open file descriptors, etc • Symbiotic Interface • Collect task information in standard location • Organize information to be easily parsable • Reserved memory page that holds pointers to high priority processes • List of CR3 values that should be cached

28. Functional Interface • Mechanism for OS to expose functionality to VMM • Guest OS services VMM requests • Possible interfaces • Guest OS notifications • VMM can force explicit upcalls • Iterator based system

29. Initial Functional Interface • Partial initial test implementation • Prosnitz and Xia • Implemented inside GeekOS and Palacios • Iterator based • Modelled on RPCs

30. Issues • New VMM/OS interaction model • Traditional virtualization assumptions no longer true • No longer a black box • Some new issues to be addressed • Trust • Design Complexity

31. Symbiotic Trust Model • Current Architectures: unidirectional trust • Guest OS fully trusts VMM • VMM should not trust guest • Restricts VMM from interacting with guest • Symbiotic VMM must trust guest interfaces • BUT it doesn’t have to use them • Selectively enable interfaces depending on trust level • I will examine the implications Symbiotic virtualization has on the trust model

32. Symbiotic Complexity • Symbiotic interfaces can increase complexity of VMMs • Implications for Trusted Computing platforms • Complexity is already there • See examples of bridging the gap • Correct functionality does not require VMM support

33. Evaluation • Performance impact of Symbiotic Interfaces • Comparison against existing interfaces • Lines of code • Complexity of other approaches • Explanation of how the symbiotic functionality is not otherwise possible • Evaluate functionality with several example cases • Examine how issues are addressed by design • Also evaluating virtualization and HPC at scale

34. Implementation • Implementation of formalized design • Environment • VMM: Palacios • Host OS: Kitten • Guest OS: Kitten and Linux • Reasoning: • Relatively small code size • Familiarity with both

35. Code size

36. Symbiotic Examples • Demonstrating symbiotic virtualization • Symbiotic Swap • Symbiotic Device Drivers • Symbiotic Assists • Made possible by a symbiotic design

37. Virtualized Memory • VM memory model same as physical memory • Shadow/Nested paging designed to mimic • OS has memory set at boot time • Exceptions • Rare support for hot pluggable memory • Paravirtualized memory • Usually a large change to Guest OS • Swap storage allows over allocation • Can be exhausted • Can lead to thrashing

38. Current Swap Architectures • In Linux, swap storage is an array of pages • Easily accessible • When a page is swapped its given an index value • Points to array location • Page faults occur on page access • OS retrieves page and moves it to physical memory

39. Symbiotic Swap • Purpose: prevent thrashing situations • Temporarily expand memory • A symbiotic OS would expose swapped page map • VMM could find swapped page with minimal effort • Guest OS begins to thrash • Detected by VMM • Guest page is swapped out, but VMM copies it to free page • Shadow memory is altered to point to swapped page • Accesses no longer cause faults • Thrashing ends • VMM synchronizes swapped out page • Next access will fault the page back in to guest memory

40. Symbiotic Swap Architecture

41. Device Drivers • Guests often need direct device access • High performance networks • Driver included inside guest OS • Self-virtualization • Devices still require their own drivers • Not all devices are capable • Does not map well to virtual environments • Migration changes underlying hardware • Difficult to share between multiple VMs • VMM must fully trust guest driver

42. VPIO: Virtual Passthrough IO • Modeling-based approach to high performance I/O virtualization for commodity devices • Devices with no virtualization support • VMM runs Device Model Monitor (DMM) • Intercepts a subset of IO commands • Maintains model of internal device state • Transitions model state based on IO operations • Prevents security violations • Determines when device can be context switched L. Xia, J. Lange, and P. Dinda, Towards Virtual Passthrough I/O on Commodity Devices, Proceedings of the First Workshop on I/O Virtualization at OSDI

43. NE2k Device Model

44. Symbiotic Device Drivers • VMM provides passthrough driver to guest • Passthrough driver can include VPIO model • Design OS to allow driver injection • Guest OS no longer needs to include full set of drivers for all possible hardware • VMM can optimize driver behavior to the environment • Drivers can be dynamically swapped as conditions change • Passthrough network driver • Overlay network driver • Paravirtual driver

45. VMM Extended Services • VMMs perform operations on OS • Migration, suspend/resume, checkpointing • One sided approaches are often overly complex • VMM must account for OS behavior • Many have not been successfully implemented inside an OS • Ideally services are supported by both VMM and Guest OS • VMM and guest OS share responsibility • Each one does what is suitable to their environment

46. Symbiotic Assists • Possible Uses • Notifications • Guest OS is aware of VMM events • Optimizations • VMM can request guest optimize itself for an operation • Example: Migration • Allow guest OS to optimize itself for migration • Flush memory, freeze processes, disable devices • Pre/Post migration notifications • Possibly interrupt based • A non-symbiotic OS will still work • But won’t be optimized

47. Schedule

48. Contributions • Bridging the Semantic Gap • Automatic network reservations (VRESERVE) • Virtual network services (VTL) • Palacios • A new VMM architecture for HPC • Kitten • Lightweight HPC OS • Evaluation of virtualization in HPC at scale

49. Expected Contributions • Formal definition of Symbiotic Virtualization • Design of a set of symbiotic interfaces. • Implementation of Symbiotic Virtualization • Based on formal design • Implemented in Palacios • Linux and Kitten guests • Evaluation of the Symbiotic Virtualization • Raw performance • Complexity comparison

50. Expected Contributions • Example extensions • Symbiotic Swap • Guest OS thrashing detection • Symbiotic Device Drivers • Dynamic insertion of device drivers • Symbiotic Assists • Optimize VM operations inside guest OS