The Denali Project

The Denali Project Andrew Whitaker, Marianne Shaw, Rick Cox, Steven D. Gribble http://denali.cs.washington.edu {andrew,rick,mar,gribble}@cs.washington.edu

A Brief History of VMMs • Introduced by IBM in the late 1960’s: • CP/67 (1966) • VM/370 (1972) • z/VM (2004) • Found wide use as a time-sharing system system • Ideological competitor to Multics, UNIX • Today: VMMs have enjoyed a popularity resurgence • Industry: VMWare, Virtual PC (Microsoft), IBM • Academia: Disco, Denali, ReVirt, Xen

… VM VM VM VM virtual arch = VMM phys arch hardware Virtualizable Architectures • Previous systems focused on strict virtualizability [Goldberg, 74] • Goal: minimize OS changes • VMM strictly emulates the phys. hardware

The Denali Philosophy • Key insight: software is not hardware • We can modify the VMM to achieve other goals • Two ways to exploit change: • Modifying the virtual architecture [OSDI 2002] • Changing the VMM implementation [NSDI 2004]

Outline • Virtual Machine Monitor Background • Denali and Architectural Changes • Motivation and approach • Architecture changes • Results • Denali and Implementation Changes

Motivation • Growing importance of untrusted code: • Internet services: ASP model, service authors upload code into hosting center • PlanetLab: “slice” distributed platforms amongst competing experiments, apps • Your desktop: Java{script}, untrusted binaries, email attachments • Needed: strong isolation platform • must be secure, efficient, and extensible

Why we need a new system • Previous platforms are are inadequate: • Java sandboxes, ActiveX controls, system call monitoring, virtual private servers, enhanced reference monitors • Why? • require complex security policy • complexity, wide OS API • performance and/or scaling bottlenecks

Basic approach • Focus on Internet services • Little sharing required • Straw man: one service per physical machine • not cost-effective for many services • Instead, use virtual machines to emulate separate hardware Web server Game server Overlay router VMM hardware

VMMs provide strong isolation • namespace isolation • no sharing possible, except through network • eliminates security policy problems • type-I VMMs run on bare hardware • no layer-below vulnerabilities, unfettered resource management ability • no high-level abstractions • eliminates complexity, reduces width of API

But, VMMs do too much! • VMMs support isolation and full HW emulation • though some VMMs are moving away from this… • Isolation kernels separate the goals of isolation and emulation • make a clean break from the underlying phys arch.

… VM VM VM VM virtual arch  Denali phys arch hardware The Denali Virtual Architecture • Retain x86 instruction set for performance • Change other aspects to gain simplicity, scale, performance • gives up legacy OS support

Arch. Changes for Simplicity • Real hardware is unnecessarily complex • Denali “scrubs” the hardware architecture: • Omit rarely used features • Revise other features (interrupts, MMU) • Streamlined hardware devices • Hardware Ethernet drivers make 12 PIOs per send • Denali’s Ethernet requires single PIO

Source Code Complexity a) total lines b) core kernel breakdown Denali’s core kernel is roughly an order-of-magnitude smaller than bare-bones Linux 2.4.16

Arch. Change for Scale • Denali exposes an idle-with-timeout instruction • Avoids a 50% throughput reduction for many VMs

virtual interrupts 4 4 scheduled VM VM 1 VM 2 VM 3 VM 4 4 4 4 4 physical interrupts Arch. Change for Performance • Denali uses asynchronous, batched interrupt delivery • avoiding context switches results in a 30% performance improvement time

Experimental Setup • Hardware • Intel Pentium 4 CPU; 1 GB RAM • 3 IDE disks • Intel PRO/1000 gigabit Ethernet NIC • Software: apps run on Ilwaco guest OS • includes threads package, TCP/IP stack, libc • doesn’t use Denali MMU • apps linked against Ilwaco like a libOS

Ethernet Virtualization Overhead Breakdown of UDP receive costs (100 byte , 1400 byte packets) • Virtualization overhead is less than 20% VM’s device driver TCP Stack read( ) 1% , 1% 37% , 42% total cost (cycles): 43477, 49002 user/kernel crossing 3% , 2% Ethernet packet arrival VNIC demux Ethernet driver packet copy 12% , 13% 5% , 4% 43% , 39%

Web Server Performance • BSD: one kernel crossing per syscall • Denali: one kernel crossing per packet

Swapping in Denali • Mechanism: demand-loading of “virtual” physical memory from statically allocated swap • Factors that influence performance: • # disks • memory size • popularity distribution • block size • data locality • striping policy

Scale in Denali: 10,000 Web Servers In memory performance Disk-bound performance memory exceeded disk paging bandwidth • Denali utilizes most of the available hardware resources

The VMM as a Service Platform • Adding services to a VMM: • Migration (Intel Research Pittsburgh, Stanford) • Intrusion detection (Stanford) • System replay (Michigan) • Fault Tolerance (Cornell) • Advantages: • Virtual devices are relatively simple • VMM has a “whole-system” perspective

Denali: An Extensible VMM • Problem: existing VMMs are hard to extend • New services require significant re-implementation or reverse-engineering • Denali approach: • Factor out VMM design decisions (disk, swap, etc.) • “Parent” virtual machines can implement functionality on behalf of their “children” • Encapsulate these decisions with clean interfaces • Supply default implementations for commonly used components

A Denali Interface // the virtual Disk device callback functions struct Disk { int (*diskWrite)(char *buffer, int offset, int num_sectors); int (*diskRead)(char *buffer, int offset, int num_sectors); int (*getSizeInSectors)(void); int (*getSectorSize)(void); } Disk;

Migrating an Apache Web Server MachineB MachineA Migration Implemented in only 289 lines of C source code

Conclusions • Denali exploits the degree-of-freedom offered by software to enhance VMMs • Architecture changes for scale, performance, simplicity • Implementation changes for providing virtual machine services

The Denali Project