NoHype : Virtualized Cloud Infrastructure without the Virtualization

1. NoHype: Virtualized Cloud Infrastructure without the Virtualization Eric Keller, JakubSzefer, Jennifer Rexford, Ruby Lee Princeton University (ISCA 2010 + follow up soon to be “in submission”)

2. Virtualized Cloud Infrastructure • Run virtual machines on a hosted infrastructure • Benefits… • Economies of scale • Dynamically scale (pay for what you use)

3. Without the Virtualization • Virtualization used to share servers • Software layer running under each virtual machine Guest VM1 Guest VM2 Apps Apps OS OS Hypervisor servers Physical Hardware

4. Without the Virtualization • Virtualization used to share servers • Software layer running under each virtual machine • Malicious software can run on the same server • Attack hypervisor • Access/Obstruct other VMs Guest VM1 Guest VM2 Apps Apps OS OS Hypervisor servers Physical Hardware

5. Are these vulnerabilities imagined? • No headlines… doesn’t mean it’s not real • Not enticing enough to hackers yet?(small market size, lack of confidential data)

6. Are these vulnerabilities imagined? • No headlines… doesn’t mean it’s not real • Not enticing enough to hackers yet?(small market size, lack of confidential data) Large Attack Surface * 56 different exit reasons * Tremendous interaction Modest load => 20,000 exits/sec During boot => 600,000 exits/sec (Only VM, dedicated device, etc.) Guest VM1 Guest VM2 Apps Apps OS OS Hypervisor Physical Hardware

7. Are these vulnerabilities imagined? • No headlines… doesn’t mean it’s not real • Not enticing enough to hackers yet?(small market size, lack of confidential data) Large Attack Surface * 56 different exit reasons * Tremendous interaction Modest load => 20,000 exits/sec During boot => 600,000 exits/sec (Only VM, dedicated device, etc.) Guest VM1 Guest VM2 Apps Apps OS OS Complex Underlying Code * 100K lines of code in hypervisor * 600K++ lines of code in dom0 * Derived from existing OS Hypervisor Physical Hardware

8. NoHype • NoHype removes the hypervisor • There’s nothing to attack • Complete systems solution • Still retains the needs of a virtualized cloud infrastructure Guest VM1 Guest VM2 Apps Apps OS OS No hypervisor Physical Hardware

9. Virtualization in the Cloud • Why does a cloud infrastructure use virtualization? • To support dynamically starting/stopping VMs • To allow servers to be shared (multi-tenancy) • Do not need full power of modern hypervisors • Emulating diverse (potentially older) hardware • Maximizing server consolidation

10. Roles of the Hypervisor • Isolating/Emulating resources • CPU: Scheduling virtual machines • Memory: Managing memory • I/O: Emulating I/O devices • Networking • Managing virtual machines Push to HW / Pre-allocation Remove Push to side NoHype has a double meaning… “no hype”

11. Today Scheduling Virtual Machines • Scheduler called each time hypervisor runs(periodically, I/O events, etc.) • Chooses what to run next on given core • Balances load across cores VMs switch timer switch I/O timer switch hypervisor time

12. NoHype Dedicate a core to a single VM • Ride the multi-core trend • 1 core on 128-core device is ~0.8% of the processor • Cloud computing is pay-per-use • During high demand, spawn more VMs • During low demand, kill some VMs • Customer maximizing each VMs work, which minimizes opportunity for over-subscription

13. Today Managing Memory • Goal: system-wide optimal usage • i.e., maximize server consolidation • Hypervisor controls allocation of physical memory

14. NoHype Pre-allocate Memory • In cloud computing: charged per unit • e.g., VM with 2GB memory • Pre-allocate a fixed amount of memory • Memory is fixed and guaranteed • Guest VM manages its own physical memory(deciding what pages to swap to disk) • Processor support for enforcing: • allocation and bus utilization

15. Today Emulate I/O Devices • Guest sees virtual devices • Access to a device’s memory range traps to hypervisor • Hypervisor handles interrupts • Privileged VM emulates devices and performs I/O Guest VM1 Guest VM2 Priv. VM Device Emulation Apps Apps Real Drivers OS OS trap trap hypercall Hypervisor Physical Hardware

16. Today Emulate I/O Devices • Guest sees virtual devices • Access to a device’s memory range traps to hypervisor • Hypervisor handles interrupts • Privileged VM emulates devices and performs I/O Guest VM1 Guest VM2 Priv. VM Device Emulation Apps Apps Real Drivers OS OS trap trap hypercall Hypervisor Physical Hardware

17. NoHype Dedicate Devices to a VM • In cloud computing, only networking and storage • Static memory partitioning for enforcing access • Processor (for to device), IOMMU (for from device) Guest VM1 Guest VM2 Apps Apps OS OS Physical Hardware

18. NoHype Virtualize the Devices • Per-VM physical device doesn’t scale • Multiple queues on device • Multiple memory ranges mapping to different queues Network Card Classify MAC/PHY Processor Chipset Peripheral bus MUX Memory

19. Today Networking • Ethernet switches connect servers server server

20. Today Networking (in virtualized server) • Software Ethernet switches connect VMs Virtual server Virtual server Software Virtual switch

21. Today Networking (in virtualized server) • Software Ethernet switches connect VMs Guest VM2 Guest VM1 Apps Apps OS OS Hypervisor hypervisor

22. Today Networking (in virtualized server) • Software Ethernet switches connect VMs Guest VM2 Guest VM1 Priv. VM Apps Apps Software Switch OS OS Hypervisor

23. NoHype Do Networking in the Network • Co-located VMs communicate through software • Performance penalty for not co-located VMs • Special case in cloud computing • Artifact of going through hypervisor anyway • Instead: utilize hardware switches in the network • Modification to support hairpin turnaround

24. Removing the Hypervisor Summary • Scheduling virtual machines • One VM per core • Managing memory • Pre-allocate memory with processor support • Emulating I/O devices • Direct access to virtualized devices • Networking • Utilize hardware Ethernet switches • Managing virtual machines • Decouple the management from operation

25. NoHype Double Meaning Means no hypervisor, also means “no hype” • Multi-core processors • Extended Page Tables • SR-IOV and Directed I/O (VT-d) • Virtual Ethernet Port Aggregator (VEPA)

26. NoHype on Commodity Hardware Goal: semantics of today’s virtualization • xm create guest_01.cfg • xm shutdown guest_01 • Pre-allocate resources • Use only Virtualized I/O • Short circuit the discovery process • Unwind indirection

27. Pre-allocate Resources So a hypervisor doesn’t need to manage dynamically • CPU • Pin a VM to a core • Give complete control over that core (including per core timer and interrupt controller) • Memory • Utilize processor mechanism to partition memory • In Intel, EPT can be used for this

28. Use Only Virtualized I/O So a hypervisor doesn’t have to emulate • Network card: supports virtualization today • Disk: use network boot, iSCSI presence) Guest VM1 Priv. VM DHCP/ gPXE servers Loader/OS iSCSI servers hypervisor core core

29. Short Circuit System Discovery So a hypervisor doesn’t have to respond to queries (at run time) • Allow guest VM to do queries during boot up • Requires a temporary hypervisor • Modify guest OS to read this during initialization(save results for later) • Cloud provider supplies the kernel • For security purposes and functionality OS What are the processor’s features? What devices are there? What is the clock freq.? hypervisor

30. Unwind Indirection So a hypervisor doesn’t have to do mappings • Send IPI from core 0 to core 1 (actually core 2 to 3) • Interrupt vector 64 arrives at core 2(actually vector 77 of Guest 2) Guest 0 Guest 2 Guest 2 VMs can move VMs can share Apps Apps Apps OS OS OS VCPU0 VCPU0 VCPU1 Core 2 Core 3

31. Bring it together: Setup Guest VM1 Priv. VM xm e.g., Pre-set EPT, assign virtual devices Xen core core Guest VM space create loader kernel Customer code VMX Root

32. Bring it together: Network Boot Guest VM1 Priv. VM DHCP gPXE servers xm Xen core core Guest VM space create loader kernel Customer code VMX Root

33. Bring it together: OS Boot-up Guest VM1 Priv. VM xm System Discovery kernel Xen core core Guest VM space create loader kernel Customer code VMX Root

34. Bring it together: Switchover Guest VM1 Priv. VM xm Hypercall from kernel Before any user code (last command in initrd) kernel Xen core core Guest VM space create loader kernel Customer code VMX Root

35. Block All Hypervisor Access Guest VM1 Priv. VM iSCSI servers xm kernel Any VM Exit kills the VM Xen Kill VM core core Guest VM space create loader kernel Customer code VMX Root

36. Evaluation • Raw performance • Assess main limitations on today’s hardware: • Ability to send IPIs • Resource sharing (side channels)

37. Raw Performance About 1%performance improvement over Xen (VTd and EPT alleviate main bottlenecks)

38. IPI DoS Attack • Victim: SPEC (libquantum), Apache • Less than 1% performance degradation Attacker VM core core … core core Victim VM core

39. Memory Side Channel Information • Can attacker tell how loaded victim is?0%, 25%, 50%, 75%, 100%

40. Next Steps • Assess needs for future processors • e.g., receiver should know source of IPI (and can mask) • Assess OS modifications • e.g., push configuration instead of discovery • Asses vulnerabilities from outside • e.g., management channel from customer to start VM

41. Conclusions • Trend towards hosted and shared infrastructures • Significant security issue threatens adoption • NoHype solves this by removing the hypervisor • Performance improvement is a side benefit

42. Questions? Contact info: ekeller@princeton.edu http://www.princeton.edu/~ekeller szefer@princeton.edu http://www.princeton.edu/~szefer