Virtualization

1. Virtualization Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

2. What is virtualization? “The construction of an isomorphism between a guest system and a host” [Popek, Goldberg, ’74] Creating a virtual version of something • Hardware, operating system, application, network, memory, storage

3. Example: virtual disk • Partition a single hard disk to multiple virtual disks • Virtual disk has virtual tracks & sectors • Implement virtual disk by file • Map between virtual disk and real disk contents • Virtual disk write/read mapped to file write/read in host system

4. What is virtualization? (continued) Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili A way to run multiple operating systems (and their applications) on the same hardware (virtual machines) Only virtual machine manager (a.k.a. hypervisor) has full system control Virtual machines completely isolated from each other (or so we hope)

5. Basic concepts Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili Virtual Machine (VM) Host Guest Hypervisor (type ||) / Virtual Machine Monitor

6. Basic concepts Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili Virtual Machine (VM) Host Guest Hypervisor (type ||) / Virtual Machine Monitor

7. Basic concepts Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili Virtual Machine (VM) Host Guest Hypervisor (type ||) / Virtual Machine Monitor

8. Basic concepts Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili Virtual Machine (VM) Host Guest Hypervisor (type ||) / Virtual Machine Monitor

9. Types of virtualization Our focus is on full virtualization Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili Full virtualization – guest OS runs unmodified Para-virtualization – guest OS must be aware of virtualization, source-code modifications required Hardware virtualization support may be used for both

10. Virtualization advantages Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Cost-effectiveness – less hardware • Multiple virtual machines / operating systems / services on single physical machine (server consolidation) • Various forms of computation as a service • Isolation • Good for security • Great for reliability and recovery: If VM crashes it can be rebooted, does not affect other services (fault containment) • VM migration • Development tool • Work on multiple OS in parallel • Develop and debug OS in user mode • Origins of VMware as a tool for developers

11. Virtualization vs. Multi-Processing Process1 Process2 ∙∙∙ User space/ kernel separation OS HW interface HW (disk, NIC,…) Multi-processing Pr1 ∙∙∙ Pr2 Pr1 Pr2 VM OS1 OS2 ∙∙∙ Virtual HW interface VMM/Hypervisor Virtualization Real HW interface HW (disk, NIC,…) Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

12. Type 1 and type 2 hypervisors VMware ESX, Microsoft Hyper-V, Xen VMware Workstation, Microsoft Virtual PC, Sun VirtualBox, QEMU, KVM Figure 7-1. Location of type 1 and type 2 hypervisors. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

13. Type 1 and type 2 hypervisors (continued) Figure 7-2. Examples of the various combinations of virtualization type and hypervisor. Type 1 hypervisors always run on the bare metal whereas type 2 hypervisors use the services of an existing host operating system. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

14. What's required of a (classic) hypervisor • Hypervisor should provide the following: • Safety: have full control of virtualized resources • Fidelity: program behavior on VM should be identical to its behavior on bare hardware • Efficiency: As much as possible, run directly on hardware without hypervisor intervention • Full interpretation isn't efficient Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

15. Classic virtualization: trap and emulate VM1 VM2 Return to process (3) VMM HW emulation HW Interrupt handler (2) Trap (1) Emulation is the process of implementing the functionality/interface of one system on a system having different functionality/interface

16. Trap and emulate: difficulties on x86 • Sensitive instructions: Provide control over HW resources  behave differently in kernel/supervisor and user modes • I/O instructions, enable/disable interrupts, access CR3 register… • Privileged instructions: cause a trap if executed in user mode Theorem [Popek and Goldberg, 1974] A machine can be virtualized [using trap and emulate] if every sensitive instruction is privileged. Not supported by x86 processors prior to 2005 In 2005, Intel/AMD introduced virtualization HW support. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

17. What is sensitive? Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • CPU – some registers • MMU • Page table • Segments • Interrupts • Timers • IO devices

18. X86 virtualization problem I Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • The x86 architecture (w/o virtualization extensions) can't be virtualized by trap and emulate. • Some sensitive instructions are not privileged. • Example: the popf instruction • Pops 16 bits from stack to flags register • One of the flags masks (i.e. disables) interrupts • The instruction is not privileged • What happens if the OS of a VM runs popf?

19. X86 virtualization problem II Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Some instructions: push, pop, mov can have code segment selectors (cs, ds, ss) as arguments even in user mode, so they can be read • The selectors have two bits that are their current privilege level • In x86 (beginning with 386), four privilege levels (ring 0 to ring 3) • The two lower bits of the cs register are the Current Privilege Level (CPL) of the code. • Guest OS thinks that it is in ring 0. • Guest OS is actually in ring 1 • Result - guest OS confusion.

20. Implementation options Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Avoid executing sensitive instructions • Interpretation (BOCHS, JSLinux). • Binary translation – change executed code (VMware, QEMU). • Para-virtualization – re-compile guest OS (XEN, Denali). • Hardware assistance– Intel VT-x and AMD-V (used by KVM, XEN, Vmware).

21. Outline • Concepts, classical CPU virtualization • Binary translation • Memory virtualization Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

22. Binary translation Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Binary translation is the process of translating one instruction set to another one. • Approach I: translate entire OS when loaded to VM • Key problem – indirect control flow

23. Dynamic binary translation Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Approach II: translate code on the fly • Simplest approach • Keep table mapping old instructions to new instructions. • Fetch old instruction. • Use table to translate. • Execute new instruction(s). • Problem: performance • Overhead for every instruction similarly to interpretation.

24. Dynamic BT with caching Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Cache translated code region: • After translation run from cache. • Translation occurs only once. • Static translation cannot handle dynamic control transfer, when: • Jump depending on content of memory address. • Indirect function call (by function pointer). • Translation of dynamic control transfer must be done at execution time. • User code does not have to be translated

25. Virtualization prior to HW support Figure 7-4. The binary translation rewrites the guest operating system running in ring 1, while the hypervisor runs in ring 0 Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

26. VMWare binary translation: example C code 64-bit binary Invoking isPrime(49), logging all code translated Binary (hex) representation Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

27. VMWare binary translation: example First TU Compiled code fragment (CCF) Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Translator reads guest memory at the address indicated by guest PC • Decodes instructions, creates Intermediate Representation - IR objects • Accumulates IR objects to translation units (TUs) • Basic blocks (BB), stops upon control flow

28. VMWare binary translation: example Identical code First TU Compiled code fragment (CCF) Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Translator reads guest memory at the address indicated by guest PC • Decodes instructions, creates Intermediate Representation - IR objects • Accumulates IR objects to translation units (TUs) • Basic blocks (BB), stops upon control flow

29. VMWare binary translation: example Translation of jump BB First TU Compiled code fragment (CCF) Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Translator reads guest memory at the address indicated by guest PC • Decodes instructions, creates Intermediate Representation - IR objects • Accumulates IR objects to translation units (TUs) • Basic blocks (BB), stops upon control flow

30. VMWare binary translation: example Translation of fall through BB First TU Compiled code fragment (CCF) Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili • Translator reads guest memory at the address indicated by guest PC • Parses instructions, creates Intermediate Representation - IR objects • Accumulates IR objects to translation units (TUs) • Basic blocks (BB), stops upon control flow

31. VMWare binary translation: example C code 64-bit binary Invoking isPrime(49), logging all code translated Which basic block will be translated next? Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

32. VMWare binary translation: example C code 64-bit binary Invoking isPrime(49), logging all code translated Which basic block will be translated next? Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

33. VMWare binary translation: example C code 64-bit binary Invoking isPrime(49), logging all code translated Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

34. VMWare binary translation example: output

35. VMWare binary translation example: output These continuations remain becauserespective basic blocks were not executed

36. VMWare binary translation operation Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili Translation cache (TC) stores translations done so far A hash table tracks the input-to-output correspondence Chaining optimization allows one CCF to jump directly to another without calling out of the translation cache As TC gradually captures guest's working set, proportion of translation decreases User code does not have to be translated

37. Dealing with privileged instructions: example Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili The cli (clear interrupts) instruction is privileged Translated to: “vcpu.flags.IP=0” Much faster than source binary!

38. Outline • Concepts, classical CPU virtualization • Binary translation • Memory virtualization Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

39. Memory allocation • Each VM usually receives a contiguous set of physical addresses. • 1 Gbyte– 4 Gbyte are typical values. • As far as VM is concerned, this is the physical memory of the machine. • The guest OS allocates pages to guest processes.

40. Memory management • Assumptions of OS in VM: • Physical memory is a contiguous block of addresses from 0 to some n. • OS can map any virtual page to any page frame. • Hypervisor must: • Partition memory among VMs. • Ensure virtual page mapping only to assigned page frames. • TLB miss: cache miss in HW-managed TLB (e.g. x86) causes HW to select a page from page table. • VM OS must not manage real page table. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

41. Option 1: brute force Define these pages as not R/W Guest OS Hypervisor Page table VMM SW VM memory layout Page dir. Interrupt & VMM corrects address. TLB CPU CR3 HW Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

42. Brute force – description • Guest page tables are read and write protected in host system. • If guest OS reads page table (e.g. for page eviction), writes page table (e.g. after page fault), or changes CR3, the system traps. • The hypervisor then uses a VM memory layout to: • Return answers to VM • Update the layout • Hypervisor switches VM memory layout when new VM is scheduled. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

43. Option 2: shadow page tables Guest OS Hypervisor Page table VMM SW Shadow page table Page dir. Interrupt & VMM corrects page table. G-CR3 TLB CPU CR3 HW Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

44. Shadow page tables – description • Hypervisor maintains “shadow page tables”. • Guest page tables map: Guest VA (GVA) Guest PA (GPA) • Shadow tables map: Guest VA Host PA (HPA). • Hypervisor does not trap guest updates to its page table. • Result – inconsistent guest page table and shadow page table. • When guest process accesses virtual address • The physical address is not in the guest page table, but in the shadow page table. • HW translates correctly, because it is aware only of shadow tables. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

45. Shadow page tables – description (continued) • If address in TLB – TLB hit and no problem. • When guest process causes a page fault • Hypervisor begins execution. • If required, hypervisor updates shadow page table. • Performance is as good as native execution as long as there are no page faults. • Shadow page tables should be cached so that once a VM is re-scheduled the page table does not have to be rebuilt from scratch. Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

46. Shadow page tables – page faults (continued) Two scenarios when handling a page fault. Hypervisor ``walks’’ guest page table to determine which it is. • Guest page fault – No translation in guest page tables ``inject’’ page fault for guest to handle • Guest translation found  update shadow table respectively Operating Systems, Spring 2018, I. Dinur, D. Hendler and R. Iakobashvili

47. Shadow page tables – updating CR3 Virtual CR3 Guest Page Table Shadow Page Table Guest Page Table Guest Page Table Shadow Page Table Shadow Page Table Real CR3 Slide taken from a presentation by VMWare.

48. Shadow page tables – updating CR3 Virtual CR3 Guest Page Table Shadow Page Table Guest Page Table Guest Page Table Shadow Page Table Shadow Page Table Real CR3 Slide taken from a presentation by VMWare.

49. Shadow page tables – updating CR3 Virtual CR3 Guest Page Table Shadow Page Table Guest Page Table Guest Page Table Shadow Page Table Shadow Page Table Real CR3 Slide taken from a presentation by VMWare.

50. Undiscovered guest page table Virtual CR3 Guest Page Table Shadow Page Table Guest Page Table Guest Page Table Shadow Page Table Shadow Page Table Guest Page Table Real CR3 Slide taken from a presentation by VMWare.