Disco: Running Commodity Operating Systems on Scalable Multiprocessors

1. Disco: Running Commodity Operating Systems on Scalable Multiprocessors Presented by: Pierre LaBorde, Jordan Deveroux, Imran Ali, Yazen Ghannam, Tzu-Wei Kuo Paper by: Edouard Bugnion, Scott Devine, Kinshuk Govil, Mendel Rosenblum

2. Introduction Pierre LaBorde

3. CC-NUMA • Cache-Coherent Non-Uniform Memory Access • Coupling with standard distributed protocols • TCP/IP • NFS •  Global Buffer Cache Introduction

4. Hide NUMA-ness • Page placement • Dynamic page migration • Dynamic page replication  Introduction

5. Operating systems for innovative hardware • Scalable shared memory multiprocessors • Significant changes required • OS typically have millions of lines of code Problem

6. Solution

7. Instead of modifying existing OS • Additional layer of software between hardware and OS • Multiple copies of existing operating systems • Support a variety of workloads • Virtualizes all of the resources • Exports conventional hardware interface  • Schedules virtual resources on the physical • Processor • Memory Virtual Machine Monitors

8. Monitor and distributed protocols need to scale • Simplicity of the monitor • Fault-containment • NUMA memory management issues • Global policies • Fine-grained resource sharing Virtual Machine Monitor

9. Overheads • Privileged instructions • I/O Devices • Resource Management • Instruction execution stream • Idle loop • Lock busy-waiting  • Communication and Sharing • Virtual disk Challenges

10. Disco: A Virtual Machine Monitor Jordan Deveroux

11. Processors • Abstraction of MIPS R10000 processor • Does not  support complete virtualization of kernel virtual address space • Extends architecture to support efficient access to some processor functions • Physical Memory • Abstraction of main memory that resides in contiguous physical address space  • Uses dynamic page migration and replication  to export nearly uniform memory architecture to the software • I/O Devices • Each virtual machine has specified set of I/O devices • Intercepts communication from all of it's I/O devices for translation or emulation • Virtualizes access to the networking devices of the underlying system Disco's Interface

12. Multithreaded, shared memory program • Disco vs. Other Systems • NUMA memory placement • cache-aware data structures • interprocessor communication patterns • NUMA memory management • Copy DISCO into all memories of FLASH machine • Cache-aware data structures • Partitioned so that parts accessed only by a certain processor are in memory near that processor • Interprocessor communication patterns • Very few locks • Wait-free synchronization Implementing Disco

13. Emulates virtual CPU's by using direct execution of real CPU's • Same execution speed as running on real CPU's • Each virtual CPU has a data structure like a process table entry in traditional O.S. • Contains state of virtual CPU • Runs in kernel mode with full access • Simple scheduler allows virtual processors to be shared Implementing Disco: Virtual CPU's

14. Add a level of address translation and maintains physical-to-machine address mappings • Translation performed using translation-lookaside buffer •  Memory references are translated through this mapping from now on • Each TLB entry is marked with an address space identifier to avoiding the flushing the TLB on context switches  • Each miss is more expensive • emulation of trap architecture • emulation of privileged instructions • remapping of physical addresses Implementing Disco: Virtual Physical Memory

15. Optimization that enhances data locality • Fast translation of virtual-to-physical addresses • Allocation of real memory to virtual machines • Only moves pages that will have performance benefit • Contains a memmap data structure with an entry for each real machine memory page Implementing Disco: NUMA Memory Management

16. Two different virtual processors of the same virtual machine logically read-share the same physical page, but each virtual processor accesses a local copy

17. Intercepts all device access from the virtual machine and forwards them to the physical devices • Each disco device defines a monitor call used by the device driver to pass all command arguements • Disks and network interfaces include a map as part of their arguements • list of address pairs that specify the source and destination of I/O operations Implementing Disco: Virtual I/O

18. VM Sharing Imran Ali

19. Uses Virtual Memory Addressing to Map Data to physical Memory • Multiple Virtual Machines(VM) Share Machine Memory • Copy on write means that VM is unaware of Machine Memory being shared Copy-on-Write Disks

20. VM Sharing Pages

21. Virtual Machines are not allowed to communicate with each other • Uses Standard Protocols to communicate through Ethernet- type addressing • All read only pages can be shared through virtual machines reducing memory overhead • Pages are shared whenever possible and are replicated when needed to improve proformance Virtual Network Interfaces

22. Transparent Sharing of Pages

23. Experimental Results Yazen Ghannam

24. Experimental Setup Experiments are Simulated, not using real hardware Used four different workloads Software Development (Pmake) OS, I/O Intensive Hardware Development (Engineering) OS light; Large memory footprint Scientific Computing (Raytrace, Radix) OS light; uses shared memory regions Commercial Database I/O light; Single memory intensive

25. Execution Overheads

26. Memory Overheads

27. Scalability

28. Page Migration and Replication

29. Experiences and Related Work Tzu-Wei Kuo

30. Disco was ported to run on a real hardware in order to confirm the simulation test results  • Run on SGI Origin200 board which forms the basis of the FLASH machine • Single - 180MHz MIPS R10000 processor • 128MB of memory Experiences on Real Hardware

31. Overheads of Virtualization • Two workloads • Pmake: compiles Disco itself using the SGI development tools, two files at a time • Engineering: simulates the memory system of the FLASH machine Experiences on Real Hardware

32. Experiences on Real Hardware • This table shows a breakdown of the execution time for the two workloads and a comparison between IRIX and Disco running IRIX. The execution time is broken down into the user, system, and idle time. 

33. System Software for Scalable Shared Memory Machines • Virtual Machine Monitors • Other System Software Structuring Techniques • CC-NUMA Memory Management Related Work

34. Developing system software for scalable shared memory multiprocessors without massive development effort • Experimental results shows that the overhead of virtualization is modest in both processing time and memory footprints • Disco provides simple solution for scalability and reliability • Lower implementation cost Conclusion

35. Disco: Running Commodity Operating Systems on Scalable Multiprocessors Presented by: Pierre LaBorde, Jordan Deveroux, Imran Ali, Yazen Ghannam, Tzu-Wei Kuo Paper by: Edouard Bugnion, Scott Devine, Kinshuk Govil, Mendel Rosenblum

36. Text Title