Porting Xen to the ARM Architecture

1. Porting Xen to the ARM Architecture Dongyun Jin Michael LeMay Sundeep Reddy Brian Schoudel

2. Introduction • Xen is a hypervisor for para-virtualized operating systems • Xen currently supports IA32, IA32 PAE, IA64, and AMD64 architectures • The Xen hypervisor is heavily based on Linux, but can host any open-source or shared-source operating system that has been ported to Xen • Our project was simply to port the Xen hypervisor to the ARM architecture

3. Outline • Capabilities background • Motivation • System background and design • System construction • Demonstration • Results and contributions • Conclusion

4. Xen Capabilities • Strong isolation • Mandatory access control (sHype) • Mediated communication between VMs • Virtual trusted platform module (vTPM) http://www.tashian.com/perl/jpeglook.pl?berlin-wall-remnants.2

5. Motivation • How can Xen benefit embedded applications? • Provide isolation on PDAs and smartphones, to mitigate damage from viruses • Support advanced security functionality in pervasive meters: http://www.echelon.com/metering/default.htm

6. Pervasive Metering Capabilities • Real-time pricing • Electricity prices vary according to time of day, to encourage peak load reduction • Outage detection • Meter detects electricity outage and automatically reports it • Broadband Internet • 100Mbps broadband in California through meter! • Assisted living • Medical devices communicate over meter communication lines

7. Pervasive Metering Architecture

8. Pervasive Metering Problems • Privacy: Interval metering data from single meter on residence can tell which burner is powered up on stove, or whether waterbed is covered or uncovered! • Security: • Meter can cut off power to residence or business • Customers can tamper with data • Bandwidth usage: • Untrusted meters must transmit usage data frequently. Multiply by millions of meters = major networking issues. Hart 1989, Residential energy monitoring and computerized surveillance viautility power flows

9. Solution • Use isolation guarantees provided by Xen to prevent tampering with metering software • Isolation also permits multiple parties to share meters • Use remote attestation with vTPM to assure meter data management agency that trusted software is running • Software is trusted, so can perform billing locally and reduce bandwidth usage

10. Solution Architecture

11. Beginnings of System

12. TPMs in ARMs http://www.intel.com/design/pca/applicationsprocessors/whitepapers/30086801.pdf

13. System Background • We used several resources in our project: • ARM Linux 2.6.12 and 2.6.16 • ARM Software Packages (http://www.arm.com) • QEMU • CodeSourcery GCC cross-compiler (http://www.codesourcery.com) • Subversion software configuration manager • Doxygen (for documenting source code as HTML pages)

14. QEMU Review • QEMU Emulates ARM1026EJ-S processor • ARMv5 family • Full emulation, emulates ARM processor on most host machines, including IA32 and AMD64 • Great for testing esoteric OSs, too (We booted Plan9 on it for fun) • Emulates network card • Works great with NFS and other network clients • Quite slow, so not valuable for performance benchmarking

15. ARM Linux Review • Linux 2.6.12 kernel • Patched using patch from arm.com to use CodeSourcery cross-compiler GCC toolchain • Supports NFS • Doesn’t support VGA (limitation of QEMU) • Software packages built using special tool from arm.com

16. Xen Architecture

17. System Design • Based on Xen-unstable, checked out from Xen Mercurial repository • Based on pre-Linux 2.6.12 kernel • Supports IA32, IA32 PAE, AMD64, IA64 • Doesn’t include any support for ARM

18. Xen Source Layout • xen-unstable • buildconfigs: Makefiles • docs: Documentation • extras • mini-os: Minimal guest OS used for testing • linux-2.6-xen-sparse: Bare sources for dom0 and domU • patches: Patches for full Linux sources • tools: Large collection of tools to run on dom0 • xen: Hypervisor sources

19. Xen Source Layout (cont.) • xen • acm: sHype mandatory access control • arch: Architecture-specific implementation • ia64: Itanium sources • x86: IA32, IA32 PAE, and AMD64 sources • common: Architecture-independent sources • drivers: Common drivers for ACPI and console • include: Include files • tools: Figlet, for making block letter ASCII art, and other simple tools

20. Xen Initial Design • Xen is based heavily on Linux, but modified: • Task structure replaced with virtual CPU structure • Kernel initialization functions largely rewritten • Elegant kernel makefiles discarded and replaced with ugly, ad-hoc makefiles • Unneeded portions of Linux discarded • Many include directories have the same name, so you never quite know where your header files come from

21. Overall Xen SE Quality http://www.jansenkiener.com/Collapsed%20Barn.JPG

22. Our First Porting Approach • Insert Xen into Linux kernel: • Advantages: • Kernel already works, provides functional baseline • Kernel includes sophisticated makefiles and configuration system • Pitfalls: • Kernel headers too heavily modified to be compatible with Xen

23. Our Second Porting Approach • Insert parts of kernel into Xen: • Advantages: • Consistent with Xen design philosophy • Disadvantages: • Ugly makefiles • Headers not compatible with reintroduced kernel code • No operational baseline

24. Initial Modifications • We attempted to remain true to Xen’s original design philosophy, and extend it similarly to IA64 • Copied IA64-specific directories to form ARM-specific directories • IA64 is simpler and better-organized than x86 • Copied ARM-specific architectural implementation files from Linux 2.6.12 into xen/arch • Copied ARM headers from Linux into include/asm-arm • Modified Makefile rules to use CodeSourcery cross-compilers and pass proper flags

25. Initial Milestones • We disabled functionality in the architecture-specific Xen sources until all sources compiled and linked • System image didn’t boot in QEMU • QEMU provides remote interface to GDB, which we used for debugging • Determined that Xen entered tight infinite loop immediately upon entry

26. What Happened? • ARM bootloader doesn’t boot ELF images like GRUB • Needs raw compressed image piggybacked on standalone decompressor (zImage) • We wouldn’t have encountered this if Xen had retained Linux’s makefile system

27. Construction Phase #2 • Leveraged operational kernel we compiled earlier • Copied many architecture-dependent and architecture-independent directories from Linux 2.6.12 into Xen • Copied in parts of Linux makefiles to properly configure compiler options, etc. • Replicated ARM Linux zImage creation procedure

28. Secondary Milestones • Repeated compilation/linkage procedures • Crashed CodeSourcery linker (segfault) • Disabled long division to workaround crash • Finally, zImage booted on QEMU and produced a boot message! • It was an error • Attempts to fix the error induce linker segfault • Currently cooperating with CodeSourcery to resolve error

29. Demonstration • Boot partial hypervisor on QEMU

30. Results • Partially-operational hypervisor for ARM • Fully-operational, full-featured ARM Linux system image for QEMU • Plan for completing port

31. Personal Results • We gained a great deal of experience in this project: • Linux system initialization process variations between architectures • How to deal with linker scripts • Debugging virtual machines in QEMU with GDB

32. Conclusions • Good: • Support for virtualization with strong isolation and TPM functionality on the ARM will support critical new research directions • It is feasible to port Xen to the ARM, and all of the necessary pieces to do so are now in place. • Bad: • Xen is poorly engineered, but the best we have • The shaky design of Xen makes it very time-consuming to port to new architectures