1 / 24

Virtual Node bootstrapper (VNB)

Explore the innovative Virtual Node Bootstrapper to dynamically deploy virtual platforms on-demand, enhancing ETICS build and test infrastructure. Achieve increased availability, reproducible environments, and efficient resource utilization. Discover achievements, lessons learned, and future work.

derrickbaum
Download Presentation

Virtual Node bootstrapper (VNB)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtual Node bootstrapper (VNB) An On-Demand Dynamic Virtualization Manager Øyvind Valen-Sendstad CERN – IT/GD, ETICS Oeyvind.valen-sendstad@cern.ch

  2. Contents • Brief introduction to ETICS • ETICS build infrastructure • Improving the build/test process • The Virtual Node Bootstrapper • Achievements • Status, Lessons learned and Further work • Summary

  3. What ETICS is • It’s a software engineering management system • It’s a build and test infrastructure • It provides tools and resources to configure, manage and analyse build and test runs • It provides a common interface to diverse projects to facilitate knowledge sharing and operations management • It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information • It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions • It’s multi-platform and independent from any specific build or test tool

  4. ArchitectureETICS is not ‘just’ a build system

  5. ETICS build infrastructure - static platforms Problem 1: The composition of the pool is fixed. If we have 20 SL5 nodes all busy and a new request for SL5 arrives, it is queued even if many nodes are available with other platforms X 5 SLC4 X 5 Debian X 5 SL5 X 5 Ubuntu 7 Virtual static nodes New SL5 Job X 5 SL5 / 64 X 5 Mac OSX Physical node X 5 SL5 / 32 X 5 SL5 / 64

  6. ETICS build infrastructure - static platforms Problem 2: If a platform is very rarely used, but we want to make it available, one or more nodes will do nothing most of the time X 5 SLC4 X 5 Debian X 5 SL5 X 5 Ubuntu 7 Virtual static nodes New RH7 Job X 5 SL5 / 64 X 5 Mac OSX Physical node X 5 SL5 / 32 X 5 RH7

  7. The proposed solution: dynamic virtual platforms X 5 X 5 Available Available VM repository X 5 X 5 Available Available New job Virtual machine X 5 X 5 Available Available X 5 X 5 Available Available X Physical machine

  8. The bootstrap process • User submits a build/test for a specific platform • Request arrives to VNB • VNB determines platform and hypervisor support • Downloads a suitable VM from repository (or local cache) • Deploys the VM • Executes job • Returns results • Cleans up

  9. Deploying the bootstrapper

  10. Deploying the bootstrapper

  11. Deploying the bootstrapper

  12. Deploying the virtual machine

  13. Deploying the virtual machine

  14. Deploying multiple virtual machines on the same node

  15. Deploying multiple virtual machines on the same node

  16. Deploying multiple virtual machines on the same node

  17. Achievements • Perform platform deployment on demand • Increase availability • Increase capability • Provide reproducible environments • Reduce maintenance • New releases • Upgrades • Security patches • Accommodate for specific software dependencies • Offer privileged access (root access to the VM, not the host) • Enable post build analysis (VM snapshot can be saved with results) • Virtual machine image customization (users can provide their own)

  18. Status • Working • Bootstrap process, tested on VMware and KVM • Remaining • Testing on Xen • Support for NT-based operating systems • Caching VM images locally

  19. Lessons learned • Privileges • The bootstrapper runs in the user-space and need privileges to use Xen and Vmware • Compatibility • The kernel in a virtual machine created in VMware is not necessarily able to run in a Xen or KVM environment

  20. Further work • Deploy virtual machine image into a folder and chroot in order to run natively on a worker node • Update VNB on VM during startup • Support for NT-based operating systems

  21. Open issues • Reproducibility • Different implementations of the virtual machine hardware can cause different behaviour of a virtual machine • Firewall • Ephemeral port range can be blocked by strict firewalls

  22. Summary • The VNB gives • Better service availability • Better resource utilization • The VNB has • Generic design • Emulates when no hypervisors are present • The VNB allows • Running multiple bootstrappers on a single host

More Related