1 / 24

Abstract Image Management and Universal Image Registration for Cloud and HPC Infrastructures

Explore FutureGrid Testbed's image management framework to deploy custom environments across cloud and high-performance computing infrastructures effortlessly.

jessemcpherson
Download Presentation

Abstract Image Management and Universal Image Registration for Cloud and HPC Infrastructures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Abstract Image Management and Universal Image Registration for Cloud and HPC Infrastructures Javier Diaz, Gregor von Laszewski, Fugang Wang and Geoffrey Fox Community Grids Lab Pervasive Technology Institute Indiana University https://portal.futuregrid.org

  2. Motivation • FutureGrid (FG)is a testbed providing users with grid, cloud, and high performance computing resources • One of the goals of FutureGrid is to provide a testbed to perform experiments in a reproducible way among different infrastructures • We need mechanism to ease the use of these infrastructures • FG Image Management frameworkallows users to easily create customized environments by placing suitable images onto the FG resources https://portal.futuregrid.org

  3. Introduction I • Image management is a key component in any modern compute infrastructure (virtualized or non-virtualized) • Processes part of the image management life-cycle: http://futuregrid.org

  4. Introduction II • Targeting multiple infrastructures amplifies the need for mechanisms to ease these image management processes • We have identified two mechanisms • Introduce standards and best practices to interface with the infrastructure (OVF, OCCI, Amazon EC2) • Provide tools that interface with these standards and expose the functionality to the users while hiding the underlying complexities • Otherwise, only the most experienced users will be able to manage images for multiple infrastructures under great investment of time https://portal.futuregrid.org

  5. FutureGrid Image Management Framework • Framework provides users with the tools needed to ease image management across infrastructures • Users choose the software stacks of their images and the infrastructure/s • Targets end-to-end workflow of the image life-cycle • Create, store, register and deploy images for both virtualized and non-virtualized resources in a transparent way • Allows users to have access to bare-metal provisioning (departure from typical HPC centers) • Users are not locked into a specific computational environment offered typically by HPC centers https://portal.futuregrid.org

  6. Architectural Overview https://portal.futuregrid.org

  7. Image Generation • Creates images according to user’s specifications: • OS type and version • Architecture • Software Packages • Software installation may be aided by Chef • Images are not aimed to any specific infrastructure • Image stored in Repository or returned to user https://portal.futuregrid.org

  8. Image Repository • Service to query, store, and update images • Unique interface to store various kind of images for different systems • Images are augmented with some metadata which is maintained in a searchable catalog • Keep data related with the usage to assist performance monitoring and accounting • Independent from the storage back-end. It supports a variety of them and new plugins can be easily created https://portal.futuregrid.org

  9. Image Metadata User Metadata

  10. Image Registration I • Adapts and registers images into specific infrastructures • Two main infrastructures types are considered to adapt the image: • HPC: Create network bootable images that can run in bare-metal machines (xCAT/Moab) • Cloud: Convert the images in VM disks and enable VM’s contextualization for the selected cloud https://portal.futuregrid.org

  11. Image Registration II • User specifies where to register the image • Optionally, user can select kernel from a catalog • Decides if an image is secure enough to be registered • The process of registering an image only needs to be done once per infrastructure https://portal.futuregrid.org

  12. Tests Results obtained from the Analysis of the Image Management Framework https://portal.futuregrid.org

  13. Methodology • Software deployed on the FutureGrid India cluster • Intel Xeon X5570 servers with 24GB of memory • Single drive 500GB with 7200RPMm 3Gb/s • Interconnection network of 1Gb Ethernet • Software Client is in India’s login node • Image Generation supported by OpenNebula • Image Repository supported by Cumulus (store images) and MongoDB (store metadata) • HPC supported by xCAT, Moab and Torque • Performed different tests to evaluate the Image Generation and the Image Registration tools https://portal.futuregrid.org

  14. Scalability of Image Generation I • Concurrent requests to create CentOS images from scratch • Increasing number of OpenNebula compute nodes to scale http://futuregrid.org

  15. Scalability of Image Generation II • Analyze how the time is spent within the image creation process • Only one OpenNebula compute node to better analyze the behavior of each step of the process • Concurrent requests to create CentOSand Ubuntu images • Image creation performed from scratch and reusing a base image from the repository https://portal.futuregrid.org

  16. Create Image from Scratch Takes Up to 69% CentOS Ubuntu Takes Up to 83% https://portal.futuregrid.org

  17. Create Image from Base Image Reduces 55-67% CentOS Reduces 62-80% Ubuntu https://portal.futuregrid.org

  18. Scalability of Image Registration • Register the same CentOS image in different infrastructures: • OpenStack(Cactus version configured with KVM hypervisor) • Eucalyptus (2.03 version configured with XEN hypervisor) • HPC(netboot image using xCAT and Moab) • Concurrent registrations in Eucalytpus and Openstack • Only one request at a time is allowed for HPC registration (modifies important parts of the HPC system) https://portal.futuregrid.org

  19. Register Images on Cloud Eucalyptus OpenStack http://futuregrid.org

  20. Register Image on HPC https://portal.futuregrid.org

  21. Conclusions I • We have introduced the FG user-controlled image management framework to handle images for different infrastructures • Framework abstracts the details of each underlying system • Users caneasily create and manage customized environments within FG • Replicate software stack on the supported cloud and bare-metal infrastructures https://portal.futuregrid.org

  22. Conclusions II • Image management results show a linear increase in response to concurrent requests • Image Generation • Create image from scratch in only 6 min and using a base image in less than 2 min • Scale by adding more nodes to the cloud • Support different OS and arch due to virtualization • Image Registration registers images in any supported infrastructure in less than 3 min • Image Repository supports perfectly the rest of the framework with a negligible overhead https://portal.futuregrid.org

  23. Ongoing Work • Integrate a messaging queue system (like RabbitMQ or ZeroMQ) to process user’s requests in an asynchronous way • Develop a portal interface • On-demand resource re-allocation between infrastructures (usage, user’s requests) https://portal.futuregrid.org

  24. Thank for your attention!! Contact info: Javier Diaz: javier.diazmontes@gmail.com Gregor Laszewski:laszewski@gmail.com http://futuregrid.github.com/rain/ https://portal.futuregrid.org https://portal.futuregrid.org

More Related