1 / 13

A Vision for Virtualisation in WLCG

A Vision for Virtualisation in WLCG. Tony Cass HEPiX LBL, 27 th October 2009. Goals. Enable experiments/users to choose environment for job execution. Ensure sites have control/traceability over resource usage. Approach. Step-by-step: Build on established successes established trust

wren
Download Presentation

A Vision for Virtualisation in WLCG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Visionfor Virtualisationin WLCG Tony Cass HEPiX LBL, 27th October 2009

  2. Goals Enable experiments/users to choose environment for job execution. Ensure sites have control/traceability over resource usage. Virtualisation Vision- 2

  3. Approach • Step-by-step: Build on • established successes • established trust • But end goal in view. Prepare for this now with • technical agreements/developments • user behaviour (especially explicit statement of resource requirements) Virtualisation Vision- 3

  4. Approach • Five steps • Steps 1-3 • realistic • relatively uncontroversial(?) • achievable by end-2010? • Steps 4 & 5 • kite-flying • probably controversial • interesting Virtualisation Vision- 4

  5. Step 1 • Users can choose between virtual images created at sites. • Not really any different from now; could be rephrased “sites provide virtual machines for job execution, not real hardware”. • Key issue is (full) understanding of resource requirements • OS type, memory, (range of) #cores, ... Virtualisation Vision- 5

  6. Step 2 • Distribution of virtual machine images between sites (or from CERN...). • Image limited to minimalist operating system (SL4/5/6...) • Requires • transparent process for image generation guaranteeing content • mechanism for sites to hook into local monitoring and batch scheduling. • trusted and verifiable method of image distribution Virtualisation Vision- 6

  7. Step 3 • Distributed virtual image includes experiment software environment • So users can choose ATLAS version X on OS Y. • Requires “transparent process for image generation” to be extended to include experiment software. • Snapshot of experiment build servers at CERN? • Removes need for pilot jobs to verify (or create) correct environment. Virtualisation Vision- 7

  8. What about CernVM? • Instantiation of CernVM machines being discussed between IT and PH teams; could be an option at CERN. • But scalability and verifiability of CernVM distribution for widespread use as remote batch image is far from evident. • Not excluded, but more likely after successful experience with static images. Virtualisation Vision- 8

  9. Step 4 • Distributed virtual image includes client to connect directly to experiment pilot job framework (Dirac, PanDA). • Initially with virtual machine images instantiated according to jobs arriving at sites. • Later, sites instantiate virtual machines according to observed load and local policy • Lots of busy ATLAS machines? Start more... • Requires some way for pilot job frameworks to know (remaining) lifetime of virtual machine. • VM unlikely to be updated (security patches...), so lifetime will be limited. Virtualisation Vision- 9

  10. Step 5 • Experiment pilot job frameworks replaced by commercial/public domain schedulers. • Virtual LSF cluster for ATLAS • Virtual SGE cluster for CMS • ... • ... Virtualisation Vision- 10

  11. Questions? Comments?

  12. Not vision, but... • ALICE jobs, at least, require outgoing network connections. • CERN will provide public (routed) IP addresses for virtual machines. This option may not be available to all... • IP masquerading also tested at CERN • Internal traffic routed, external traffic NATted by the hypervisor. • Works, but no way to initiate remote connection to a virtual machine. • Should not be a problem based on previous statements. • Would be good if need for outgoing connections could be removed... • Or at least requirements documented per experiment. Virtualisation Vision- 12

  13. Questions? Comments?

More Related