1 / 32

CernVM News

CernVM News. Predrag Buncic (CERN/PH-SFT). CernVM Project. Approved in 2007 as R&D activity under extra funding (“White Paper”) Portable Analysis Environment using Virtualization Technology (WP9) Led by SFT Group in CERN PH Department Started in January 2008 Funding for 4 (2+2) years

dyre
Download Presentation

CernVM News

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CernVM News Predrag Buncic (CERN/PH-SFT)

  2. CernVM Project • Approved in 2007 as R&D activity under extra funding (“White Paper”) • Portable Analysis Environment using Virtualization Technology (WP9) • Led by SFT Group in CERN PH Department • Started in January 2008 • Funding for 4 (2+2) years • Fellow, student + 0.5 staff CERN, 18/06/2008 - 2

  3. Motivation Emerging technologies Multi and many core CPUs Making laptop/desktop ever more powerful and underutilised Hardware support for virtualization Minimal performance penalty for application running in Virtual Machine Virtualization software is quickly becoming integral part of OS or given as a free add-on KVM, Xen, Vmware Player & Server, Sun’s VirtualBox Old problems Millions of lines of code Complicated software installation/update/configuration procedure, different from experiment to experiment Only tiny portion of it is really used at runtime in most of the cases Often incompatible or lagging behind OS version on desktop/laptop CERN, 18/06/2008 - 3

  4. Project Goals Use virtualization and extra cores to get extra comfort Hoping that this will come at acceptable cost in terms of performance Decouple application lifecycle from evolution of system infrastructure Close to zero effort to install, maintain and keep up to date the experiment software Reduce the cost of software development by reducing the number of compiler-platform combinations CERN, 18/06/2008 - 4

  5. How do we want to do that? Build a “thin” Virtual Software Appliance for use by LHC experiments This appliance should provide a complete, portable and easy to configure user environment for developing and running LHC data analysis locally and on the Grid be independent of physical software and hardware platform (Linux, Windows, MacOS) CERN, 18/06/2008 - 5

  6. Key Building Blocks • rPath Linux 1 (www.rpath.org) • Slim Linux OS binary compatible with RH/SLC4 • rAA - rPath Linux Appliance Agent • Web user interface • XMLRPC API • CVMFS - CernVM file system • Read only file system optimized for software distribution • Aggressive caching • Operational in offline mode • For as long as you stay within the cache • rBulder • Build VM images for various platforms CERN, 18/06/2008 - 6

  7. CernVM component group-<experiment>-desktop Lightweight X working environemnt group-<experiment> groups and extra packages required by experiment group-cernvm-devel (development tools) group-slc4 (SLC4 compatibility libs) compat-db4 compat-openssl compat-linstdc++slc3 compat-libxml2 compat-readline compat-tcl compat-tk group-cernvm (core packages) 100 MB CERN, 18/06/2008 - 7

  8. r A A rAA plugin Extra Liibs & Apps F I L E S Y S T E M JeOS (based on rPath Linux) KERNEL fuse module “Thin” Software Appliance H T T P D LAN/WAN (HTTP) Software Repository Cache 0.1 GB 1 GB 10 GB CERN, 18/06/2008 - 8

  9. CernVM File System On same host: On File (Web) Server /opt/<experiment>-> /mnt/cmfs/<experiment> /opt/<experiment> USERS App CernVM Fuse RSYNC HTTP open(“/opt/lcg”) /opt/alice /opt/alice /opt/alice /opt/alice Kernel Cache NFS LFS FUSE V.O. ADMIN CERN, 18/06/2008 - 9

  10. Web UI CERN, 18/06/2008 - 10

  11. XML-RPC API import xmlrpclib import os url = 'http://' + user + ':' + password + '@' + host + ':8004/rAA/xmlrpc' server = xmlrpclib.ServerProxy(url) r = server.cernvm.Config.configUpdate('http://cern.ch/cernvm/config', 'ALICE', # (can be None,ATLAS, ALICE, LHCB, CMS) 'off', # (on/off - if on it will make cvmfs locally writeable) 'off', # (on/off - enable/disable grid UI) 'off') # (on/off - enable VO profile) print r["message"] #### # Here we create initial user 'test' with the same password as Web admin #### r = server.cernvm.User.userUpdate('test', password,password) print r["message"] #### # Select grid UI version #### r = server.cernvm.Config.configGridUIVersion("") print r["message"] CERN, 18/06/2008 - 11

  12. Developing CernVM… • 2-4 developers • Monthly releases • Bi-weekly meetings with experiments • Good collaboration with experiments • ATLAS, ALICE, LHCb • CMS is joining now • Initial feedback from the experts • From version 1.0 (end of October) it will be offered to wider public • Thanks to Yves for constructive comments • Hope to get soon more input from ALICE CERN, 18/06/2008 - 12

  13. CernVM Release Timeline JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC -Preparation Release 0.5 Kickoff Workshop Release 1.0 Scheduled for end of October - Release 0.6 - Release 0.7 - Release 0.8 - Release 0.91 (RC1) -Release 0.92 (RC2) On time! CERN, 18/06/2008 - 13

  14. 0.92 Release (RC2) Available for download from http://rbuilder.cern.ch/project/cernvm-devel/releases Can be run on Linux (KVM, Xen,VMware Player, VirtualBox) Windows(WMware Player, VirtualBox) Mac (Fusion, Parallels, VirtualBox) Release Notes http://cernvm.web.cern.ch/cernvm/index.cgi?page=ReleaseNotes HowTo http://cernvm.web.cern.ch/cernvm/?page=HowTo Appliance can be configured and used with ALICE, LHCb, ATLAS (and CMS) software frameworks CERN, 18/06/2008 - 14

  15. Content distribution network CVMFS File Server CVMFS File Server CVMFS File Server HTTP Proxy CERN, 18/06/2008 - 15

  16. More than a User Interface.. • BOINC • Open-source software for volunteer computing and grid computing • CernVM is now extended to include BOINC client • Summer student project in OpenLab • based on LHC@HOME experience and CernVM image to run ATLAS simulation using BOINC infrastructure • http://boinc.berkeley.edu/ • Successfully run PanDA job in CernVM • Globus Virtual Workspace Service • An open source infrastructure for the deployment and management of virtual machines • Google Summer School (hosted at ANL) project to deploy a one-click, auto-configuring virtual Grid overlay for Alice/AliEn • http://workspace.globus.org/ • Successfully created virtual AliEn site with one command CERN, 18/06/2008 - 16

  17. CernVM CoPilot Adapter AliEn/DIRAC/PanDA 0. Send host JDL (free disk space, free memory, available packages) 1. Append framework specific information (e.g. CE name) and request a job 2. Send user job JDL from Task Quee 3. Send input files and commands for execution (packages are already there) 4. When the job is done send back the output files (and the result of validation) 5. Register output files CERN, 18/06/2008 - 17

  18. Mailing lists cernvm-talk@cern.ch (Open list to discuss about design, user experience and related issues with the CernVM project) cernvm.support@cern.ch (End-user support for the CernVM project) Savannah Portal Please submit bugs and feature requests to Savannah at http://savannah.cern.ch/projects/cernvm Web site(s) http://cernvm.cern.ch http://rbuilder.cern.ch Links… CERN, 18/06/2008 - 18

  19. Extra slides…

  20. http://cernvm.cern.ch CERN, 18/06/2008 - 20

  21. http://rbuilder.cern.ch/project/cernvm-devel/ CERN, 18/06/2008 - 21

  22. CERN, 18/06/2008 - 22

  23. NOTE - initial settings are:User Name: adminPassword: password CERN, 18/06/2008 - 23

  24. Please change Web admin password! CERN, 18/06/2008 - 24

  25. Select experiment for which VM will be configured CERN, 18/06/2008 - 25

  26. Status page CERN, 18/06/2008 - 26

  27. Select your top level group and migrate appliance configuration to get the latest version CERN, 18/06/2008 - 27

  28. Create local user account. First account created using this interface will be allowed to to executed sudo comand CERN, 18/06/2008 - 28

  29. Hands on… pbuncic$ ssh -X pbuncic@192.168.197.129 pbuncic@cernvm's password: Last login: Fri Jul 11 10:15:48 2008 from 192.168.197.1 [pbuncic@pb-d-128-141-224-235 ~]$ alienv list AliRoot/v4-13-Rev-02 GEANT3/v1-9 ROOT/v5-19-04 CERN, 18/06/2008 - 29

  30. [pbuncic@cernvm ~]$ alienv use AliRoot/v4-13-Rev-02 …. --------------------------------------------------------------- Great! Your compiler setup is ok! --------------------------------------------------------------- ****************************************************** Attention: You don't have a valid grid proxy - doing grid-proxy-init ****************************************************** Your identity: /DC=ch/DC=cern/OU=…. Enter GRID pass phrase for this identity: Creating proxy ....................................... Done Your identity: pbuncic Creating token ..................................... Done Your token is valid until: Sat Jul 12 10:18:52 2008 CERN, 18/06/2008 - 30

  31. bash-3.00$ which aliroot /opt/alice/packages/AliRoot/v4-13-Rev-02/bin/tgt_linux/aliroot bash-3.00$ aliroot -b ******************************************* * W E L C O M E to R O O T * * Version 5.19/04 14 May 2008 * * You are welcome to visit our Web site * * http://root.cern.ch * ******************************************* ROOT 5.19/04 (tags/v5-19-04@23850, Jun 11 2008, 02:19:19 on linux) CINT/ROOT C/C++ Interpreter version 5.16.29, Jan 08, 2008 Type ? for help. Commands must be C++ statements. Enclose multiple statements between { }. root [0] CERN, 18/06/2008 - 31

  32. Cache: ~200 MB bash-3.00$ cernvm-cache -i CONFIGURATION VARIABLES CERNVM_CONFIG_URL=http://cern.ch/cernvm/config CERNVM_ORGANISATION=ALICE CERNVM_PLATFORM=i686-pc-linux-gnu CURRENT LOCATION Public Hostname: pb-d-128-141-224-235.cern.ch Domain: cern.ch Time zone: Europe/Zurich Location: Geneva WEB FILE SYSTEM Server: cernvm-webfs.cern.ch Protocol: grow Cashe directory: /tmp/parrot.500 Cache size: 191 MB CERN, 18/06/2008 - 32

More Related