SPI Report for the LHCC Comprehensive Review

1. SPI Report for the LHCC Comprehensive Review Stefan Roiser for the SPI project

2. SPI services • Software process support • Software testing, configuration & release • Documentation support • Code documentation, web services • Collaboration tools • Hypernews, Savannah • Hardware infrastructure • Build servers, special purpose machines LHCC Comprehensive Review - SPI

3. Software

4. Time to user <- Deployment <- Release <- Building / Testing <- -> Bug report / Feature request -> Implementation • The time from a bug report / feature request until it is used by the physicists in the experiments • Especially important to minimize this time when running with “live” data. LHCC Comprehensive Review - SPI

5. Nightly Builds • Recommendation of the 2006 internal review • Motivation • Early integration into experiment software • Fast feedback to LCG/AA developers of changes • Usage • Building with own experiment nightlies on top • Developers testing their code on LCG/AA stack • Used by • LHCb • Atlas LHCC Comprehensive Review - SPI

6. Nightly Builds (ctd.) 5 configurations Builds & Tests 13 possible platforms LCG/AA projects test evolution LHCC Comprehensive Review - SPI

7. Software release Experiment software Experiment software eg. Athena, CMSSW, Gaudi eg. Athena, CMSSW, Gaudi LCG / AA software LCG / AA software External Software External Software Operating System Operating System Linux (slc) 32/64 bit, Mac OSX, Windows Linux (slc) 32/64 bit, Mac OSX, Windows LCG release • rubberstamps all package versions • provides several products • afs installation • web pages • src tar balls • bin tar balls • documentation COOL, CORAL, POOL, RELAX, ROOT, SEAL 56 packages, eg. Boost, Python, Qt, etc… LHCC Comprehensive Review - SPI

8. Software processes historically grown nightly builds web sites docu python mysql cron / python tar balls xml xml / python install. mgr. cmt scram extern s/w project s/w LHCC Comprehensive Review - SPI

9. Software processes new nightly builds web sites docu Python / LCGCMT tar balls install. mgr. extern s/w project s/w LHCC Comprehensive Review - SPI

10. LCGCMT • Is a CMT based project • Contains the configuration information • How to build software packages from source • Using sh/bat scripts • Extra information • Eg. homepage, package maintainer, description, etc. • In addition for other CMT based projects • How to use/build against the packages • Runtime environment LHCC Comprehensive Review - SPI

11. Example: package build package gsl use LCG_BuildPolicy v* LCG_Builders macro gsl_build_config_version "$(GSL_config_version)" set LCG_tarfilename "gsl-$(gsl_build_config_version).tar.gz" apply_pattern buildscripts_local #!/bin/sh cd ${LCG_builddir} tar xvfz ${LCG_tarfilename} cd gsl-${LCG_package_config_version} ./configure --prefix=${LCG_extdir}/GSL/${LCG_package_config_version} /${LCG_CMTCONFIG} #!/bin/sh cd ${LCG_builddir}/gsl-${LCG_package_config_version} make #!/bin/sh cd ${LCG_builddir}/gsl-${LCG_package_config_version} make install lxplus:~ >setenv CMTCONFIG slc4_amd64_gcc34 lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_get lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_config lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_make lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_install LHCC Comprehensive Review - SPI

12. Benefits of new s/w process <- Deployment <- Release <- Building / Testing <- -> Bug report / Feature request -> Implementation Current configuration LCG 53f has been built with these tools • Avoid duplication of information • Easier maintenance / automation • All parties use the same info (nightly builds, release, users rebuilding the software) • Speedup time to user LHCC Comprehensive Review - SPI

13. Communication

14. Hypernews • Bulletin board system used by LHC experiments • Moved to new IT provided hardware • 1 production machine, 1 hot standby • Helping with instance maintenance • Managing user DB, setup of for a • Atlas and CMS heavy users • Also used in Alice and Totem • LHCb is rethinking their usage LHCC Comprehensive Review - SPI

15. Savannah • Savannah is the main bug reporting system in LCG/AA • Moved to new IT provided hardware • Used by many other parties (IT, Grid, etc.) • Reasonable system / user maintenance LHCC Comprehensive Review - SPI

16. Hardware

17. Hardware • LCG / AA build servers • 4 linux machines (managed by IT/FIO) • 2 mac osx machines (one ppc) • 2 windows machines • Special purpose machines • Web / database servers • Recycling old Hypernews machines (located in CC) • New machines for • Savannah, Hypernews LHCC Comprehensive Review - SPI

18. Staff & Program of Work

19. Staff • Change of project leader • Andreas Pfeiffer left to CMS • Stefan Roiser took over • 1 staff left, 1 fellow joined • Manuel Gallas left for Atlas • Piotr Kolet joined • 1 summer student • Luana Micallef • Currently 3.2 FTE LHCC Comprehensive Review - SPI

20. Program of work • Completion of the integrated software configuration / testing / release • Web pages derived from LCGCMT • Integrate remaining release steps • Consistency checking • Production of distribution kits • Documentation generation • Completion of integration with nightly builds • Integration of installation manager LHCC Comprehensive Review - SPI

21. Program of work (ctd.) • Nightly build system • Integrate Geant4 builds • Using IT infrastructure for • Setup of common LCG/AA web services • Setup of database services • Virtual windows machines for s/w build LHCC Comprehensive Review - SPI

22. Program of work (ctd.) • Hypernews • Investigate usage of IT provided SSO • Find solution for longterm support LHCC Comprehensive Review - SPI

23. Conclusion • Current focus on LCG/AA s/w stack • Optimize release process • Facilitate and unify processes for • Nightly builds • Release procedure • User rebuilds • Outsourcing of maintenance for other services, but still a lot of SPI manpower involved • Savannah • Hypernews • Hardware LHCC Comprehensive Review - SPI