1 / 23

UKI-SouthGrid Overview GridPP30

UKI-SouthGrid Overview GridPP30. Pete Gronbech SouthGrid Technical Coordinator and GridPP Project Manager Glasgow - March 2012. UK Tier 2 reported CPU – Historical View to present. Last reported stats at CERN in Sept 2011, so data since then. SouthGrid Sites Accounting as reported by APEL.

winchell-vance
Download Presentation

UKI-SouthGrid Overview GridPP30

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UKI-SouthGrid OverviewGridPP30 Pete Gronbech SouthGrid Technical Coordinator and GridPP Project Manager Glasgow - March 2012

  2. UK Tier 2 reported CPU – Historical View to present • Last reported stats at CERN in Sept 2011, so data since then. SouthGrid March 2013

  3. SouthGrid SitesAccounting as reported by APEL SouthGrid March 2013

  4. VO Usage Usage dominated by LHC VOs. 6% Non LHC SouthGrid March 2013

  5. Non LHC VOs A wide range of ‘Other VOs’

  6. Gridpp4 h/w generated MoU for 2012-14 from Steve Lloyd Q412 Resources SouthGrid December 2012

  7. Question of MoU’s • New experimental requirements in Sept 2012 generated new increased shares. SouthGrid March 2013

  8. Dave Britton generated MoU shown at GridPP29 Q412 Resources Up 151TB Up 2933 HS06 SouthGrid December 2012

  9. JET • The site has been under used for the last 6 months. • This is a non Particle Physics site so all LHC work is a bonus. • Essentially a pure CPU site • 1772 HepSPEC06 • 10.5 Tb of storage • All service nodes have been upgraded to EMI2 • CVMFS has been setup and configured for LHCb and Atlas. • Could be utilised much more!! • Active non LHC VOs : Biomed, esr, fusion and Pheno SouthGrid March 2013

  10. Birmingham Tier 2 Site • Most active Non LHC VOs : Biomed and ILC. • MS has helped a local Neuroscience group set up a VO, grid work to follow. (Some involvement with CERN@school) • Mark Slater is the LHCb UK Operations rep/ shifter • Major VO’s are Atlas , ALICE and LHCb. • Middleware: • DPM, WN: EMI 2, Everything else: UMD 1 • Complete overhaul of aircon in the last 12 months. One unit (14kW) left to be installed in the next couple of weeks (hopefully!) • CVMFS fully installed. • Now providing 110TB space for ALICE in their own xrootd area. No other xrootd/webDAV updates SouthGrid March 2013

  11. Bristol Status • StoRM SE upgraded to SL6 Storm 1.10. problematic at first but StoRM developers helped with modify from default config. Helped debug why it was publishing 0 used, apparently known bug. • Upgrades to EMI middleware has improved CMS site readiness • CVMFS setup for CMS, Atlas & LHCb • Onoing development of Hadoop SE: gridFTP + SRM server ready, set-up of PhEDEx(Debug) in progress • Active non LHC Vos : ILC • Landslides VO work currently on hold. • Working with CMS to plan the best way forward for Bristol SouthGrid March 2013

  12. Cambridge • Status • CPU : 140 job slots, 1657 HS06 • Storage : 277TB [si] • Most active non LHC Vos: Camont, but almost exclusively and Atlas /LHCb site. • The Camtology/Imense work at Cambridge has essentially finished (we still host some of their kit) • We have an involvement in a couple of Computational Radiotherapy projects (VoxTox and AccelRT) where they may possibly be some interest in using the Grid SouthGrid March 2013

  13. RALPP • SouthGrid’s biggest site. • Major VO’s CMS, Atlas and LHCb • Non LHC VOs – Biomed, ILC and esr. • Planned migration to a new computer room this year, with six water cooled racks. Will try to minimise downtime. Other racks will move to the Atlas building. • SE is dCache 1.9.12 – planning to upgrade to 2.2 in the near future. • 20Gbit link between the two computer rooms. • Rob Harper is on the security team • New member of staff (Ian Loader) starting very soon. SouthGrid March 2013

  14. Oxford • Oxford’s workload is dominated by ATLAS analysis and production • Most active non LHC VOs : esr, fusion, hone, pheno, t2k and zeus • Recent Upgrades • We have a 10Gbit link to the JANET router but is currently rate capped at 5Gbps. • Oxford will get a second 10Gbit line enabled soon and then the rate cap can get lifted • SouthGrid Support • Providing support for Bristol, Sussex and JET • The Landslides VO supported at Oxford and Bristol • Helped bring Sussex onto the Grid as an Atlas site • Oxford Particle Physics Masterclasses with Grid Computing talk. SouthGrid March 2013

  15. Other Oxford Work • CMS Tier 3 • Supported by RALPPD’s PhEDEx server. Now configured to use CVMFS and xrootd as the local file access protocol • Useful for CMS, and for us, keeping the site busy in quiet times • However can block Atlas jobs soduring accounting period max running jobs limit applied • Largest non CMS Tier-2 site in UK • ALICE Support • The ALICE computational requirements are shared between Birminghamand Oxford. • UK Regional Monitoring • Kashif runs the nagios based WLCG monitoring on the servers at Oxford • These include the Nagios server itself, and support nodes for it, SE, MyProxy and WMS/LB • KM also remotely manages the failover instance at Lancaster • There are very regular software updates for the WLCG Nagios monitoring. • VOMS server replication at Oxford (and IC) • Early Adopters • In the past were official early adopters for testing of CREAM, ARGUS and torque_utils. Recently have tested early in a less official way. SouthGrid March 2013

  16. Multi VO Nagios Monitoring • Monitoring three VO’s • T2k.org • Snoplus.snolab.ca • Vo.southgrid.ac.uk • Customized to suit small VO’s • Only using direct job submission • Test jobs submitted every 8 hours • Test jobs can stay in queue for 6 hours before being cancelled by Nagios

  17. Multi VO Nagios Monitoring • Using VO-feed for topology information • VO-feed is also hosted at Oxford • Fine grained control of what to monitor • But it requires manual changes • Probably it is the first production Multi-VO Nagios instance in egi • Found many bugs • It worked very well with two VO’s but after adding third VO it showing some strange issues. • Opened a GGUS ticket and working on it

  18. GridPP Cloud work at Oxford • Working to provide an OpenStack based cloud infrastructure at Oxford • OpenStack Folsom release installed with the help of Cobbler and Puppet • Using RHEL 6.4 • Most of installation and configuration is automated • Started with three machines • One controller node running OpenStack core services • One Compute node running Nova-compute and Nova-network • One storage node to provide block storage and NFS mount for glance images • Plan to add more compute nodes in future • We are open to provide our infrastructure for testing • https://www.gridpp.ac.uk/wiki/GridPP_Cloud_sites

  19. XrootD and WebDAV • Xrootd and WebDAV on DPM • Completely separate services, but similar minimum requirements, so if you can do XrootD, you can do WebDAV • Local XrootD is pretty well organised, federations currently require name lookup libraries which lack a good distribution mechanism. • Configuration is mostly boiler plate – copy ours: # Federated xrootd DPM_XROOTD_FEDREDIRS="atlas-xrd-uk.cern.ch:1094:1098,atlas,/atlas xrootd.ba.infn.it:1094:1213,cms,/store" # Atlas federated xrootd DPM_XROOTD_FED_ATLAS_NAMELIBPFX="/dpm/physics.ox.ac.uk/home/atlas" DPM_XROOTD_FED_ATLAS_NAMELIB="XrdOucName2NameLFC.so root=/dpm/physics.ox.ac.uk/home/atlas match=t2se01.physics.ox.ac.uk" DPM_XROOTD_FED_ATLAS_SETENV="LFC_HOST=prod-lfc-atlas-ro.cern.ch LFC_CONRETRY=0 GLOBUS_THREAD_MODEL=pthread CSEC_MECH=ID" # CMS federated xrootd DPM_XROOTD_FED_CMS_NAMELIBPFX="/dpm/physics.ox.ac.uk/home/cms" DPM_XROOTD_FED_CMS_NAMELIB="libXrdCmsTfc.so file:/etc/xrootd/storage.xml?protocol=xroot" # General local xrootd DPM_XROOTD_SHAREDKEY=“bIgl0ngstr1ng0FstuFFi5l0ng" DPM_XROOTD_DISK_MISC="xrootd.monitor all rbuff 32k auth flush 30s window 5s dest files info user ioredir atl-prod05.slac.stanford.edu:9930 if exec xrootd xrd.report atl-prod05.slac.stanford.edu:9931 every 60s all -buff -poll sync fi" DPM_XROOTD_REDIR_MISC="$DPM_XROOTD_DISK_MISC" DPM_XROOTD_FED_ATLAS_MISC="$DPM_XROOTD_DISK_MISC" SouthGrid March 2013

  20. XrootD • The main source of documentation is here: https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/Xroot/Setup • We’ve also switched to using xrootd for ATLAS and CMS local file access (which is a VO side change, not a site one), but this isn’t making use of the federation yet. • ATLAS are currently using XrootD based file stager copies, not XrootD direct IO. We do hope to try that too. • All the xrootd traffic currently gets reported to the ATLAS FAX monitoring, which made the graphs look a bit odd when we turned it on: SouthGrid March 2013

  21. WebDAV • WebDAV is simpler: # DPM webdav DPM_DAV="yes" # Enable DAV access DPM_DAV_NS_FLAGS="Write" # Allow write access on the NS node DPM_DAV_DISK_FLAGS="Write" # Allow write access on the disk nodes DPM_DAV_SECURE_REDIRECT="On" # Enable redirection from head to disk using plain HTTP. • Commodity clients work, but nothing supports everything you’d want. • More here: https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/WebDAV/ClientTutorial SouthGrid March 2013

  22. Sussex • Sussex has a significant local ATLAS group, their system is designed for the high IO bandwidth patterns that ATLAS analysis can generate. • EMI2 middleware installed • CVMFS installed and configured • Setup as an Atlas production site running jobs in anger since February 2013 • New 64 core node arriving in the next week and a further 128 cores and 120TB to be added in the summer. • SNO+ is expected to start using the site shortly. • JANET link scheduled to be upgraded from current 2Gb (plus 1Gb resilient failover) to 10Gb in Autumn 2013. SouthGrid March 2013

  23. Conclusions • SouthGrid seven sites well utilised, but some sites small compared with others. • Birmingham supporting Atlas, Alice and LHCb. • Bristol; Have upgraded to the latest version of STORM, and EMI middleware and have been available to CMS since mid December. Hope to be better utilised by CMS now. Local funding will be used to enhance the CPU and storage. • Cambridge; size was reduce when Condor part of the cluster decommissioned, local funding to be used to boost capacity. • JET continue to be available as a CPU site but very little storage available. However CVMFS and all middleware at EMI2. Should be a useful MC site for LHCb • Oxford wide involvement in many VOs and areas of development and GridPP infrastructure. • RALPPD remain SouthGrid s largest site and a major CMS contributor. • Sussex; successfully running as an Atlas site with some local upgrades planned for this year. SouthGrid March 2013

More Related