1 / 23

UKI-SouthGrid Overview and Oxford Status Report

UKI-SouthGrid Overview and Oxford Status Report. Pete Gronbech SouthGrid Technical Coordinator HEPSYSMAN RAL 30 th June 2009. SouthGrid Tier 2. The UK is split into 4 geographically distributed tier 2 centres SouthGrid comprise of all the southern sites not in London.

terrycopeland
Download Presentation

UKI-SouthGrid Overview and Oxford Status Report

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UKI-SouthGrid Overview and Oxford Status Report Pete Gronbech SouthGrid Technical Coordinator HEPSYSMAN RAL 30th June 2009

  2. SouthGrid Tier 2 • The UK is split into 4 geographically distributed tier 2 centres • SouthGrid comprise of all the southern sites not in London • New sites likely to join SouthGrid Status June 2009

  3. UK Tier 2 reported CPU – Historical View to present SouthGrid Status June 2009

  4. SouthGrid SitesAccounting as reported by APEL SouthGrid Status June 2009

  5. Site Upgrades in the last 6 months • RALPPD Increase of 960 cores (1716KSI2K) +380TB • Cambridge 32 cores (83KSI2K) + 20TB • Birmingham 64 cores on pp cluster and 192 cores HPC cluster which add ~535KSI2K • Bristol original cluster replaced by new quad cores systems 24 cores + increased share of the HPC cluster 125KSI2k + 44TB • Oxford extra 208 cores 540KSI2K + 60TB • Jet extra 120 cores 240KSI2K SouthGrid Status June 2009

  6. New Total Q209SouthGrid SouthGrid Status June 2009

  7. Site Setup Summary SouthGrid Status June 2009

  8. New Staff • Jan 2009 Kashif Mohammad Deputy Technical Coordinator based at Oxford • May 2009 Chris Curtis SouthGrid Hardware support based at Birmingham • June 2009 Bob Cregan HPC support at Bristol SouthGrid Status June 2009

  9. Oxford Site Report SouthGrid Status June 2009

  10. Oxford Central Physics • Centrally supported Windows XP desktops (~500) • Physics wide Exchange Server for email • BES to support Blackberries • Network services for MAC OSX • Astro converted entirely to Central Physics IT services (120 OSX systems) • Started experimenting with Xgrid • Media services • Photocopiers/printers replaced – much lower costs than other departmental printers. • Network • Network is too large. Looking to divide into smaller pieces – better management and easier to scale to higher performance. • Wireless – introduced EDUROAM on all physics WLAN base stations. • Identified problems with 3com 4200G switch which caused a few connections to run very slowly. Now fixed. • Improved network core and computer room with redundant pairs of 3com 5500 switches. SouthGrid Status June 2009

  11. Oxford Tier 2 ReportMajor Upgrade 2007 • Lack of decent Computer room with adequate power and A/C held back upgrading our 2004 kit until Autumn 07 • 11 systems, 22 servers, 44 cpus, 176 cores. Intel 5345 Clovertown cpu’s provide ~430KSI2K, 16GB memory for each server. Each server has a 500GB SATA HD with IPMI remote KVM cards. Old News • 11 servers each providing 9TB usable storage after RAID 6, total ~99TB, 3ware 9650-16ML controller. • Two racks, 4 Redundant Management Nodes, 4 APC 7953 PDU’s, 4 UPS’s SouthGrid Status June 2009

  12. Oxford Physics now has two Computer Rooms • Oxford’s Grid Cluster initially housed in the departmental Computer room late 2007 • Later moved to the new shared University room at Begbroke (5 miles up the road) SouthGrid Status June 2009

  13. Oxford Upgrade 2008 • 13 systems, 26 servers, 52 cpus, 208 cores. Intel 5420 Harpertown cpu’s provide ~540KSI2K, 16GB Low Voltage FBDIMM memory for each server. Each server has a 500GB SATA HD. More of the same but better! • 3 servers each providing 20TB usable storage after RAID 6, total ~60TB, Areca Controllers • 3 3com 5500 switches and backplane interconnects SouthGrid Status June 2009

  14. T2wn40 t2torque02 t2ce02 t2ce04 t2ce05 T2wn87 t2torque T2wn86 T2wn40 T2wn40 T2wn40 T2wn85 T2wn40 Grid Cluster setup SL5 test nodes available Glite 3.2 SL5 Glite 3.1 SL4 SouthGrid Status June 2009

  15. Nov 2008 Upgrade to the Oxford Grid Cluster at Begbroke Science Park SouthGrid Status June 2009

  16. Local PP Cluster (tier 3) • Nov 2008 Upgrade same h/w as the Grid • 3 Storage Nodes • 8 Twins • 3 3com 5500 switches with backplane interconnects • 100Mb/s switches used for the management cards (ipmi and RAID) • APC Rack; Very easy to mount the APC pdu’s • Still running SL4 but have a test SL5 system for users to try. We are ready to switch over when we have to. • Lustre FS not yet implemented due to lack of time. SouthGrid Status June 2009

  17. Electrical Power consumption • Newer generation Intel Quads take less power • Tested using one cpuburn process per core on both sides of a twin killing a process every 5 minutes. Busy 645W Idle 410W Busy 490W Idle 320W Intel 5345 Intel 5420 SouthGrid Status June 2009

  18. Electricity Costs* • We have to pay for the electricity used at the Begbroke Computer Room: • Cost in electricity to run old (4 years) Dell nodes is ~£8600 per year. (~79 KSI2k) • Replacement cost in new twins is ~£6600 with electricity cost of ~£1100 per year. • So saving of ~£900 in the first year and £7500 per year there after. • Conclusion is, its not economically viable to run kit older than 4 years. * Jan 2008 figures SouthGrid Status June 2009

  19. IT related power saving • Shutting down desktops when idle • Must be idle, logged off, no shared printers or disks, no remote access etc. • 140 machines regularly shut down • Automatic power up early in the morning to apply patches and get ready for user (using Wake-On-LAN) • Old cluster nodes removed/replaced with more efficient servers • Virtualisation reduces number of servers and power. • Computer room temperatures raised to improve A/C efficiency (from 19C to 23-25C) • Windows 2008 server allows control of new power saving options on more modern desktop systems SouthGrid Status June 2009

  20. CPU Benchmarking HEPSPEC06 These figures match closely with those published on http://www.infn.it/CCR/server/cpu2006-hepspec06-table.jpg Nehalem Dell server just arrived for testing. SouthGrid Status June 2009

  21. Cluster Usage at Oxford Roughly equal share between LHCb and ATLAS for CPU hours. ATLAS runs many short jobs. LHCb longer jobs. Cluster occupancy approx 70% so still room for more jobs. Local contribution To Atlas MC storage SouthGrid Status June 2009

  22. 200mbps In Oxford recently had its network link rate capped to 100mbs This was as a result of continuous 300-350mbs traffic caused by CMS commissioning stress testing. As it happens this test completed at the same time as we were capped, so we passed the test, and current normal use is not expected to be this high Oxfords Janet link is actually 2 * 1gbit links which had become saturated. Short term solution is to only rate cap JANET traffic to 200mbs which doesn’t impact on normal working (for now) all other on site traffic remains at 1gbs. Long term plan is to upgrade the JANET link to 10gbs within the year. out SouthGrid Status June 2009

  23. gridppnagios Have setup a nagios monitoring site for the UK which several other sites use to get advanced warnings of failures.https://gridppnagios.physics.ox.ac.uk/nagios/https://twiki.cern.ch/twiki/bin/view/LCG/GridServiceMonitoringInfo SouthGrid Status June 2009

More Related