Hardware Reliability at the RAL Tier1

1. Hardware Reliability at the RAL Tier1 Gareth Smith 16th September 2011

2. Staffing The following staff have left since GridPP26: • James Thorne (Fabric - Disk servers) • Matt Hodges (Grid Services Team leader) • Derek Ross (Grid Services Team) • Richard Hellier (Grid Services & Castor Teams) We thank them for their work whilst with the Tier1 team. Tier-1 Status

3. Some Changes • CVMFS in use for Atlas & LHCb: • The Atlas (NFS) software server used to give significant problems. • Some CVMFS teething issues but overall much better! • Virtualisation: • Starting to bear fruit. Uses Hyper-V. • Numerous test systems • Production systems that do not require particular resilience. • Quattor: • Large gains already made. Tier-1 Status

4. CernVM-FS Service Architecture Atlas install box in PH LHCb install box in PH Stratum 0 web in PH cvmfs-public@cern cvmfs-ral@cern cvmfs-bnl@cern • Replication to Stratum 1 by hourly cron (for now) • Stratum 0 moving to IT by end of year • BNL almost in production Squid Random site Batch Node Batch Node Batch Node 6th May 2011

5. CernVM-FS Service At RAL webfs.gridpp.rl.ac.uk WEB Server - replicated Stratum 0 at CERN iSCSI Storage cvmfs.gridpp.rl.ac.uk The Replica at RAL - presented as a virtual host in 2 reverse proxy squids accelerating webfs in the background lcgsquid05 lcgsquid06 squid cache squid cache Squid Squid(s) Random site RAL batch Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node Batch Node 6th May 2011

6. Database Infrastructure We making Significant Changes to the Oracle Database Infrastructure. Why? • Old servers are out of maintenance • Move from 32bit to 64bit databases • Performance improvements • Standby systems • Simplified architecture

7. Database Disk Arrays - Now Oracle RAC Nodes Fibrechannel SAN Power Supplies (on UPS) Disk Arrays Tier-1 Status

8. Database Disk Arrays - Future Oracle RAC Nodes Fibrechannel SAN Data Guard Power Supplies (on UPS) Disk Arrays Tier-1 Status

9. Castor Changes since last GridPP Meeting: • Castor upgrade to 2.1.10 (March) • Castor version 2.1.10-1 (July) needed for the higher capacity "T10KC" tapes. • Updated Garbage Collection Algorithm (to “LRU” rather than the default which is based on size). (July) • (Moved ‘logrotate’ to 1pm rather than 4am.) Tier-1 Status

10. Castor Issues. • Load related issues on small/full service classes (e.g. AtlasScratchDisk; LHCbRawRDst) • Load can become concentrated on one or two disk servers. • Exacerbated if uneven distribution if disk server sizes. • Solutions: • Add more capacity; clean-up. • Changes to tape migration policies. • Re-organization of service classes. Tier-1 Status

11. Procurement All existing bulk capacity orders in production or being deployed. Problems ‘SL08’ generation overcome. Tenders under way for disk and tape. Disk: • Anticipate 2.66 PB usable space. • Vendor 24 day proving test using our tests. Then • Re-install and 7 days acceptance tests by us. CPU: • Anticipate 12k HEPSpec06 • 14 day proving tests by vendor. Then • 14 day acceptance tests by us. Evaluation based on 5 year Total Cost of Ownership. Tier-1 Status

12. Disk Server Outages by Cause (2011) Tier-1 Status

13. Disk Server Outages by Service Class (2011) Tier-1 Status

14. Disk Drive Failure – Year 2011

15. Double Disk Failures (2011) In process of updating the firmware on the particular batch of disk controllers. Tier-1 Status

16. Disk Server Issues - Responses New Possibilities with Castor 2.1.9 or later: • ‘Draining’ (‘passive’ and ‘active’) • Read –only server. • Checksumming (2.1.9) – easier to validate files plus regular check of files written. All of these used regularly when responding to a disk server problems. Tier-1 Status

17. Data Loss Incidents Summary of losses since GridPP26 Total of 12 incidents logged: • 1 – Due to a disk server failure (loss of 8 files for CMS) • 1 – Due to a bad tape (loss of 3 files for LHCb) • 1 - Files not in Castor Nameserver but no location. ( 9 LHCb files) • 9 – Cases of corrupt files. In most cases the files were old (and pre-date Castor checksumming). Checksumming in place of tape and disk files. Daily and random checks made on disk files. Tier-1 Status

18. T10000 Tapes Type Capacity In Use Total Capacity A 0.5TB 5570 2.2PB B 1TB 2170 1.9PB (CMS) C 5TB We have 320 T10KC tapes (capacity ~1.5PByte) already purchased. Plan is to move data (VOs) using the ‘A’ tapes onto the ‘C’ tapes, leapfrogging the ‘Bs’. Tier-1 Status

19. T10000C Issues • Failure of 6 out of 10 tapes. • Current A/B failure rate roughly 1 in 1000. • After writing part of a tape an error was reported. • Concerns are three fold: • A high rate of write errors cause disruption • If tapes could not be filled our capacity would be reduced • We were not 100% confident that data would be secure • Updated Firmware in drives. • 100 tapes now successfully written without problem. • In contact with Oracle. Tier-1 Status

20. Long-Term Operational Issues • Building R89 (Noisy power v EMC units); Electrical Discharge in 11kV supply. • Noisy electrical current: Fixed by use of isolating transformers in appropriate places. • Await final information on resolution of 11kV discharge. • Asymmetric Data Transfer rates in/out of RAL Tier1. • Many possible causes: Load; FTS settings, Disk server settings; TCP/IP tuning, network (LAN & WAN performance). • Have modified FTS settings with some success. • Looking at Tier1-UK Tier2 transfers as within GridPP. Tier-1 Status

21. Long-Term Operational Issues • BDII issues. (Site BDII stops updating). • Monitoring & re-starters in place. • Packet Loss on RAL Network Link • Particular effect on LFC updates (reported by Atlas) • Some evidence of being load related. • Some ‘hangs’ within Castor JobManager • Difficult to trace, very intermittent. Tier-1 Status

22. Other reported hardware issues The following were reported via theTier1-Experiments Liaison Meeting. • (Isolating transformers in disk array power feeds - final resolution) • Network switch stack failure • Network (transceiver) failure in switch – in link to tape system • CMS LSF machine was also turned off by mistake - resulting in a short outage for srm-cms. • T10KC tapes • 11kV feed into building - discharge. • Failure of Site Access Router Tier-1 Status

23. Other Hardware Issues The following were reported via theTier1-Experiments Liaison Meeting. • (Isolating transformers in disk array power feeds - final resolution of long standing problem) • Network switch stack failure • Network (transceiver) failure in switch – in link to tape system • CMS LSF machine was also turned off by mistake - resulting in a short outage for srm-cms. • T10KC tapes • 11kV feed into building – electrical discharge. • Failure of Site Access Router Tier-1 Status

24. A couple of final comments Disk server issues are the main area of effort for hardware reliability / stability. ...but do not forget the network. Hardware that has performed reliably in the past may throw up a systematic problem. Tier-1 Status

25. Additional Slide: Procedures.... Post Mortem: 1 Post Mortem During 6 months since GridPP26. 10th May 2011: LFC Outage After Database Update • 1 hour outage following an At Risk. Caused by configuration error in Oracle ACL lists. Disaster Management: • Triggered once (for T10KC tape problem). Tier-1 Status