Options for Expanding CERN’s Computing Capacity Without A New Building

1. Options for Expanding CERN’s Computing Capacity Without A New Building Medium term plans to handle growth of CERN’s computing needs HEPiX NERSC Alan Silverman (with input by T.Cass, L.Pregernigand W.Salter) 29th October 2009

2. Outline • Background • Options – hosting, containers, new centre • Features of Hosting • Features of Containers • IT’s plans • Current status

3. Limits of Building 513 Capacity • Building 513 arriving to maximum capacity for power and cooling while demands to expand LCG and infrastructure services continue to grow. • Limit probably now reached with respect to “critical” systems – systems whose loss would mean major disruption (corporate data and physics meta-data, mail servers, network devices, etc). • Plans being made to accommodate more critical services but short term measures needed for next round of database servers among remaining 2009 purchases. IT Department – 29 October 2009 – HEPiX NERSC- 3

4. Long Term – A New Building? • Plans and budget exist for a Tier 0 centre – a major new building, probably on Prevessin site. • Uncertainty created by - • Uncertain physics needs in advance of real LHC data • Ongoing debate of cost-effectiveness of one-time new building versus incrementally adding containers • Unknown costing of different options • Our unfamiliarity with these options • Offer from Norway to host a remote centre - • costing unclear • remote operations support also unclear • Prefer to take time to understand these various options before coming to a decision. IT Department – 29 October 2009 – HEPiX NERSC- 4

5. Short and Medium term • Short-term: Price Enquiry issued to host offsite some “critical” systems to be acquired in the next round of purchases. • Medium-term: Call for Tender to purchase (or rent) a number of Containers to install on the CERN site. • Should be enough for the coming 2-5 years to give us time - • to get some experience of the Container option • to decide long term future • if a new building is chosen, to get it designed and built. IT Department – 29 October 2009 – HEPiX NERSC- 5

6. Hosting – choices - 1 • Rent space and use our own racks or rent space in their racks? • If their racks, does our choice of PC fit with the specifications we select from our PC suppliers? • Do they support the power rating we need per rack? • Options for cooling the racks • Do we rent space now for possible expansion or add this later? (Risk of renting non-contiguous spaces – does this even matter?) IT Department – 29 October– HEPiX- 6

7. Hosting – choices - 2 • Dual power supply? UPS cover? • Networking facilities in the Hosting Centre and between the Hosting Centre and CERN? • Local (near CERN) so we can install our own systems and manage them; or remote in which case what service is available? Affect on cost (Geneva prices)? • Any software licences needed which may be tied to CERN site? • Maintenance contracts for PCs still valid? • Data protection laws if personnel or financial data stored offsite? IT Department – 29 October– HEPiX- 7

8. Status of Hosting Plans - 1 • Stay close to CERN so that we can get some experience with minimal risk. • Supplier identified after Price Enquiry. • One attractive supplier did not bid because his current installation is full and it would have cost too much for him/us to equip the next part of his building in time for our needs. • Contract under negotiation with target of February start. • Data protection laws in Switzerland are rather solid (fall-out from Swiss banking) • Need 10GB network to CERN – under negotiation IT Department – 29 October– HEPiX- 8

9. Status of Hosting Plans - 2 • We have decided to use our own racks (because they are cheaper and we can use our manpower for installation) • We will probably only rent space for immediate use and take the risk of extra space if we need it later being in a distant part of the hosting site?? • No software licence issue but some critical servers can never be off-site for this reason. • Since the hosting site is in Geneva, no maintenance issue this time. IT Department – 29 October– HEPiX- 9

10. Container Features • Size typically 3 by 12 metres. • Can be stacked but then you need infrastructure to work on upper boxes. • Can be in open air (e.g. at SLAC) but easier if indoors (e.g. Microsoft’s new supercentre in Chicago – space for 56 containers times 2000 servers per container) - • safer access when raining/snowing • covered area to unpack equipment to install. • If indoors, need crane or access for mobile crane to install the boxes. • No special access needs on Building, already on the Containers themselves. IT Department – 29 October 2009 – HEPiX NERSC- 10

11. Container Features - 2 • Need access to - • power • water (even if we buy chillers we need to fill them) • technical galleries (for the power and high-speed networking). • Need 400V, 3 phase power - • typically up to 15-20 racks times 15-20Kw/rack = ~300Kw per unit for computers (depends on PCs installed) • 150Kw per unit for cooling. IT Department – 29 October 2009 – HEPiX NERSC- 11

12. IT’s Plans for containers • Initially buy (or rent?) 1 unit for “critical power” systems. Improves IT’s Business Continuity - different power source from 513, distance from 513. • Plus 1 unit for LCG physics. • Plan on 1 additional unit for physics per year for up to 4 years. • Two options (in fact decided on second):- • Buy/rent containers and use Site Services for all power and cooling (then need power and cooling redundancy for critical power container plus UPS for physics units) • Buy/rent a complete service which implies an extra unit to provide safe power and external chillers for the containers. IT Department – 29 October 2009 – HEPiX NERSC- 12

13. IT’s needs for containers • Ideally a building (Prevessin site? Meyrin West Area?) with capacity as needed • Prevessin site offers better “business continuity”. This was chosen. • Open ground (recommended to be concreted) with foreseen capacity for 4 years growth; ideally one area but not essential. This is the likely choice. IT Department – 29 October 2009 – HEPiX NERSC- 13

14. Status of Container Plans - 1 • Outdoor site (with a nearby small building) identified for initial set of containers. Near CCC so good networking close by. • EN agree to supply 400V power from a nearby 18KV line and a basic water supply but no cooling facilities (so we need to acquire the chillers as well as backup diesels and UPS) • CERN’s complicated purchase procedure has started; target installation Oct 2010 IT Department – 29 October 2009 – HEPiX NERSC- 14

15. Status of Container Plans - 2 • Initial installation is planned to be – • One container for critical services – needs UPS in addition to dual power feeds; • One container for physics – needs only dual power feeds; • Each container will need a second container for diesel backup and chillers (and UPS for critical services container). • Possibility to add one physics container (probably plus services container) per year for 2-3 years. • Open question – buy or rent? What is the rental amortisation period? IT Department – 29 October– HEPiX- 15