Where Does the Power go in DCs & How to get it Back

1. Where Does the Power go in DCs & How to get it Back Foo Camp 2008 2008-07-12 James Hamilton email :JamesRH@microsoft.com web: http://mvdirona.com blog: http://perspectives.mvdirona.com

2. Agenda • Power is the important measure • Power drives costs in that Data Center costs are 80% providing power and cooling infrastructure • Increasing concern about DC power consumption • Work done/watt • Power In: Power Distribution & Optimizations • Servers: Critical Load & Optimizations • Heat Out: Mechanical Systems & Optimization http://perspectives.mvdirona.com

3. Power Distribution: Utility to CPU • Power Conversions to server (each roughly 98%) • High (115kVAC) to medium(13.2kVAc) [differs by geo] • Uninterruptable Power Supply & Generators: • Running at 13.2VAC • UPS: can be rotary or battery • Good ones in 97% range. Much more common 93 to 94% • Common: rectify to DC, trickle to batteries, then invert to AC (~93%) • No loss at generators (please don’t start them: ~130gallons/hour * 10 or so) • 13.2kVACto 480VAC • 480VAC to 208VAC • Conversions in Server to CPU & Memory: • Power Supply: 208VAC to 12VDC (80% common, ~95% affordable) • VRM: 12VDC to ~1.5VDC (80% common, 90% affordable) http://perspectives.mvdirona.com

4. Power Redundancy at Geo-Level • Over 20% of entire DC costs is in power redundancy • Batteries able to supply up to 15 min at some facilities • N+2 generation (2.5MW) at over $2M each • Instead, use more smaller, cheaper data centers • Eliminate redundant power & bulk of shell costs • Average UPS in the 93% range • Over 1MW wasted in 15MW facility http://perspectives.mvdirona.com

5. Power Distribution Optimization • Rules to minimize power distribution losses: • Avoid conversions (Less transformer steps & efficient or no UPS) • Increase efficiency of conversions • High voltage as close to load as possible • Size voltage regulators (VRM/VRDs) to load & use efficient parts • DC distribution potentially a small win • With regulatory issues • Two interesting approaches: • 480VAC (or higher) to rack & 48VDC (or 12VDC) within • 480VAC to PDU and 277VAC to load • 1 leg of 480VAC 3-phase distribution • Common design: 44% lost in distribution • 1*.98*.98*.93*.98*.8*.8 => 56% (~4.4MW lost on 10MW total) • Affordable technology: 1*.99*.99*.95*.95 => 88% (~1.2MW total) http://perspectives.mvdirona.com

6. Critical Load Optimization • Power proportionality is great but “off” is even better • Today: Idle server consumes ~60% power of full load • Industry secret: “good” data center server utilization around ~30% • Off requires changing workload location • What limits 100% dynamic workload distribution? • Networking constraints • VIPs can’t span L2 nets, ACLs are static, manual configuration, etc. • Data Locality • Hard to efficiently move several TB & workload needs to be close to data • Workload management: • Scheduling work over resources optimizing for power with SLA constraint • Server power management • Most workloads don’t fully utilize all resources on server • Need ability to shut off or de-clock unused server resources • Very low power states recover more quickly • Move from 30% utilization to 80% http://perspectives.mvdirona.com

7. CEMS: Thin Slice Computing • Cooperating Expendable Micro-Slice Servers • Correct system balance problem with less-capable CPU • Too many cores, running too fast for memory, bus, disk, … • Power consumption scales with cube of clock frequency • Goal: ¼ the price & much less than ½ the power • Utilize high-volume client parts in server environment • Goal: 20 to 50W at under $500 • 1U form factor or less with service-free design • Longer term goals • High-density, shared power supply & boot disk • Eliminate non-server required components • Establish viability of service free designs http://msblogs/JamesRH

8. Conventional Mechanical Design • Server fans (from components to air) • CRACs: (from air to chilled water) • Air moving over long distance expensive • Air control often poor with hot/cold mixing • Secondary water circuit (variable flow) • Primary water circuit (fixed flow) • Water side economizer & A/C evaporator • Condensate circuit • A/C condenser • Water side economizer • Cooling tower http://perspectives.mvdirona.com

9. Mechanical Optimization • Simple rules to minimize cooling costs: • Raise data center temperatures • Tight control of airflow with short paths • Cooling towers rather than A/C • Air side economization (open the window) • Low grade, waste heat energy reclamation • Best current designs have water close to load but don’t use direct water cooling • Lower heat densities could be 100% air cooled but density trends suggest this won’t happen • Common mechanical designs: 24% lost in cooling • Assume reduction to 1/3 current • 24% to 8% for 16% savings http://perspectives.mvdirona.com

10. Summary • Some low-scale facilities incredibly bad • Assuming current high-scale installation: • Power distribution savings ~32% • Save 8% in power distribution to server • Save further 24% power distribution losses in server • Cooling Savings: ~16% • Conservatively estimate 1/3 the power using air-side economization • 24% loss down to 8% for a 16% power savings • Server Utilization: ~90% • Move from 30% to 80% through DC-wide workload scheduling • 30% load @ 60% of full load power to 80% load @ 100% of full load power • 2.6x work at 1.7x more power for a gain of 90% • Cooperative, Expendable, Micro-slice Servers: ~12% • ½ the power but less capable server (most workloads are memory or disk I/O bound) • Conservatively assume .8x work done .5x power => 30% savings • 4.0x gains in work done/watt look attainable: • 1*1.32*1.16*1.90*1.30 => 3.8x (some overlap between CEMS & power dist savings) • Power is #3 expense in DC behind server h/w, power distribution & cooling • Data center capital expense savings nearly 100% driven by power • Reductions in power reduce capex, opex & is good for environment http://perspectives.mvdirona.com

11. Slides • These Slides: • http://mvdirona.com/jrh/TalksAndPapers/JamesRH_DCPowerSavingsFooCamp08.ppt • Perspectives Blog: • http://perspectives.mvdirona.com http://msblogs/JamesRH