Cutting the Electric Bill for Internet-Scale Systems

1. Cutting the Electric Bill for Internet-Scale Systems Andreas Andreou Cambridge University, R02 aa773@cam.ac.uk

2. What’s this all about? • Energy expenses are an increasingly important fraction of data center operating costs • Electricity prices show both temporal and geographical variation • Exploit variations in electricity prices for economic gain

3. Key observations • Electricity prices vary • Prices vary on an hourly basis • Often not well correlated at different locations • Substantial variations • Large distributed systems already incorporate request routing and replication • Dynamic request routing to map clients to servers • Mechanisms to replicate data necessary to process requests at multiples sites

4. Problem Specification • Large system composed of server clusters spread out geographically • Map client requests to clusters such that the total electricity cost is minimized • Assumptions • System fully replicated • Optimize for cost every hour • No knowledge of the future • Rate of change slow enough to be compatible with existing routing mechanisms • Fast enough to respond to electricity market fluctuations • Incorporate bandwidth and performance goals as constraints

5. Terminology • Energy Elasticity • Degree to which energy consumed by a cluster depends on the load placed on it • Ideally: no load, no power • Worst case: no difference between peak and idle power • State-of-the-art: idle power around 60% of peak • Differential Duration • Number of hours one location is favored over another by more than $5/MWh • PUE • Power usage effectiveness (measure of data center energy efficiency)

6. Background

7. Wholesale Electricity Markets (1) • Generation • Government and independent power producers • Coal (~50%), natural gas (~20%), nuclear power (~20%), hydroelectric generation (~6%) • Different regions, different power generation profiles • Transmission • Producers and consumers are connected to an electric grid • 8 reliability regions

8. Wholesale Electricity Markets (2) • Market Structure • Each region managed by Regional Transmission Organization (RTO) • RTO administer wholesale electricity markets • Auctioning mechanism: • Producers present supply offers • Consumers present demand bids • Coordinating body determines flow and sets prices • Market Types • Day-ahead markets • Real-time markets

9. Wholesale Electricity Markets (3) • Market Structure • Assumptions • Real-time prices are known and vary hourly • Electric bill is proportional to consumption and indexed to wholesale prices • Request routing behavior induced by our method doesn’t significantly alter prices and market behavior

10. Daily Variation

11. Different Market Types • Hourly real-time (RT) market is more volatile than day-ahead market

12. Hour-to-Hour Volatility

13. Geographic Correlation

14. Price Differentials

15. Differential Distributions

16. Time-of-Day

17. Differential Duration

18. Akamai: Traffic and Bandwidth • Over 2000 content provider customers in the US • 9-region traffic with electricity price data • Data covering 24 days worth of traffic • Traffic data of 5-minute intervals from public clusters • Bandwidth costs are significant • Aggressively optimized to reduce bandwidth costs • 95/5 billing model • Client-Server Distances • Use geographic distance as a coarse proxy for network performance

19. Cluster Energy Consumption (1) • Roughly linear to its utilization • Pidle : average idle power draw of single server • Ppeak : average peak power draw of single server • r: empirical derived constant • ut : average CPU utilization at time t • what is important in determining savings

20. Routing Energy • Increased path lengths will not alter energy consumption significantly • Average energy for a packet to pas through is on the order of 2mJ • Incremental energy dissipated by each packet passing through a core router would be as low as 50μJ per medium size packet • New routes may overload existing routers • Additional bandwidth could lead to upgrade • Can ignore by incorporating 95/5 bandwidth constraints

21. Simulation Strategy • Real-time market prices for 29 different locations • Traffic data for Akamai public clusters in 9 of those • Data set spanning Jan 2006 through Mar 2009 • Workload data set contains 5-minute samples in 25 cities • Period of 24 days and some hours • Discarded 7 and grouped remaining 18 cities to 9 clusters • Akamai’s geographic server distribution • Two routing schemes • Akamai’s original allocation • Distance constrained electricity price optimizer • Energy model as shown before

22. 24 Days of Traffic (1) • Energy Elasticity • Bandwidth Costs

23. 24 Days of Traffic (2) • Distance and savings

24. 39 Months of Prices • Derived from 24-day Akamai workload (US traffic only) • Dynamic beats static

25. Results • Existing systems can reduce energy costs be at least 2% without any increase in bandwidth costs or significant reduction in client performance • Google-like energy elasticity • Akamai-like server distribution • 95/5 bandwidth constraints • Savings increase with energy elasticity • Fully elastic system with relaxed bandwidth constraints can reduce energy cost be 30% (13% with bandwidth constraints) • Allowing increase of client-server distances leads to increased savings

26. Considerations (1) • Not reacting immediately to price changes noticebly reduces overall savings

27. Considerations (2) • Server operators should be able to negotiate contractual arrangements • Distributed systems with energy elastic clusters can be more flexible than traditional consumers • Triggered demand response programs

28. Future Work • Implementing Joint Optimization • RTO Interaction • Weather Differentials • Environmental Cost