Chapter 12: Green Data Centers

1. HANDBOOK ON GREEN INFORMATION AND COMMUNICATION SYSTEMS Chapter 12: Green Data Centers Yan Zhang and Nirwan Ansari Advanced Networking Laboratory New Jersey Institute of Technology Newark, NJ 07102

2. Power Consumption of Data Centers • In 2007, Environmental Protection Agency (EPA) Report to Congress on Server and Data Center Energy Efficiency assessed trends in the energy usage and energy costs of data centers and servers in U.S. • Based on the power consumption of data centers in U.S. from year 2000 to 2006, the power consumption of data centers is predicted under five different scenarios: • Two baseline prediction scenarios: historical trend scenario, current efficiency trend scenario. • Three energy-efficiency scenarios: improved operation scenario, best practice scenario, and state-of-the-art scenario.

3. Power Consumption of Data Centers Improved operation trend: utilizes any essentially operational technologies requiring little or no capital investment to improve energy efficiency beyond “current efficiency trends”. Best practice trend: adopts more widespread technologies and practices in the most energy-efficient facilities in operation. State-of-the-art trend: maximizes the energy efficiency of data centers using the most energy-efficient technologies and best management practices available today. • This prediction was performed with the total power consumption of the installed base of servers, external disk drivers, and network ports in data centers multiplied by a power overhead factor caused by the power usage of power distribution and cooling infrastructure in data centers. Historical trend: simply estimates the power consumption trends based on the observed power usage from year 2000 to 2006. Data centers and servers in U.S. consumed about 61 billion kWh in 2006 for a total electricity cost of about $4.5 billion. The energy use of data centers and servers in 2006 was more than doubled the electricity that was consumed by data centers in 2000. It is estimated the energy usage of data centers could nearly double again in 2011 to more than 100 billion kWh with historical and current efficiency trends. Current efficiency trend: estimates the power usage trajectory of U.S. servers and data centers by considering the observed efficiency trends for IT equipment and site infrastructure systems.

4. Energy Efficiency of Data Centers • APC White Paper #6 (Ref [2]) investigated the total cost of ownership (TCO) of physical data center infrastructure, and found that the cost of electrical power consumption contributed to about 20% of the total cost. • Numerous studies have shown (Ref [3][4]): • Data center servers rarely operate at full utilization. • The average server utilization is often below 30 percent of the maximum utilization in data centers. • At low levels of workload, servers are highly energy-inefficient.

5. Energy Efficiency of Data Centers Typical servers: consume about half of its full power at the idle state. Power proportional servers: consume about half of its full power at the idle state. Server power usage and energy efficiency at varying utilization levels, from idle to peak performance (L. A. Barroso and U. Hölzle, “The Case for Energy-Proportional Computing,”Computer, 40(12):33–37, Dec. 2007).

6. Energy Efficiency of Data Centers A typical data center power usage (adapted from M. Ton, B. Fortenbery, and W. Tschudi, “DC Power for Improved Data Center Efficiency,”http://hightech.lbl.gov/documents/DATA_CENTERS/DCDemoFinalReport.pdf, Mar. 2008).

7. Energy Efficiency Metrics for Data Centers • In order to quantify the energy efficiency of data centers, several energy efficiency metrics have been proposed to help data center operators to improve the energy efficiency and reduce operation costs of data centers: • Power usage effectiveness (PUE) and data center infrastructure efficiency (DCiE) • Data Center energy Productivity (DCeP) • Datacenter Performance Per Energy (DPPE) • Green Grid Productivity Indicator

8. Power Usage Effectiveness (PUE) • The most commonly used metric to indicate the energy efficiency of a data: PUE and its reciprocal DCiE. • PUE definition: • PUE=1.0 implies: there is no power overhead and all power consumption of the data center goes to the IT equipment. • PUE measures the total power consumption overhead caused by the data center facility support equipment, including the cooling systems, power delivery, and other facility infrastructure like lighting.

9. Power Usage Effectiveness (PUE) • The average data center PUE in the US in 2006 is 2.0, implying that one Watt of overhead power is used to cool and deliver every Watt to IT equipment (Ref[1]). • It also predicts that “state-of-the-art" data center energy efficiency could reach a PUE of 1.2 (Ref [66]). • Google publishes quarterly the PUE results from data centers with an IT load of at least 5MW and time-in-operation of at least 6 months (Ref [67]). • The twelve-month, energy-weighted average PUE result obtained in the first quarter of 2011 is 1.16, which exceeds the EPA's goal for state-of-the-art data center efficiency.

10. Data Center energy Productivity (DCeP) • Energy efficiency and energy productivity are closely related to each other. • Energy efficiency focuses on reducing unnecessary power consumption to produce a work output. • Energy productivity of a data center measures the quantity of useful work done relative to the amount of power consumption of a data center in producing this work. • DCeP allows the continuous monitoring of the productivity of a data center as a function of power consumed by a data center. • DCeP metric tracks the overall work product of a data center per unit of power consumption expended to produce this work.

11. Datacenter Performance Per Energy (DPPE) • DPPE evaluates the energy efficiency of data centers as a whole. The DPPE metric indicates data center productivity per unit energy. • DPPE defines four sub-metrics: • IT Equipment Utilization (ITEU) • ITEE (IT Equipment Energy Efficiency) • PUE • GEC (Green Energy Coefficient) • These four sub-metrics reflect four kinds of independent energy-saving efforts, and are designed to prevent one kind of energy-saving effort from affecting others.

12. Datacenter Performance Per Energy (DPPE) • ITEU • It measures the degree of energy saving by efficient operation of IT equipment through virtual techniques and other operational techniques. • ITEE • It is defined as the ratio of the total capacity of IT equipment to the total rated power of IT equipment. • This metric aims to encourage the installation of equipment with high processing capacity per unit electric power in data centers to promote energy savings.

13. Datacenter Performance Per Energy (DPPE) • PUE: • It indicates the power saving for data center facilities. • The less power consumption of facility infrastructure, the smaller the value of PUE. • GEC: • It is defined as the ratio of the Green Energy produced and used in a data center to its total power consumption. • The value of GEC becomes larger if the production of non-CO2 energy is increased in a data center. • DPPE • Considering the definitions of the above four sub-metrics, DPPE incorporates these four sub-metrics and can be expressed as a function of them as follows:

14. Green Grid Productivity Indicator • Green Grid Productivity Indicator is a multi-parameter framework to evaluate overall data center efficiency. • Through the use of a radial graph, relevant indicators such as DCiE, data center utilization, server utilization, storage utilization, and network utilization can be quickly, concisely and flexibly emerged to provide organizational awareness. • How it works: • Set up the target value for each indicator. • Plotting the peak and average values of each indicator during the period of monitoring, together with their target and theoretical maximum values on a radial graph. • Assess how well the data center resources are utilized, • Check if the business targets are achieved visually and quickly, • Figure out how to spend their efforts to maximize the benefits.

15. Green Grid Productivity Indicator Examples of using the Green Grid indicator tool. (adopted from The Green Grid. “The Green Grid Productivity Indicator,” http://www.thegreengrid.org/~/media/WhitePapers/White_Paper_15_-TGG_Productivity_Indicator_063008.pdf?lang=en).

16. Techniques to Improve Energy Efficiency of Data Centers • IT Infrastructure Improvements • Servers and Storages • Network Equipment • Power Distribution • Smart Cooling and Thermal Management • Power Management Techniques • Provisioning • Consolidation • Virtualization • Others

17. IT Infrastructure Improvements • Approximately 40% - 60% power consumption of a data center is devoted to IT infrastructure, which consists of servers, storage, and network equipment • Servers and Storages • CPU • Dynamic Voltage/Frequency Scale (DVFS): 23% - 36% energy savings. • memory and disk • power shifting (Ref [14]): re-budget the available power between processor and memory. Power shifting is a threshold-based throttling scheme to limit the number of operations performed by each subsystem during an interval of time, but power budget violations and unnecessary performance degradation may be caused by improper interval length. • Mini-rank (Ref [16]): an adaptive DRAM architecture to limit power consumption of DRAM by breaking a conventional DRAM rank into multiple smaller mini-ranks with a small bridge chip. • Dynamic Rotations per Minute (DRPM) (Ref [17]): a low-level hardware-based technique to dynamically modulate disk speed to save power in disk drives since the slower the disk drive spins the less power it consumes.

18. IT Infrastructure Improvements • Energy proportional systems • PowerNap (Ref [19]): • Attune the server power consumptions to server utilization patterns. • Transit rapidly between a high-performance active state and a minimal-power nap state in response to instantaneous load. • PowerNap can be modeled as an M/G/1 queuing system.

19. IT Infrastructure Improvements • Network Equipment: switches, routers, wireless access points. • Sleeping mode • Transit into the low-power sleep mode when no transmission is needed, and return back to the active mode when transmission is requested. • The transition time overhead of putting a device into and out of the sleep mode may reduce energy efficiency significantly. • Rate-adapting • The lower the line-speed is, the less power the devices consume. • Adapt the transmission rate of network operation to the offered workload. • Speed negotiation is required in the rate-adaption scheme for both of the transmission ends.

20. Power Distribution • Current typical power delivery systems for data centers still use alternating current (AC) power: • Distributed from utility to the facility, and is then stepped down via transformers and delivered to uninterruptible power supplies (UPS). • Several levels of power conversion exist in both data center facilities and within IT equipment that results in significant electrical power losses, including power losses in UPS, transformers, and power line losses. • DC power distribution system has been demonstrated and evaluated for data centers (Ref [32]).

21. Smart Cooling and Thermal Management • Most data centers use liquid cooling for computer room air conditioning (CRAC). • Rack-level liquid-cooling solutions bring chilled water or liquid refrigerant closer to the servers. • Rear-door liquid cooling • Sealed rack liquid cooling • In-row liquid-coolers • Overhead liquid-coolers • Data center liquid cooling techniques tend to use naturally-cooled water, like lake or sea water • Electrical savings by eliminating or reducing the need for water chillers in data centers.

22. Smart Cooling and Thermal Management • The predominant air cooling scheme for current data centers is to use the CRAC units and an under-floor cool air distribution system.

23. Power Management - Provisioning • Provisioning is an effective solution to reduce the power consumption by turning off the idle servers, storages, and network equipment, or by putting them into a lower power mode. • An adaptive dynamic server provisioning technique (Ref [49]) • Effective to dynamically turn on a minimum number of servers required to satisfy application specific quality of service and load dispatching. • Tailored for long-lived connection-intensive Internet services. • A power-proportional cluster (Ref [50]) • Consists of a power-aware cluster manager and a set of heterogeneous machines. • Uses currently available energy-efficient hardware, mechanisms for transiting in and out of low-power sleep states, and dynamic provisioning and scheduling to minimize power consumption. • Especially tailored for short lived request-response type of workloads.

24. Power Management - Provisioning • Sierra (Ref [53]) • A power-proportional, distributed storage system. • Turning off a fraction of storage servers during trough traffic period. • Utilizing a set of techniques: power-aware layout, predictive gear scheduling, and a replicated short-term store, to maintain data consistency and fault-tolerance as well as system performance. • Rabbit (Ref [54]) • A power-proportional distributed file system • Provides ideal power-proportionality for large-scale cluster-based storage and data-intensive computing systems by using a new cluster-based storage data layout. • Rabbit can maintain near ideal power proportionality even with node failures.

25. Power Management - Provisioning • ElasticTree (Ref [55]) • Dynamically adjust the set of active network elements, links and switches, to satisfy changing data center traffic loads. • Given the data center network topology, the traffic demand matrix, and the power consumption of each link and node, ElasticTree minimizes the total power consumption of a data center by solving a capacitated multi-commodity cost flow (CMCF) optimization problem. • Urja (Ref [56]) • A network wide energy monitoring tool • Integrated with network management operations to collect configuration and traffic information from live network switches and to accurately predict their power consumption.

26. Power Management - Consolidation • Power savings with application consolidation • Application consolidation in cloud computing (Ref [57]): • The energy performance trade-offs for consolidation. • The application consolidation problem can be modeled as a modified bin-packing problem, and the optimal points exist. • Generic application-layer energy optimization (Ref [58]): • Guides the design choices by using energy profiles of various resource components of an application. • Intelligent data placement and/or data migration can be used to save energy in storage systems. • Hibernator (Ref [59]): • A disk array energy management system. • Several techniques to reduce power consumption while maintaining performance goals, including disk drives that rotate at different speeds and migration of data to an appropriate-speed disk drive.

27. Power Management - Virtualization • Effective to enhance server utilization, consolidate servers and reduce the total number of physical servers. • Power consumption caused by servers is reduced. • Cooling requirement should also be reduced comparably. • Effective to build energy proportional storage systems. • Sample-Replicate-Consolidate Mapping (SRCMap) (Ref [61]) • A storage virtualization solution for energy-proportional storage. • Consolidating the cumulative workload on a minimal subset of physical volumes proportional to the I/O workload intensity. • GreenCloud (Ref [62]) • Enabling comprehensive online monitoring, live virtual machine migration, and virtual machine placement optimization to reduce data center power consumption while guaranteeing the performance goals.

28. Others • Energy-aware routing (Ref[63]) • An energy-aware routing optimization model. • The objective is to find a route for a given traffic matrix that minimizes the total number of switches. • The proposed energy-aware routing model is NP-hard, and a heuristic algorithm was required to solve the energy-aware routing problem. • Energy proportional datacenter network architecture (Ref[64]) • A flattened butterfly data center topology is inherently more power efficient than the other commonly proposed topology for high-performance data centers.

29. Conclusions • Power consumption is a central critical issue for data centers. • As reported in 2005, the electricity usage of data centers has been almost doubled from 2000 to 2005. • The electricity cost accounts for about 20% of the total cost of data centers. • Numerous studies have shown that the average server utilization is often below 30% of the maximum utilization in data centers. • To quantify the energy efficiency of data centers, several energy efficiency metrics have been proposed: • PUE, DCiE, DCeP, DPPE, and Green Grid Productivity Indicator. • Techniques to improve energy efficiency of data centers.

30. Thanks for your attention!