Robust Design of Air Cooled Server Cabinets

1. Robust Design of Air Cooled Server Cabinets Nathan Rolander, Jeff Rambo, Yogendra Joshi, Farrokh Mistree ASME InterPACK Conference 19 July 2005 Systems Realization Laboratory Support for this work provided by the members of CEETHERM Microelectronics & Emerging Technologies Thermal Laboratory METTL

2. Background: What is a data center? • 10,000-500,000 sq. ft. facilities filled with cabinets which house data processing equipment, servers, switches, etc. • Tens to hundreds of MW power consumption for computing equipment and associated cooling hardware • Trend towards very high power density servers (30 kW/cabinet) requiring stringent thermal management 1 * Image: B. Tschudi, Lawrence Berkeley Laboratories

3. Introduction & Motivation • Up to 40% of data center operating costs can be cooling related* • Cooling challenges are compounded by a lifecycle mismatch: • New computer equipment introduced ~ 2 years • Center infrastructure overhauled ~ 25 years How do we efficiently integrate high powered equipment into an existing cabinet infrastructure while maximizing operational stability? 2 * Source: W. Tschudi, Lawrence Berkeley Laboratories

4. Cabinet Design Challenges • Flow complexity • The turbulent CFD models required to analyze the air flow distribution in cabinets are impractical to use iterative optimization algorithms • Operational stability • Variations in data center operating conditions, coupled with model inaccuracies mean computed “optimal” solutions do not translate to efficient or feasible physical solutions • Multiple design objectives • Objectives of efficient thermal management, cooling cost minimization, & operational stability are conflicting goals 3

5. Challenge Construct Integration Flow complexity POD* based turbulent modeling Thermally efficient & robust server cabinet design approach Operational stability Robust design principles Multiple objectives The compromise DSP** * Proper Orthogonal Decomposition ** Decision Support Problem Approach Overview • Integration of three constructs to tackle cabinet design challenges: 4

6. Introduction to the POD • Modal expansion of basis functions, : • Fit optimal linear subspace through a series of system observations, . • Maximize the projection of the basis functions onto the observations: φ u 5

7. Introduction to the POD • Modal expansion of basis functions, : • Fit optimal linear subspace through a series of system observations, . • Maximize the projection of the basis functions, onto the observations: φ u Constrained variational calculus problem < , > denotes ensemble averaging ( , )denotes L2 inner product 5

8. Introduction to the POD • Modal expansion of basis functions, : • Fit optimal linear subspace through a series of system observations, . • Maximize the projection of the basis functions onto the observations: φ u Assemble observations covariance matrix < , > denotes ensemble averaging ( , )denotes L2 inner product 5

9. Introduction to the POD • Modal expansion of basis functions, : • Fit optimal linear subspace through a series of system observations, . • Maximize the projection of the basis functions onto the observations: φ u < , > denotes ensemble averaging Take cross correlation tensor of covariance matrix ( , )denotes L2 inner product 5

10. Introduction to the POD • Modal expansion of basis functions, : • Fit optimal linear subspace through a series of system observations, . • Maximize the projection of the basis functions onto the observations: φ u < , > denotes ensemble averaging Take eigen-decomposition of the cross-correlation tensor ( , )denotes L2 inner product 5

11. POD Based Turbulent Flow Modeling • Vector-valued eigenvectors form empirical basis of m-dimensional subspace, called POD modes • Superposition of modes used to reconstruct any solution within the range of observations ~10% error* • Flux matching procedure applied at boundaries >> areas of known flow conditions, resulting in the minimization problem: • Values of found using method of least squares • Resulting model has ~O(105) reduction in DoF* Gis the flux goal F(.)is contribution to boundary flux from the POD modes ais the POD modeweight coefficient ai 6 * see: Rambo HT2005-72143 paper for complete analysis

12. Robust Design Principles • Determine superior solutions through minimizing the effects of variation, without eliminating their causes. • Type I – minimizing variations in performance caused by variations noise factors (uncontrollable parameters) • Type II – minimizing variations in performance caused by variation in control factors (design variables) • A common implementation of Type I robust design is Taguchi Parameter Design 7

13. Optimization minimizes the objective function: Deviation at Optimal Solution Deviation at Robust Solution Solution insensitivity is obtained by minimizing curvature: Optimal Solution Robust Solution nis the no. control variables Robust Design Application • Goals: Y Objective Function X Design Variable 8

14. Variability bounds in control parameters must be accounted for to to avoid infeasible solutions: Optimal Solution Bounds Robust Solution Bounds 2ΔX2 E (.)is the mean function Optimal Solution Robust Solution nis the no. control variables p is the no. constraints Robust Design Application • Constraints: X2 Feasible Design Space Design Variable Constraint Boundary X1 Design Variable 9

15. Mathematical Programming Goal Programming Goal function Objective function Inequality constraints G is a goal Equality constraints is under/over achievement of goal Minimization of Archimedean Deviation Function W is the goal weight vector The Compromise DSP Mathematics • Hybrid of Mathematical Programming and Goal Programming optimization routines: 10

16. Hot/Cold airflow W Ls Section c Qa,b,c Hs Fan H Dimensions: H = 2 m W = 0.9 m Ls = 0.6 m Hs = 2U ~ 0.09m Section b Section a Vin Problem Geometry • Enclosed Cabinet containing 10 servers • Cooling air supplied from under floor plenum Cabinet Profile Server Profile 11

17. Cabinet Modeling • 9 Observations of Vin = 0:0.25:2 m/s for POD • k-ε turbulence model for RANS implemented in commercial CFD software • Finite difference energy equation solver used for thermal solution, using POD computed flow field • 1 iteration ~ 12 sec Vin = 0.95 m/s 12

18. Hot/Cold airflow Response: Chip Temperatures (oC) W Section c H Section b Control Variables: Inlet air velocity, Vin[0, 1] m/s Section a chip power, Qa [0, 200] W Section b chip power, Qb [0, 200] W Section c chip power, Qc [0, 200] W Section a Vin Design Variables & Objectives Server Cabinet Model 13

19. Objective: Minimize ΔTmax & cooling energy for a given cabinet heat generation rate Qtotal Response: Chip Temperatures (oC) Goals: Minimize Inlet air velocity Minimize Chip Temperatures Minimize Chip Temperature Variation Constraints: Meet Target Cabinet Power Qtotal All Chip Temperatures < 85oC Control Variables: Inlet air velocity, Vin[0, 1] m/s Section a chip power, Qa [0, 200] W Section b chip power, Qb [0, 200] W Section c chip power, Qc [0, 200] W Design Variables & Objectives Server Cabinet Model 13

20. Response: Chip Temperatures (oC) Goals: Minimize Inlet air velocity Minimize Chip Temperatures Minimize Chip Temperature Variation Constraints: Meet Target Cabinet Power Qtotal All Chip Temperatures < 85oC Control Variables: Inlet air velocity, Vin[0, 1] m/s Section a chip power, Qa [0, 200] W Section b chip power, Qb [0, 200] W Section c chip power, Qc [0, 200] W Design Variables & Objectives iterate Server Cabinet Model 13

21. Results • Baseline vs. Maximum efficient power dissipation • Without server power re-distribution, increasing flow of cooling air alone is ineffective 14

22. Results • Inlet air velocity vs. Total cabinet power level • Cooling air is re-distributed to different cabinet sections depending upon supply rate >> server cooling efficiency 15

23. Results • Maximum chip temperature and bounds • Maximum chip temperature constraint met as variation in response changes with varying power & flow rates 16

24. Conclusions How do we efficiently integrate high powered equipment into an existing cabinet infrastructure while maximizing operational stability? 17

25. Conclusions • For the typical enclosed cabinet modeled, over 50% more power than baseline can be reliably dissipated through efficient configuration • Robust solutions account for variability in internal & external operating conditions, as well as a degree of modeling assumptions &inaccuracies • Server cabinet configuration design can be accomplished without center level re-design 17

26. Questions? • Thank you for attending! 18

27. Final Validation • Comparison of results obtained using robust design and compact model to FLUENT

28. Robust vs. Optimal Configuration • Pareto Frontier

29. Effects of Robust Solution • Optimal >> Robust :Temperature Variation

30. Effects of Robust Solution • Optimal >> Robust :Temperature Variation

31. System Model Control Factors, x Inlet air velocity, Vin[0, 1] m/s Section a chip power, Qa [0, 200] W Section b chip power, Qb [0, 200] W Section c chip power, Qc [0, 200] W Response, y Inlet Air Velocity (m/s) Chip Temperatures (oC) Total cabinet power (W) Signal Factors, M Inlet air velocity (minimize) Chip Temperatures (minimize) Cabinet Power (nominalize) Server Cabinet System Noise Factors, Z Inlet air temperature, Tin = 25 oC

32. Design Objective Specification • System Design Objectives >> Goals • Minimize flow rate of cooling air supplied to cabinet • Minimize server chip temperatures • Minimize sensitivity of configuration to changes in cabinet operating conditions • System Design Specifications >> Constraints • All server chips must operate at under 85oC • Total cabinet power must meet target value