Your Data Center Is a Router: The Case for Reconfigurable Optical Circuit Switched Paths

1. Your Data Center Is a Router: The Case for Reconfigurable Optical Circuit Switched Paths

2. Core Switch End of Row Switch Top of Rack Switch Data Center Network • Today’s Data Center Network Picture from: James Hamilton, Architecture for Modular Data Centers • Data intensive applications are experiencing bandwidth bottleneck in the tree structure data center networks. • E.g. Video data processing, MapReduce …

3. Tree FatTree BCube Picture from: Ken Hall, Green Data Centers Full bisection bandwidth solutions • Re-structure data center network to provide full bisection bandwidth among all the servers. • Complicated network structure, hard to construct and expand.

4. Full bisection bandwidth may not be necessary • Temporal Traffic Locality • Applications might hit CPU, disk IO or Sync bounds. • e.g. MapReduce • Spatial Traffic Locality • Nodes only communicate with a small number of partners. • e.g. Earthquake simulation • Many measurement studies have suggested evidence of traffic locality. • [SC05][WREN09][IMC09][HotNets09] Full bisection bandwidth solutions provide too much with high costs.

5. A B C D E F An alternative design: hybrid data center network • Hybrid network may give us best of both worlds: • Optical circuit-switched paths for data intensive transfer. • Electrical packet-switched paths for timely delivery. Electrical packet-switched network Optical circuit-switched network

6. Optical Circuit Switching • MEMS Optical Switching Module Picture from: http://www.ntt.co.jp/milab/en/project/pr05_3Dmems.html Switching at whatever rate modulated on input/output ports Up to tens of ms physical reconfiguration time

7. 40G, 100Gbps technology has been developed. 15.5Tbps over a single fiber! Price data from: Joe Berthold, Hot Interconnects’09 Optical Channels • Ultra-high bandwidth • Dropping prices

8. Advantage: • Simple and flexible: easy to construct, expand and manage • Ultra-high bandwidth • Low power • Disadvantage: • Fat pipes are not all-to-all. • Reconfiguration overhead Optical circuits in datacenters A B C D E F A - E, B - D, C - F A - D, B - E, C - F A - F, B - E, C - D

9. Research questions • Enough traffic locality in data centers to leverage optical path? • Reconfigure optical paths fast enough to meet dynamic traffic? • How to integrate optical circuits into data centers at low costs? • How to manage and leverage optical paths? • How do applications behave over the hybrid network?

10. 10 sec TM 10s 10s 10s … Time Is there enough traffic locality? • Analyzing production data center traffic trace: • 7 racks, 155 servers, 1060 cores • One week NetFlow traces collected at all servers • Configure 3 optical paths out of total 21 cross-rack paths with maximum optical traffic, reconfigure every 10s. Traffic locality: a few optical paths have the potential to offload significant amount of traffic from electrical networks.

11. Graph G: (V, E) R1 R2 R3 R4 R5 R6 R7 R8 wxy= vol(Rx, Ry) + vol(Ry, Rx) R1 R2 w12 R1 R2 w27 w14 R4 w43 R3 R3 Optical path configuration is a maximum weight perfect matching on graph G. R5 w35 w38 w47 R4 R8 w36 w68 R5 R6 R7 R6 Solved by polynomial time Edmonds’ algorithm[1]! R7 R8 [1] J. Edmonds, Paths, trees and flowers, Canadian J. of Mathematics, pp 449-467, 1965 Can optical paths be reconfigured fast enough? - Optical Path Configuration Algorithm

12. Can optical paths be reconfigured fast enough? - Optical Path Configuration Time • Several time factors • Computation time • 640ms for a 1000-rack data center using Edmonds’ algorithm. • Signaling time • < 1ms in data centers • Physical reconfiguration time • Up to tens ms for MEMS optical switches Even in very large data centers, optical paths can still be reconfigured at small time scales (< 1 sec).

13. How to manage optical paths in data centers? • Routing over dynamicdual-path (electrical/optical) network: • Ethernet Spanning Tree? • NO, dual paths will be blocked • Link State Routing? • NO, long routing convergence time after reconfiguration

14. VLAN1: Electrical VLAN2: Optical How to manage optical paths in data centers? • VLAN based dual-path routing: • Advantages: • Leverage both electrical and optical paths by tagging packets • No route convergence delay after optical reconfiguration • No need to modify switches

15. Servers • Extensive buffering at servers • Traffic demands measurement • Aggregate traffic and batch for optical transfer • Per-rack virtual output queuing: • Avoid head-of-line blocking User Apps Kernel Per-rack Virtual Output Queue Scheduler Network Interface How to manage optical paths in data centers? • How to measure application traffic demand?

16. Stats • Configurable virtual output queue scheduler to control traffic to optical paths. • A centralized manager to control the optical path configuration. Config Config Stats Per-rack Virtual Output Queue Config Scheduler How to manage optical paths in data centers? • How to configure optical paths and schedule traffic to them? Servers User Apps Daemon Configuration Manager Kernel A B C D Switches with VLAN settings Traffic Network Interface

17. Challenges • TCP/IP reacting to optical path reconfiguration. • Potential long delays caused by extensive queuing at servers. • Collecting traffic demand from a million servers. • Choosing the right buffer sizes and reconfiguration intervals.

18. Summary • Adding optical circuit switched paths into data centers. • Potential benefits: • A simpler and flexible data center network design. • Relieving data intensive applications from network bottlenecks.