Scalable Data Center Network Design with Jellyfish Topology

1. CMPE 252A : Computer Networks Chen Qian Computer Engineering UCSC Baskin Engineering

2. Jellyfish:Networking Data Centers Randomly Paper by Ankit Singla, et.al . NDSI 2012. Some figures are from slides presented by  Chi-Yao Hong, UIUC.

3. Facebook • ‘Add capacity On a DAILY BASIS’ • Amazon http://news.netcraft.com/archives/2013/05/20/amazon-web-services-growth-unrelenting.html

4. Fat-Tree Topology Incremental growth??

5. Structured networks

6. Fat tree: Structure VS Limit • N_switches: • 3-level Fat tree : 5k2/4 • for fat tree usingk-port switches • 24-port  3456 hosts • 32-port  8192 hosts • 48-port  27648 hosts • What for 10000 hosts? • Over utilize? Leave unused ports?

7. Goals • Bandwidth & Capacity • Better VM Placement  Reduce Traffic • Better Topology  Avoid Bottleneck • Robustness Failure resistance • Flexbility: Incremental Expansion • Easy to add VM • Easy to remove VM No structure = no restriction

8. Jellyfish : no structure

9. Topology of jellyfish networks for 432 severs, 180 switches, degree = 12

10. Random graph • Regular Graph • RG(n,r) • Each vertex has the same degree r • Random Regular Graph • Random sampled from all RG(n,r) • Hard to generate • Question: How to generate?

11. Not-so-uniformRandom-RG(n,r) :: RRG(n,r) • Procedure to modify RRG(n-1,r) to RRG(n,r) • r=3 • RRG(4,3) • RRG(5,3)

13. Goals • Bandwidth & Capacity • Better VM Placement  Reduce Traffic • Better Topology  Avoid Bottleneck • Incremental Expansion • Easy to add VM • Easy to remove VM

14. About the Evaluation • bisection bandwidth: Theoretical calculation for RRG, • Bollobas’ theoretical lower bounds • Throughput: random permutation traffic • Each host choose one to send (at full speed)

15. Jellyfish VS LEGUP

16. Vs. FatTreeBisection bandwidth Jellyfish: larger B-bandwidth using same # switches & servers Jellyfish: more servers under the same B-bandwidth and # switches

17. Lower cost

18. Better failure resilience

19. Larger Throughput

20. Jellyfish vs. Small World • Smallworld: grid + random Small World Ring (2 reg + 4 rand) Small World 2D Torus (4 reg + 2 rand) Small World 3D Hexagon Torus (5 reg + 1 rand)

22. Reason of better performance

26. Better than jellyfish ??? • More hosts using same # of switches? • Connecting more switches , each of which has same # ports, (limit the diameter) • How many switches can be connected , with 3 switch-to-switch ports , and switch-to-switch path length <= 2? • Petersen Graph

27. Degree-diameter-graph

28. Degree-Diameter Graph have (nearly) highest throughput • Jellyfish is only little bit worse.

29. But… • Practical constraint: • Routing / Congestion Control • Cable

30. Routing & Congestion Control • Utilize capacity without structure • no layers! • Routing : • ECMP: fail to provide large path diversity • K shortest path: • Congestion Control • TCP/ multipath TCP If all available capacity is fully utilized,

31. K-shortest path • Different Path • S-e1-e2-e3-…ex...-en-T • S-e1-e2-e3-…ey…-em-T • Algorithm to find 2nd-shortest path: • Find a shortest path P from S to T in G • For each e in P • …Remove e from G • …Calculate shortest path on G , namely SP(e) • …add e back to Graph • Return min(SP(e)) O(k2N*ShortestPath(N))

32. K-shortest path forwarding • Shortest Paths (S,T):SAB1C1DT, SAB2C2DT, SAB3C2DT, (B1,T) C1 A A B C D S T A (S,T) A A A (B4,T) C2 A A (A,T) B1 B2 B3

33. Inter-switch link’s path count in ECMP and k-shortestpath routing for random permutation traffic at the server-level on a typical Jellyfish of 686 servers. For each link, we count the number of distinct paths it is on.

34. Multi Path TCP (MPTCP) http://blogs.citrix.com/2013/08/23/networking-beyond-tcp-the-mptcp-way/

35. Packet simulation results for different routing and congestion control protocols

37. cabling • Jellyfish uses 20% less # cables ,

38. Cabling in large data centers • Topology generated automatically, • Cables connected manually.. ( 10% of cost) • Error detect : link-layer discovery protocol.

39. Jellyfish of Jellyfish • Restrict some connections in pod • Result: 2-layered random Graph

40. Jellyfish of Jellyfish • Restrict some connections in pod • Result: 2-layered random Graph

41. Cables between pods can be aggregated

42. Conclusion • Bandwidth & Capacity • Incremental Expansion • Lower Cost • Limitation: slow to compute forwarding paths. Large forwarding tables.

43. Space Shuffle:A Scalable, Flexible, and High-BandwidthData Center Network Ye Yu and Chen Qian

45. Motivation: Goals of Data Center Design High-bandwidth • Data center applications generates high internal & external communication Flexibility • Adding servers and expanding network bandwidth incrementally. Scalability • Routing and Forwarding should rely on small forwarding state.

46. Motivation:Existing Data Center Architectures No shortest paths. Does not support multipath well. Greedy Routing Random Interconnection K-shortest path routing is inefficient. Big forwarding state.

47. Motivation:Goal of Space Shuffle (S2) • How to build a flexible data center architecture that achieves high-throughput and scalability ? • Approach: Greedy routing on random interconnection. • Challenges: • How to build a random interconnection that enables greedy routing? • How does the greedy routing protocol achieve high-throughput and near-optimal path length?

48. Outline • Motivation • Space Shuffle Data Center Topology • The Routing Protocol in Space Shuffle Data Center • Discussion & Evaluation

49. S2 Topology Construction-Assign Servers • Servers and Top-Of-Rack switches. • Uniformly assign servers to switches. • Connect servers to switches. • The rest ports are used for inter-switch connections.

50. S2 Topology Construction:-Virtual Coordinates