1 / 56

Chapter 10: Scalable Interconnection Networks

Chapter 10: Scalable Interconnection Networks. Goals. Low latency to neighbors to average/furthest node High bandwidth per-node and aggregate bisection bandwidth .. Low cost switch complexity, pin count wiring cost (connectors!) Scalability. Requirements from Above.

aleda
Download Presentation

Chapter 10: Scalable Interconnection Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 10:Scalable Interconnection Networks EECS 570: Fall 2003 -- rev1

  2. Goals • Low latency • to neighbors • to average/furthest node • High bandwidth • per-node and aggregate • bisection bandwidth .. • Low cost • switch complexity, pin count • wiring cost (connectors!) • Scalability EECS 570: Fall 2003 -- rev1

  3. Requirements from Above • Communication-to-computation ratio implies bandwidth needs • local or global? • regular or irregular (arbitrary)? • bursty or uniform? • broadcasts? multicasts? • Programming Model • protocol structure • transfer size • importance of latency vs. bandwidth EECS 570: Fall 2003 -- rev1

  4. Basic Definitions switch is a device capable of transferring data from input ports to output ports in an arbitrary pattern network is a graph V={ switches/nodes} connected by communication channels (aka links) C  V x V • direct – node at each switch • indirect – node only on edge (like internet) • route: sequence of links message follows from node A to node B EECS 570: Fall 2003 -- rev1

  5. What characterizes a network? • Topology • interconnection structure of the graph • Switching Strategy • circuit vs. packet switching • store-and-forward vs. cut-through • virtual cut-through vs. wormhole, etc. • Routing Algorithm • how is route determined • Control Strategy • centralized vs. distributed • Flow Control Mechanism • when does a packet (or a portion of it) move along its route • can't send two packets down same link at same time EECS 570: Fall 2003 -- rev1

  6. Topologies Topology determines many critical parameters: • degree: number of input (output) ports per switch • distance: number of links in route from A to B • average distance: average over all (A,B) pairs • diameter: max distance between two nodes (using shortest paths) • bisection: min number of links to separate into two halves EECS 570: Fall 2003 -- rev1

  7. Bus • Fully connected network topology • bus plus interface logic is a form of switch • Parameters: • diameter = average distance = 1 • degree = N • switch cost O(N) • wire cost constant • bisection bandwidth constant (or worse) • Broadcast is free EECS 570: Fall 2003 -- rev1

  8. Crossbar • Another fully connected network topology • one big switch of degree N connects all nodes • Parameters: • diameter = average distance = 0(1) • degree=N • switch cost O(N2) • wire cost 2N • bisection bandwidth O(N) • Most switches in other topologies are crossbars inside EECS 570: Fall 2003 -- rev1

  9. How to build a crossbar EECS 570: Fall 2003 -- rev1

  10. Input Buffer Output Buffer Receiver Transmitter Cross-bar Input Ports Output Ports Control Routing, Scheduling Switches EECS 570: Fall 2003 -- rev1

  11. Linear Arrays and Rings • Linear Array • Diameter? N -1 • Avg. Distance? N/3 • Bisection? 1 • Torus (ring): links may be unidirectional or bidirectional • Examples: FDDI, SCI, FiberChannel, KSR1 EECS 570: Fall 2003 -- rev1

  12. Multidimensional Meshes and Tori • d-dimensional k-ary mesh N = kd k = d√N • nodes in each of d dimensions • torus has wraparound, array doesn't; “mesh” ambiguous • general d-dimensional mesh • n = kd-1 x ...x k0 nodes EECS 570: Fall 2003 -- rev1

  13. Properties (N=kd,k=d√N) • Diameter? • Average Distance? • d x k/3 for mesh • Degree • Switch Cost? • Wire Cost? • Bisection? • kd+1 EECS 570: Fall 2003 -- rev1

  14. Hypercubes EECS 570: Fall 2003 -- rev1

  15. Trees Usually indirect, occasionally direct Diameter and avg. distance are logarithmic • k-ary tree, height d = logk N Fixed degree Route up to common ancestor and down • benefit for local traffic Bisection? EECS 570: Fall 2003 -- rev1

  16. Fat-Trees Fatter links (really more of them) as you go up, so bisection BW scales with N EECS 570: Fall 2003 -- rev1

  17. Butterflies • Tree with lots of roots ! • N log N (actually N/2 x logN) • Exactly one route from any source to any dest • R = A xor B, at level i use “straight” edge if ri=o, otherwise cross edge • Bisection N/2 vs N(d-1)/d EECS 570: Fall 2003 -- rev1

  18. Benes Networks and Fat Trees • Back-to-back butterfly can route all permutations • off line • What if you just pick a random mid point? EECS 570: Fall 2003 -- rev1

  19. Relationship of Butterflies to Hypercubes • Wiring is isomorphic • Except that Butterfly always lakes log n steps EECS 570: Fall 2003 -- rev1

  20. Summary of Topologies Topology Degree Diameter Ave Dist Bisection Diam/Ave Dist N=1024 1DArray 2 N-1 N/3 1 1023/341 1D Ring 2 N/2 N/4 2 512/2 2DArray 4 2(N½ -1) ⅔N½ N½ 63/21 3D Array 6 3(N⅓ -1) N⅓ N⅔ -30/-10 2DTorus 4 N½½N 2N½ 32/16 k-ary n-cube 2n nk/2 nk/4 nk/4 15/7.5 @ n=3 Hypercube n=logN n n/2 N/2 10/5 2DTree 3 2log2N -2Iog2N 1 20/-20 4DTree 5 2log4N -2Iog4N-2/3 1 10/-9 2D fat tree 4 log2N -2Iog2N N 20/-20 2D butterfly 4 log2N log2N N/2 20/20 EECS 570: Fall 2003 -- rev1

  21. Choosing a Topology Cost vs. performance For fixed cost which topology provides best performance? • best performance on what workload? • message size • traffic pattern • define cost • target machine size Simplify tradeoff to dimension vs. radix • restrict to k-ary d-cubes • what is best dimension? EECS 570: Fall 2003 -- rev1

  22. How Many Dimensions in a Network? d=2 or d=3 • Short wires, easy to build • Many hops, low bisection bandwidth • Benefits from traffic locality d>=4 • Harder to build, more wires, longer average length • Higher switch degree • Fewer hops, better bisection bandwidth • Handles non-local traffic better Effect of # of hops on latency depends on switching strategy... EECS 570: Fall 2003 -- rev1

  23. Store&Forward vs. Cut-Through Routing • messages typ. fragmented into packets & pipelined • cut-through pipelines on flits EECS 570: Fall 2003 -- rev1

  24. Handling Output Contention What if output is blocked? • virtual cut-through • switch w/blocked output buffers entire packet • degenerates to: • requires lots of buffering in switches • wormhole • leave flits strung out over network (In buffers) • minimal switch buffering • one blocked packet can tie up lots of channels EECS 570: Fall 2003 -- rev1

  25. Traditional Scaling: Latency(P) Assumes equal channel width • independent of node count or dimension • dominated by average distance EECS 570: Fall 2003 -- rev1

  26. Average Distance but equal channel width is not equal cost! Higher dimension => more channels average distance = d(k-1)/2 EECS 570: Fall 2003 -- rev1

  27. In the 3-D world For n nodes, bisection area is O(n2/3 ) For large n, bisection bandwidth is limited to O(n2/3 ) • Dally, IEEE TPDS, [Dal90a] • For fixed bisection bandwidth, low-dimensional k-ary n-cubes are better (otherwise higher is better) • i.e., a few short fat wires are better than many long thin wires • What about many long fat wires? EECS 570: Fall 2003 -- rev1

  28. Equal cost in k-ary n-cubes Equal number of nodes? Equal number of pins/wires? Equal bisection bandwidth? Equal area? Equal wire length? What do we know? switch degree: d diameter = d(k-1) total links = Nd pins per node = 2wd bisection = kd-1 = N/k links in each direction 2Nw/k wires cross the middle EECS 570: Fall 2003 -- rev1

  29. Latency(d) for P with Equal Width • total links(N)=Nd EECS 570: Fall 2003 -- rev1

  30. Latency with Equal Pin Count Baseline d=2, has w = 32 (128 wires per node) fix 2dw pins => w(d) = 64/d distance down with increasing d, but channel time up EECS 570: Fall 2003 -- rev1

  31. Latency with Equal Bisection Width • N-node hypercube has N bisection links • 2d torus has 2N½ • Fixed bisection => w(d) = N1/d/2= k/2 • 1 M nodes, d=2 has w=512! EECS 570: Fall 2003 -- rev1

  32. Larger Routing Delay (w/equal pin) • ratio of routing to channel time is key EECS 570: Fall 2003 -- rev1

  33. Topology Summary Rich set of topological alternatives with deep relationships Design point depends heavily on cost model • nodes, pins, area, … • Need for downward scalability lends to fix dimension • high-degree switches wasted in small configuration • grow machine by increasing nodes per dimension Need a consistent framework and analysis to separate opinion from design Optimal point changes with technology • store-and-forward vs. cut-through • non-pipelined vs. pipelined signaling EECS 570: Fall 2003 -- rev1

  34. Real Machines • Wide links, smaller routing delay • Tremendous variation EECS 570: Fall 2003 -- rev1

  35. What characterizes a network? Topology • interconnection structure of the graph Switching Strategy • circuit vs. packet switching • store-and-forward vs. cut-through • virtual cut-through vs. wormhole, etc, Routing Algorithm • how is route determined Control Strategy • centralized vs. distributed Flow Control Mechanism • when does a packet (or a portion of it) move along its route • can't send two packets down same link at same time EECS 570: Fall 2003 -- rev1

  36. Typical Packet Format Two basic mechanisms for abstraction • encapsulation • fragmentation EECS 570: Fall 2003 -- rev1

  37. Routing Recall: routing algorithm determines • which of the possible paths are used as routes • how the route is determined • R: N x N → C, which at each switch maps the destination node nd to the next channel on the route Issues: • Routing mechanism • arithmetic • source-based port select • table driven • general computation • Properties of the routes • Deadlock free EECS 570: Fall 2003 -- rev1

  38. Routing Mechanism need to select output port for each input packet • in a few cycles • Simple arithmetic in regular topologies • ex: ∆x, ∆y routing in a grid • west(-x) ∆x<0 • east (+x) ∆x>0 • south(-y) ∆x=0, ∆y<0 • north(+y) ∆x=0, ∆y>0 • processor ∆x=0, ∆y=0 Reduce relative address of each dimension in order • Dimension-order routing in k-ary d-cubes • e-cube routing in n-cube EECS 570: Fall 2003 -- rev1

  39. Routing Mechanism (cont) P3 P2 P1 P0 Source-based • message header carries series of port selects • used and stripped en route • CRC? header length … • CS-2, Myrinet, MIT Artic Table-driven • message header carried index for next port at next switch • o = R[i] • table also gives index for following hop • , I’ = R[i] • ATM, HPPI EECS 570: Fall 2003 -- rev1

  40. Properties of Routing Algorithms Deterministic • route determined by (source, dest), not intermediate state (i.e. traffic) Adaptive • route influenced by traffic along the way Minimal • only selects shortest paths Deadlock free • no traffic pattern can lead to a situation where no packets move forward EECS 570: Fall 2003 -- rev1

  41. Deadlock Freedom How can it arise? • necessary conditions: • shared resource • incrementally allocated • non-pre-emptible • think of a channel as a shared that is acquired incrementally • source buffer then dest. buffer • channels along a route How do you avoid it? • constrain how channel resources are allocated • ex: dimension order How do you prove that a routing algorithm is deadlock free? EECS 570: Fall 2003 -- rev1

  42. Proof Technique Resources are logically associated with channels Messages introduce dependences between resources as they move forward Need to articulate possible dependences between channels Show that there are no cycles in Channel Dependence Graph • find a numbering of channel resources such that every legal route follows a monotonic sequence => no traffic pattern can lead to deadlock Network need not be acyclic, only channel dependence graph EECS 570: Fall 2003 -- rev1

  43. Example: k-ary 2D array Theorem: dimension-order x,y routing is deadlock free • Numbering • +x channel (i,y) → (i+1,y) gets i • similarly for -x with 0 as most positive edge • +y channel (x,j) -> (x,j+ I) gets N+j • similarly for -y channels Any routing sequence' x direction, turn, y direction is increasing EECS 570: Fall 2003 -- rev1

  44. Channel Dependence Graph EECS 570: Fall 2003 -- rev1

  45. More examples Why is the obvious routing on X deadlock free? . • butterfly? • tree? • fat tree? Any assumptions about routing mechanism? amount of buffering? What about wormhole routing on a ring? EECS 570: Fall 2003 -- rev1

  46. Deadlock free wormhole networks? Basic dimension-order routing doesn't work for k-ary d-cubes • only for k-ary d-arrays (bi-directional, no wrap-around) Idea: add channels! • provide multiple “virtual channels” to break dependence cycle • good for BW too! • Don't need to add links, or x bar, only buffer resources • This adds nodes to the CDG, remove edges? EECS 570: Fall 2003 -- rev1

  47. Breaking deadlock with virtual channels EECS 570: Fall 2003 -- rev1

  48. Turn Restrictions in X, Y • XY routing forbids 4 of 8 turns and leaves no room for adaptive routing • Can you allow more turns and still be deadlock free EECS 570: Fall 2003 -- rev1

  49. Minimal turn restrictions in 2D +y +x -x -y EECS 570: Fall 2003 -- rev1 north-last negative first

  50. Example legal west-first routes • Can route around failures or congestion • Can combine turn restrictions with virtual channels EECS 570: Fall 2003 -- rev1

More Related