Interconnection Network Design Contd.

1. Interconnection Network Design Contd. Adapted from UC, Berkeley Notes

2. Switching Techniques • Circuit Switching: A control message is sent from source to destination and a path is reserved. Communication starts. The path is released when communication is complete. • Store-and-forward policy (Packet Switching): each switch waits for the full packet to arrive in switch before sending to the next switch (good for WAN) • Cut-through routing or worm hole routing: switch examines the header, decides where to send the message, and then starts forwarding it immediately • In worm hole routing, when head of message is blocked, message stays strung out over the network, potentially blocking other messages (needs only buffer the piece of the packet that is sent between switches). CM-5 uses it, with each switch buffer being 4 bits per port. • Cut through routing lets the tail continue when head is blocked, storing the whole message into an intermmediate switch. (Requires a buffer large enough to hold the largest packet). CS258 S99

3. CS258 S99

4. Store and Forward vs. Cut-Through • Advantage • Latency reduces from function of:number of intermediate switches X by the size of the packet totime for 1st part of the packet to negotiate the switches + the packet size ÷ interconnect BW CS258 S99

5. Store&Forward vs Cut-Through Routing • h(n/b + D) vs n/b + h D • what if message is fragmented? • wormhole vs virtual cut-through CS258 S99

6. CS258 S99

7. Example Q. Compare the efficiency of store-and-forward (packet switching) vs. wormhole routing for transmission of a 20 bytes packet between a source and destination, which are d-nodes apart. Each node takes 0.25 microsecond and link transfer rate is 20 MB/sec. Answer: Time to transfer 20 bytes over a link = 20/20 MB/sec = 1 microsecond. Packet switching: # nodes x (node delay + transfer time)= d x (.25 + 1) = 1.25 d microseconds Wormhole: (# nodes x node delay) + transfer time = 0.25 d + 1 Book: For d=7, packet switching takes 8.75 microseconds vs. 2.75 microseconds for wormhole routing

8. Contention • Two packets trying to use the same link at same time • limited buffering • drop? • Most parallel mach. networks block in place • link-level flow control • tree saturation • Closed system - offered load depends on delivered CS258 S99

9. Delay with Queuing Suppose there are L links per node. Each link sends a packet to another link at “Lamda” packets/sec. The service rate (link+switch) is “Mue” packets per second. What is the delay over a distance D? Ans: There is a queue at each output link to hold extra packets. Model each output link as an M/M/1 queue with LxLamda input rate and Mue service rate. Delay through each link = Queuing time + Service time = S Delay over a distance D = S x D CS258 S99

10. Congestion Control • Packet switched networks do not reserve bandwidth; this leads to contention(connection based limits input) • Solution: prevent packets from entering until contention is reduced (e.g., freeway on-ramp metering lights) • Options: • Packet discarding: If packet arrives at switch and no room in buffer, packet is discarded (e.g., UDP) • Flow control: between pairs of receivers and senders; use feedback to tell sender when allowed to send next packet • Back-pressure: separate wires to tell to stop • Window: give original sender right to send N packets before getting permission to send more; overlaps latency of interconnection with overhead to send & receive packet (e.g., TCP), adjustable window • Choke packets: aka “rate-based”; Each packet received by busy switch in warning state sent back to the source via choke packet. Source reduces traffic to that destination by a fixed % (e.g., ATM) CS258 S99

11. Flow Control • What do you do when push comes to shove? • Ethernet: collision detection and retry after delay • FDDI, token ring: arbitration token • TCP/WAN: buffer, drop, adjust rate • any solution must adjust to output rate • Link-level flow control CS258 S99

12. Examples • Short Links • long links • several flits on the wire CS258 S99

13. Routing • Recall: routing algorithm determines • which of the possible paths are used as routes • how the route is determined • R: N x N -> C, which at each switch maps the destination node nd to the next channel on the route • Issues: • Routing mechanism • arithmetic • source-based port select • table driven • general computation • Properties of the routes • Deadlock feee CS258 S99

14. Routing Mechanism • need to select output port for each input packet • in a few cycles • Simple arithmetic in regular topologies • ex: Dx, Dy routing in a grid • west (-x) Dx < 0 • east (+x) Dx > 0 • south (-y) Dx = 0, Dy < 0 • north (+y) Dx = 0, Dy > 0 • processor Dx = 0, Dy = 0 • Reduce relative address of each dimension in order • Dimension-order routing in k-ary d-cubes • e-cube routing in n-cube CS258 S99

15. Routing Mechanism (cont) • Source-based • message header carries series of port selects • used and stripped en route • CRC? Packet Format? • CS-2, Myrinet, MIT Artic • Table-driven • message header carried index for next port at next switch • o = R[i] • table also gives index for following hop • o, I’ = R[i ] • ATM, HPPI P3 P2 P1 P0 CS258 S99

16. ` CS258 S99

17. Deadlock Freedom • How can it arise? • necessary conditions: • shared resource • incrementally allocated • non-preemptible • think of a channel as a shared resource that is acquired incrementally • source buffer then dest. buffer • channels along a route • How do you avoid it? • constrain how channel resources are allocated • ex: dimension order • How do you prove that a routing algorithm is deadlock free CS258 S99

18. Proof Technique • resources are logically associated with channels • messages introduce dependencies between resources as they move forward • need to articulate the possible dependences that can arise between channels • show that there are no cycles in Channel Dependence Graph • find a numbering of channel resources such that every legal route follows a monotonic sequence • => no traffic pattern can lead to deadlock • network need not be acyclic, on channel dependence graph CS258 S99

19. Example: k-ary 2D array • Theorem: x,y routing is deadlock free • Numbering • +x channel (i,y) -> (i+1,y) gets i • similarly for -x with 0 as most positive edge • +y channel (x,j) -> (x,j+1) gets N+j • similarly for -y channels • any routing sequence: x direction, turn, y direction is increasing CS258 S99

20. Deadlock free wormhole networks? • Basic dimension order routing techniques don’t work for k-ary d-cubes • only for k-ary d-arrays (bi-directional) • Idea: add channels! • provide multiple “virtual channels” to break the dependence cycle • good for BW too! • Do not need to add links, or xbar, only buffer resources • This adds nodes the the CDG, remove edges? CS258 S99

21. Breaking deadlock with virtual channels CS258 S99

22. Adaptive Routing • R: C x N x S -> C • Essential for fault tolerance • at least multipath • Can improve utilization of the network • Simple deterministic algorithms easily run into bad permutations • fully/partially adaptive, minimal/non-minimal • can introduce complexity or anomolies • little adaptation goes a long way! CS258 S99

23. Interconnection Topologies • Class networks scaling with N • Logical Properties: • distance, degree • Physcial properties • length, width • Fully connected network • diameter = 1 • degree = N • cost? • bus => O(N), but BW is O(1) - actually worse • crossbar => O(N2) for BW O(N) • VLSI technology determines switch degree CS258 S99

24. CS258 S99

25. CS258 S99

26. Linear Arrays and Rings • Linear Array • Diameter? • Average Distance? • Bisection bandwidth? • Route A -> B given by relative address R = B-A • Torus? • Examples: FDDI, SCI, FiberChannel Arbitrated Loop, KSR1 CS258 S99

27. Multidimensional Meshes and Tori • d-dimensional array • n = kd-1 X ...X kO nodes • described by d-vector of coordinates (id-1, ..., iO) • d-dimensional k-ary mesh: N = kd • k = dÖN • described by d-vector of radix k coordinate • d-dimensional k-ary torus (or k-ary d-cube)? 3D Cube 2D Grid CS258 S99

28. Real Machines • Wide links, smaller routing delay • Tremendous variation CS258 S99