Lecture 22 Interconnection Networks

1. Lecture 22 Interconnection Networks Computer Architecture COE 501

2. Network Media Twisted Pair: Copper, 1mm think, twisted to avoid attenna effect (telephone) Used by cable companies: high BW, good noise immunity Coaxial Cable: Plastic Covering Braided outer conductor Insulator Copper core Light: 3 parts are cable, light source, light detector. Multimode light disperse (LED), Single mode sinle wave (laser) Fiber Optics Total internal reflection Transmitter Air Receiver – L.E.D – Photodiode – Laser Diode light Silica source

3. Shared vs. Switched Media • Shared media: nodes share a single interconnection medium (e.g., Ethernet) • Only one message sent at a time • Inexpensive : one medium used by all processors • Limited bandwidth : medium becomes the bottleneck • Needs arbitration • Switched media : allow direct communication between source and destination nodes (e.g., ATM) • Multiple users at a time • More expensive : need to replicated medium • Higher total bandwidth • No arbitration • Added latency to go through the switch

4. Ethernet Protocol - Shared • Ethernet is an example of a shared media - similar to a bus • The ethernet nodes must coordinate so that only one message is sent at a time. • Nodes “listens” to the network to ensure it is not being used. • If the network is not being used, the node tries to send the message. • If another node tries to send a message at the same time, a collision occurs • Both nodes detect the collision and try to resend their messages after a random amount of time • Subsequent collisons result in exponentially increasing random time between attempts to resend data • What are advantages and disadvantages of this method?

5. Switch Topology • Switched media have a topology that indicate how nodes are connected • Topology determines • Degree: number of links from a node • Diameter: max number of links crossed between nodes • Average distance: number of links to random destination • Bisection: minimum number of links that separate the network into two halves • Bisection bandwidth: link bandwidth x bisection • Warning: Three-dimensional drawings must be mapped onto chips and boards which are essentially two-dimensional media • Elegant when sketched on the blackboard may look awkward when actually constructed

6. Common Topologies Type Degree Diameter Ave Dist Bisection 1D mesh 2 N-1 N/3 1 2D mesh 4 2(N1/2 - 1) 2N1/2 / 3 N1/2 3D mesh 6 3(N1/3 - 1) 3N1/3 / 3 N2/3 nD mesh 2n n(N1/n - 1) nN1/n / 3 N(n-1) / n Ring 2 N/2 N/4 2 2D torus 4 N1/2 N1/2 / 2 2N1/2 Hypercube Log2N n=Log2N n/2 N/2 2D Tree 3 2Log2N ~2Log2 N 1 Crossbar N-1 1 1 N2/2 N = number of nodes, n = dimension

7. Butterfly or Omega Network N/2 Butterfly ° ° ° • All paths equal length • Unique path from any input to any output • Try to avoid conflicts N/2 Butterfly ° ° °

8. Multistage Fat Tree • A multistage fat tree (CM-5) avoids congestion at the root node • Randomly assign packets to different paths on way up to spread the load • Increase degree, decrease congestion

9. Example Networks(Figure 7.19, pg 591) Name Number Topology Bits Clock Link Bis. BW Year nCube/ten 1-1024 10-cube 1 10 MHz 1.2 640 1987 iPSC/2 16-128 7-cube 1 16 MHz 2 345 1988 MP-1216 32-512 2D grid 1 25 MHz 3 1,300 1989 Delta 540 2D grid 16 40 MHz 40 640 1991 CM-5 32-2048 fat tree 4 40 MHz 20 10,240 1991 CS-2 32-1024 fat tree 8 70 MHz 50 50,000 1992 Paragon 4-1024 2D grid 16 100 MHz 200 6,400 1992 T3D 16-1024 3D Torus 16 150 MHz 300 19,200 1993 MBytes/second No standard topology!

10. Connection-Based vs. Connectionless • Connection-based system: • Telephone: operator sets up connection between the caller and the receiver • Once the connection is established, the line is held until it completes • Share transmission lines over long distances by using switches to multiplex several conversations on the same lines • “Time division multiplexing” divide B/W transmission line into a fixed number of slots, with each slot assigned to a conversation • Often referred to as circuit-swtiching • Problem: lines busy based on number of conversations, not amount of information sent • Advantage: reserved bandwidth - call not interrupted once started

11. Connection-Based vs. Connectionless • Connectionless system • Postal service : package routed to destination based on address • Each message is divided into packets, with an address per packet • Packets can each take different routes through the network • Often referred to as packet-switching • Disadvantage : additional information sent with each message

12. Store and Forward vs. Cut-Through • Store-and-forward policy: each switch waits for the full packet to arrive in the switch before it is sent on to the next switch • Cut-through routing or worm hole routing: switch examines the header, decides where to send the message, and then starts forwarding it immediately • In worm hole routing, when the head of the message is blocked the message stays strung out over the network, potentially blocking other messages (needs only buffer the size of the packet that is sent between switches). • Cut through routing lets the tail continue when the head is blocked, storing the whole message into a single switch. (Requires a buffer large enough to hold the largest packet).

13. Packet Formats • Fields: Destination, Checksum(C), Length(L), Type(T) • Data/Header Sizes in bytes: • CM5: (4 to 20)/4, Ethernet: (0 to 1500)/26, ATM: 48/5

14. Congestion Control • Packet switched networks do not reserve bandwidth; this leads to contention • Solution: prevent packets from entering until contention is reduced (e.g., entrance lights on interstate) • Options: • Packet discarding: If a packet arrives at a switch and there is no room in the buffer, the packet is discarded, SW detects and resends • Flow control: between pairs of receivers and senders; use feedback to tell the sender when it is allowed to send the next packet • Back-pressure: separate wires to tell to stop • Window: give the original sender the right to send N packets before getting permission to send more (TCP uses) • Choke packets: Each packet received by busy switch in warning state sent back to the source via choke packet. Source reduces traffic to that destination by a fixed % (ATM Forum)

15. Example Architectures Interconnect MPP LAN WAN Example CM-5 Ethernet ATM Topology “Fat” tree Line, Bus Variable, constructed from multistage switches Connection based? No No Yes Data Transfer Size Variable: Variable: Fixed: 4 to 20B 0 to 1500B 48B Store & Forward? No N/A Yes Congestion control At source: At source: Rate based Flow Listen for via choke control E-net idle packetsvia back pressure

16. 110 010 111 011 100 000 101 001 Routing • Deterministic—follows a pre-specified route • mesh: dimension-order routing • (x1, y1) -> (x2, y2) • first Dx = x2 -x1, • then Dy = y2 -y1, • hypercube: edge-cube routing • X = xox1x2 . . .xn -> Y = yoy1y2 . . .yn • R = X xor Y • Traverse dimensions of differing address in order • Adaptive—route based on network state (e.g., contention)

17. Practical Issues Interconnection MPP LAN WAN Example CM-5 Ethernet ATM Standard No Yes Yes Fault Tolerance? No Yes Yes Hot Insert? No Yes Yes • Standards: required for WAN, LAN! • Fault Tolerance: Can nodes fail and still deliver messages to other nodes? required for WAN, LAN! • Hot Insert: If the interconnection can survive a failure, can it also continue operation while a new node is added to the interconnection? required for WAN, LAN!

18. Protocols • Internetworking: allows computers on independent and incompatible networks to communicate reliably and efficiently; • Enabling technologies: SW standards that allow reliable communications without reliable networks • Hierarchy of layers, giving each layer responsibility for portion of overall communications task, called protocol families or protocol suites • Transmission Control Protocol/Internet Protocol (TCP/IP) • This protocol family is the basis of the Internet • IP : layer above interconnection network - routes datagram to destination machine • TCP : layer above IP - guaranees reliable, in-order delivery and prevents corruption of data

19. TCP/IP packet • Application sends message • TCP breaks into 64KB segements, adds 20B header • IP adds 20B header, sends to network • If Ethernet, broken into 1500B packets with headers, trailers • Header, trailers have length field, destination, window number, version, ... Ethernet IP Header TCP Header IP Data TCP data (< 64KB)

20. Protocol • Key to protocol families is that communication occurs logically at the same level of the protocol, called peer-to-peer, but is implemented via services at the lower level • Danger is each level increases latency

21. Connecting Networks • Bridges: connect LANs together, passing traffic from one side to another depending on the addresses in the packet. • operate at the Ethernet protocol level • usually simpler and cheaper than routers • Routers or Gateways: these devices connect LANs to WANs or WANs to WANs and resolve incompatible addressing. • Generally slower than bridges, they operate at the internetworking protocol (IP) level • Routers divide the interconnect into separate smaller subnets, which simplifies manageability and improves security

22. SummaryInterconnection Networks • Shared vs. Switched media • Switch topology indicates how nodes are connected • Switching network can be either connection-based (circuit switched) or connectionless (packet switched) • In packet-switched networks congestion control is required • Protocols allow different computers on different networks to communicate reliably