Router Design

1. Router Design • Overview of Generic Router Architecture • Input-Queued Switches (Routers) • IP Address Look-up Algorithms • Packet Classification Algorithms Readings:Do required readings (you can skip the math in Section V and Appendix in [Mc+99]); Also do some of the optional readings if interested Q: Any volunteers for scribes? CSci5221: Router Design

2. . . . . . . Routers in a Network CSci5221: Router Design

3. Sample Routers and Switches Cisco 12416 Routerup to 160 Gb/s throughput up to 10 Gb/s ports Juniper Networks T640 Router up to 160 Gb/s throughput up to 10 Gb/s ports 3Com 495024 port gigabit Ethernet switch CSci5221: Router Design

4. High Capacity Router • Cisco CRS-1 • up to 46 Tb/s thruput • two rack types • line card rack • 640 Gb/s thruput • up to 16 line cards • up to 40 Gb/s each • up to 72 racks • switch rack • central switch stage • up to 8 racks • in-service scaling CSci5221: Router Design

5. CP OPP IPP II OI outputqueue routingtable . . . . . . Components of a Basic Router • Input/Output Interfaces (II, OI) • convert between optical signals and electronic signals • extract timing from received signals • encode (decode) data for transmission • Input Port Processor (IPP) • synchronize signals • determine required OI or OIs from routing table • Output Port Processor (OPP) • queue outgoing cells • shared bus interconnects IPPs and OPPs • Control Processor (CP) • configures routing tables • coordinates end-to-end channel setup together with neighboring routers CSci5221: Router Design

6. Data Data Data Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Address Table Address Table Address Table Generic Router Architecture 1 1 Queue Packet Buffer Memory 2 2 Queue Packet Buffer Memory N N Queue Packet Buffer Memory CSci5221: Router Design

7. Switch Fabric: Three Design Approaches CSci5221: Router Design

8. Memory Input Port Output Port System Bus Switch Fabric: First Generation Routers • Traditional computers with switching under direct control of the CPU • Packet copied to the system’s memory • Speed limited by the memory bandwidth (two bus crossings per packet) CSci5221: Router Design

9. CPU Buffer Memory Route Table CPU Line Interface Line Interface Line Interface Memory MAC MAC MAC Shared Memory (1st Generation) Shared Backplane Line Interface Typically < 0.5Gbps aggregate capacity Limited by rate of shared memory CSci5221: Router Design

10. Switch Fabric: Switching Via a Bus • Packet from input port memory to output port memory via a shared bus • Bus contention: switching speed limited by bus bandwidth • 1 Gbps bus, Cisco 1900: sufficient speed for access and enterprise routers (not regional or backbone) CSci5221: Router Design

11. CPU Buffer Memory Route Table Line Card Line Card Line Card Buffer Memory Buffer Memory Buffer Memory Fwding Cache Fwding Cache Fwding Cache MAC MAC MAC Shared Bus (2nd Generation) Typically < 5Gb/s aggregate capacity; Limited by shared bus CSci5221: Router Design

12. Switch Fabric: Interconnection Network • Banyan networks, other interconnection nets initially created for multiprocessors • Advanced design: fragmenting packet into fixed length cells to send through the fabric • Cisco 12000: switches Gbps through the interconnection network CSci5221: Router Design

13. Switched Backplane Line Card CPU Card Line Card Local Buffer Memory Local Buffer Memory Line Interface CPU Routing Table Memory Fwding Table Fwding Table MAC MAC Point-to-Point Switch (3rd Generation) Typically < 50Gbps aggregate capacity CSci5221: Router Design

14. Buffer Placement: Output Port Queuing • Buffering when the aggregate arrival rate exceeds the output line speed • Memory must operate at very high speed CSci5221: Router Design

15. Link 1, ingress Link 1, egress Link 2, ingress Link 2, egress Link 3, ingress Link 3, egress Link 4, ingress Link 4, egress Simple model of output queued switch Link rate, R Link rate, R R R R R R R CSci5221: Router Design

16. Characteristics of an output queued (OQ) switch • arriving packets immediately written into output queue, without intermediate buffering • flow of packets to one output does not affect flow to another output • OQ switch is work conserving: output line always busy when there is a packet in switch for it • OQ switch has highest throughput, lowest average delay CSci5221: Router Design

17. Data Data Data Hdr Hdr Hdr 1 1 Header Processing Header Processing Header Processing Queue Packet Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Buffer Memory Address Table Address Table Address Table N times line rate 2 2 Queue Packet Buffer Memory N times line rate N N Queue Packet Buffer Memory Switching Speed-up Needed CSci5221: Router Design

18. Buffer Placement: Input Port Queuing • Fabric slower than input ports combined • So, queuing may occur at input queues • Head-of-the-Line (HOL) blocking • Queued packet at the front of the queue prevents others in queue from moving forward CSci5221: Router Design

19. Link 2 Link 1 R1 Link 3 Link 4 Simple model of input queued switch Link 1, ingress Link 1, egress R R Link 2, ingress Link 2, egress R R Link 3, ingress Link 3, egress R R Link 4, egress Link 4, ingress R R CSci5221: Router Design

20. Cannot be transferred because is blocked by red packet Cannot be transferred because output buffer full Head-of-line Blocking • Packet at the head of an input queue cannot be transferred, thus blocking the following packets (or cells – packets of fixed size) Input 1 Output 1 Input 2 Output 2 Input 3 Output 3 CSci5221: Router Design

21. Characteristics of an input queued (IQ) switch • arriving packets written into input queue • only one packet can be sent to output link at a time • head-of-line blocking • IQ switch cannot keep output links fully utilized CSci5221: Router Design

22. Buffer Placement: Design Trade-offs • Output queues • Pro: work-conserving, so maximizes throughput • Con: memory must operate at speed N*R • Input queues • Pro: memory can operate at speed R • Con: head-of-line blocking for access to output • Work-conserving: output line is always busy when there is a packet in the switch for it • Head-of-line blocking: head packet in a FIFO cannot be transmitted, forcing others to wait CSci5221: Router Design

23. Large input-queued switch with • single FIFO at each input • packet destinations i.i.d. (independently, identically distributed), uniform across outputs • HoL blocked packets not flushed • throughput analysis • saturated switch (i.e., always arrival at each input queue) • ball/urns model: N balls, N urns • focus on first urn • Xt - number of balls in urn at time t • Dt- number balls removed from all ums at end of time t • Dt is switch thruput What is capacity of IQ: Model[optional: Karol et al Globecom’86] CSci5221: Router Design

24. Model (cont’d) • At+1 - no. balls dropped into urn 1 at t+1 • Xt+1 = (Xt-1)+ + At+1 • where • E(Dt) = ρN where ρ is output throughput • for large N, binomial distribution can be approximated by Poisson distribution, CSci5221: Router Design

25. Model (cont’d) where EA = ρ, E(A2) = ρ + ρ2 therefore EX = 1, therefore and ρ =2-√2 58.6% CSci5221: Router Design

26. A Router with Input QueuesHead of Line Blocking The best that any queueing system can achieve. CSci5221: Router Design

27. Input 1 Output 1 Output 2 Input 2 Output 3 Input 3 Solution to Avoid Head-of-line Blocking • How to improve capacity without increasing switching fabric speed ? • Maintain at each input N virtual queues, i.e., one per output • use non-FIFO scheduler, matching input/output CSci5221: Router Design

28. matching scheduler . . . . . . Virtual Output Queueing 1 1 • assume fixed length packets • each input manages separate queue per output • at each time, matching scheduler finds best possible packets from inputs to said to outputs • maximum-weight matching N N CSci5221: Router Design

29. Matching • bipartite graph (V1 x V2,E), E V1xV2 • V1,V2 inputs, outputs; (i,j) E iff Lij(t) > 0 • Lij(t): no. of packets at input i for output j at t • matching:subset of E such that no two edges are adjacent • maximal matching: no more edges can be added output input • An aside: • stability(of a [queueing] system): • Assuming the arrival rate is (i.e., # of arrivals per unit of time) less or equal to the system capacity • The system is stable if and only if no queue grows infinitely (under any arrival patterns) as t CSci5221: Router Design

30. Matching problems • maximum size matching • matching with largest number of edges • when traffic uniform, provides 100% utilization • network flow problem, O(N5/2) • maximum weight matching • add weight wij to edge from i to j • e.g., wij: # of packets from input i to output j in the queue • matching with highest weight • when wij = Lij(t) provides 100% utilization • equivalent to a network flow problem, O(N3) • MWM algorithms involve backtracking: i.e. edges laid down in one iteration may be removed later algorithm not amenable to pipelining CSci5221: Router Design

31. 19 19 18 1 7 Practical Maximal Matchings Max Size Matching Max Wt Matching  Stable  Not stable  Not stable Scheduling Algorithms 19 3 4 21 1 18 7 CSci5221: Router Design

32. 19 19 18 1 7 Better performance Max Wt Matching Max Size Matching Easier to implement Switch Algorithms Maximal matching Not stable Not stable Stable, low backlogs CSci5221: Router Design

33. Better Matching Algorithms • Need simple algorithms that perform well • efficient packet processing packets at line speeds • high throughput • low latencies/backlogs • Randomized algorithms with linear complexity available • Tassiulas’ Randomized Algorithm • LAURA • SERENA Use both randomization, history, problem structure and arrival information For more details, see optional reading [SGP02]: “Efficient Randomized Algorithms for Input-Queued Switch Scheduling” by Shah, Giaccone and Prabhakar, IEEE Micro Vol 22, Issue 1, Jan 2002 CSci5221: Router Design

34. input interface output interface Backplane RO C Combined Input-Output Queued (CIOQ) Routers • Both input and output interfaces store packets • Advantages • Easy to built • Utilization 1 can be achieved with limited input/output speedup (<= 2) • Disadvantages • Harder to design algorithms • Two congestion points • Need to design flow control CSci5221: Router Design

35. Output Queue Emulation using CIOQ (with Speed-up) Stable Marriage Problem -- Gale Shapely Algorithm (GSA) • As long as there is a free man m • m proposes to highest ranked women w in his list he hasn’t proposed yet • If w is free, m an w are engaged • If w is engaged to m’ and w prefers m to m’, w releases m’ • Otherwise m remains free • A stable matching exists for every set of preference lists • Complexity: worst-case O(N2) CSci5221: Router Design

36. Stable Marriage Problem • Consider N women and N men • Each woman/man ranks each man/woman in the order of their preferences • Stable matching, a matching with no blocking pairs • Blocking pair; let p(i) denote the pair of i • There are matched pairs (k, p(k)) and (j, p(j)) such that k prefers p(j) to p(k), and p(j) prefers k to j CSci5221: Router Design

37. men pref. list women pref. list 1 2 4 3 1 2 1 4 3 2 3 4 3 2 1 4 1 2 4 3 1 1 4 3 2 2 3 1 4 2 3 1 2 3 4 4 2 1 4 3 Example • If men propose to women, the stable matching is • 1st round: (1,2), (2,1), (3,4), (4,1) -> w1 releases m2 • 2nd round: (2,4) ->w4 releases m3; • 3rd round: (3,3); • final match: (1,2), (2,4), (3,3), (4,1) • What is the stable matching if women propose to men? CSci5221: Router Design

38. OQ Emulation with a Speedup of 2 • Each input and output maintains a preference list • Input preference list: list of cells at that input ordered in the inverse order of their arrival • Output preference list: list of all input cells to be forwarded to that output ordered by the times they would be served in an Output Queueing schedule • Use GSA to match inputs to outputs • Outputs initiate the matching • Can emulate all work-conserving schedulers For more info, see the optional reading [C+99] “Matching Output Queueing with a Combined Input Output Queued Switch.” CSci5221: Router Design

39. to/from link Transmit Receive FIB to/from switch Line Cards • Interfacing • Physical link • Switching fabric • Packet handling • Packet forwarding (FIB) • Packet filtering (ACLs) • Buffer management • Link scheduling • Rate-limiting • Packet marking • Measurement CSci5221: Router Design

40. Data Hdr Data Hdr IP Address Next Hop Address Table Buffer Memory Line Card: Abstract view Header Processing Lookup IP Address Update Header Queue Packet CSci5221: Router Design

41. FIB 4.0.0.0/8 4.83.128.0/17 12.0.0.0/8 12.34.158.0/24 126.255.103.0/24 destination 12.34.158.5 outgoing link Serial0/0.1 Line Cards: Longest-Prefix Match Forwarding • Forwarding Information Base in IP routers • Maps each IP prefix to next-hop link(s) • Destination-based forwarding • Packet has a destination address • Router identifies longest-matching prefix • Pushing complexity into forwarding decisions CSci5221: Router Design

42. Line Cards: Packet Forwarding Evolution • Software on the router CPU • Central processor makes forwarding decision • Not scalable to large aggregate throughput • Route cache on the line card • Maintain a small FIB cache on each line card • Store (destination, output link) mappings • Cache misses handled by the router CPU • Full FIB on each line card • Store the entire FIB on each line card • Apply dedicated hardware for longest-prefix match CSci5221: Router Design

43. “Five tuple” for access control lists (ACLs) Source and destination IP addresses TCP/UDP source and destination ports Protocol (e.g., UDP vs. TCP) Should arriving packet be allowed in? Departing packet let out? Line Cards: Packet Filtering With Access Control Lists CSci5221: Router Design

44. ACL Examples • Filter packets based on source address • Customer access link to the service provider • Source address should fall in customer prefix • Filter packets based on port number • Block traffic for unwanted applications • Known security vulnerabilities, peer-to-peer, … • Block pairs of hosts from communicating • Protect access to special servers • E.g., block the dorms from the grading server  CSci5221: Router Design

45. Line Cards: Mapping Traffic to Classes • Gold traffic • All traffic to/from President’s IP address • All traffic to/from the port number for DNS • Silver traffic • All traffic to/from academic and administrative buildings • Bronze traffic • All traffic on the public wireless network • Then, schedule resources accordingly • 50% for gold, 30% for silver, and 20% for bronze CSci5221: Router Design

46. Flat address Ethernet: 48 bit MAC address ATM: 28 bit VPI/VCI DS-0: timeslot location Limited scalability High speed lookup Hierarchical address IP <network>.<subnet>.<host> Telephone: country.area.home Scalable Easy lookup if boundary is fixed telephony Difficult lookup if boundary is flexible longest prefix match for IP Addressing and Look-up CSci5221: Router Design

47. Year Line 40Byte packets (Mpkt/s) 1997 622Mb/s 1.94 1999 2.5Gb/s 7.81 2001 10Gb/s 31.25 2003 40Gb/s 125 • lookup mechanism must be simple, easy to implement • memory access time long-term bottleneck Lookups Must be Fast CSci5221: Router Design

48. Memory Technology (2003-04) Note: price, speed, power manufacturer and market dependent CSci5221: Router Design

49. Lookup Mechanism is Protocol Dependent CSci5221: Router Design

50. Exact Matches in Ethernet Switches • layer-2 addresses usually 48-bits long • address global, not just local to link • range/size of address not “negotiable” • 248 > 1012, therefore cannot hold all addresses in table and use direct lookup CSci5221: Router Design