Introduction to High-Performance Internet Switches and Routers

1. Introduction to High-Performance Internet Switches and Routers

2. Network Architecture Long Haul Network DWDM Core 10GbE Core Core Routers 10GbE Core Routers Metropolitan Metropolitan Campus / Residential 10GbE Edge switch Edge Routers 10GbE GbE • • • • • • Access switch Access Routers

3. pop pop pop pop

4. How the Internet really is: Current Trend Modems, DSL SONET/SDH DWDM

5. The Internet is a mesh of routers mostly interconnected by (ATM and) SONET (and DWDM) TDM TDM TDM TDM Circuit switched crossconnects, DWDM etc.

6. Core Router Core Router ATM Switch ATM Switch MUX MUX SONET/SDH ADM SONET/SDH ADM SONET/SDH DCS SONET/SDH DCS SONET/SDH ADM SONET/SDH ADM MUX MUX ATM Switch ATM Switch Core Router Core Router Typical (BUT NOT ALL) IP Backbone (Late 1990’s)

7. POP3 POP2 POP1 D POP4 A B E POP5 POP6 C POP7 POP8 F Points of Presence (POPs)

8. Where High Performance Routers are Used (2.5 Gb/s) R2 (2.5 Gb/s) R1 R6 R5 R4 R7 R3 R9 R10 R8 R11 R12 R14 R13 R16 R15 (2.5 Gb/s) (2.5 Gb/s)

9. Hierarchical arrangement End hosts (1000s per mux) Access multiplexer Edge Routers POP Core Routers 10Gb/s “OC192” POP POP POP: Point of Presence. Richly interconnected by mesh of long-haul links. Typically: 40 POPs per national network operator; 10-40 core routers per POP. Point of Presence (POP)

10. Typical POP Configuration Transport Network DWDM/SONET Terminal Backbone routers 10G WAN Transport Links > 50% of high speed interfaces are router-to-router (Core routers) 10G Router-Router Intra-Office Links Aggregation switches/routers (Edge Switches)

11. Today’s Network Equipment Routers Switches SONET DWDM LAYER 3 LAYER 2 LAYER 1 LAYER 0 Internet FR & ATM SONET DWDM Protocol

12. Interconnect Egress linecard Ingress linecard Route lookup Buffer ing Framing TTL process ing Framing Buffer ing QoS schedul ing Interconnect scheduling Control plane Control path Data path Scheduling path Functions in a packet switch

13. Ingress linecard Egress linecard Interconnect Framing Framing Interconnect scheduling Control plane Control path Data path Functions in a circuit switch

14. Our emphasis for now is to look at packet switches (IP, ATM, Ethernet, framerelay, etc.)

15. What a Router Looks Like Cisco GSR 12416 Juniper M160 19” 19” Capacity: 160Gb/sPower: 4.2kW Capacity: 80Gb/sPower: 2.6kW 6ft 3ft 2ft 2.5ft

16. A Router Chassis Fans/ Power Supplies Linecards

17. A Circuit Board with connectors for line cards High speed electrical traces connecting line cards to fabric Usually passive Typically 30-layer boards Backplane

18. Line Card Picture

19. Cisco Catalyst 3750G Cisco CRS-1 What do these two have in common?

20. CRS-1 linecard 20” x (18”+11”) x 1RU 40Gbps, 80MPPS State-of-the-art 0.13u silicon Full IP routing stack including IPv4 and IPv6 support Distributed IOS Multi-chassis support Cat 3750G Switch 19” x 16” x 1RU 52Gpbs, 78 MPPS State-of-the-art 0.13u silicon Full IP routing stack including IPv4 and IPv6 support Distributed IOS Multi-chassis support What do these two have in common?

21. Cisco Catalyst 3750G Cisco CRS-1 What is different between them?

22. CRS-1 linecard Up to 1024 linecards Fully programmable forwarding 2M prefix entries and 512K ACLs 46Tbps 3-stage switching fabric MPLS support H-A non-stop routing protocols Cat 3750G Switch Up to 9 stack members Hardwired ASIC forwarding 11K prefix entries and 1.5K ACLs 32Gbps sharedstack ring L2 switching support Re-startable routing applications A lot…

23. Other packet switches Cisco 7500 “edge” routers Lucent GX550 Core ATM switch DSL router

24. D D R3 R1 R4 D A B E R2 C R5 Destination Next Hop F D R3 E R3 F R5 What is Routing?

25. R3 R1 R4 D D A D D 1 4 16 32 D Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset B E TTL Protocol Header Checksum 20 bytes Source Address R2 C R5 Destination Address Destination Next Hop F D R3 Options (if any) E R3 Data F R5 What is Routing?

26. R3 R1 R4 D A B E R2 C R5 F What is Routing?

27. Basic Architectural Elementsof a Router • Routing • Routing table update • (OSPF, RIP, IS-IS) • Admission Control • Congestion Control • Reservation Control Plane “Typically in Software” • Routing • Lookup • Packet • Classifier • Switching • Arbitration • Scheduling Switch (per-packet processing) “Typically in Hardware” Switching

28. Basic Architectural ComponentsDatapath: per-packet processing 3. 1. Output Scheduling 2. Forwarding Table Interconnect Forwarding Decision Forwarding Table Forwarding Decision Forwarding Table Forwarding Decision

29. Per-packet processing in a Switch/Router 1. Accept packet arriving on an ingress line. 2. Lookup packet destination address in the forwarding table, to identify outgoing interface(s). 3. Manipulate packet header: e.g., decrement TTL, update header checksum. 4. Send packet to outgoing interface(s). 5. Queue until line is free. 6. Transmit packet onto outgoing line.

30. ATM Switch • Lookup cell VCI/VPI in VC table. • Replace old VCI/VPI with new. • Forward cell to outgoing interface. • Transmit cell onto link.

31. Ethernet Switch • Lookup frame DA in forwarding table. • If known, forward to correct port. • If unknown, broadcast to all ports. • Learn SA of incoming frame. • Forward frame to outgoing interface. • Transmit frame onto link.

32. IP Router • Lookup packet DA in forwarding table. • If known, forward to correct port. • If unknown, drop packet. • Decrement TTL, update header Cksum. • Forward packet to outgoing interface. • Transmit packet onto link.

33. Special per packet/flow processing • The router can be equipped with additional capabilities to provide special services on a per-packet or per-class basis. • The router can perform some additional processing on the incoming packets: • Classifying the packet • IPv4, IPv6, MPLS, ... • Delivering packets according to a pre-agreed service: Absolute service or relative service (e.g., send a packet within a given deadline, give a packet a better service than another packet (IntServ – DiffServ)) • Filtering packets for security reasons • Treating multicast packets differently from unicast packets

34. Packet processing must be simple and easy to implement • Memory access time is the bottleneck • 200Mpps × 2 lookups/pkt = 400 Mlookups/sec → 2.5ns per lookup Per packet Processing Must be Fast !!!

35. CPU Buffer Memory Route Table CPU Line Interface Line Interface Line Interface Memory MAC MAC MAC Typically <0.5Gb/s aggregate capacity First Generation Routers Shared Backplane Line Interface

36. Bus-based Router Architectures with Single Processor • The first generation of IP router • Based on software implementations on a single general-purpose CPU. • Limitations: • Serious processing bottleneck in the central processor • Memory intensive operations (e.g. table lookup & data movements) limits the effectiveness of processor power • A severe limiting factor to overall router throughput from input/output (I/O) bus

37. Fwding Cache Second Generation Routers CPU Buffer Memory Route Table Line Card Line Card Line Card Buffer Memory Buffer Memory Buffer Memory Fwding Cache Fwding Cache MAC MAC MAC Typically <5Gb/s aggregate capacity

38. Bus-based Router Architectures with Multiple Processors • Architectures with Route Caching • Second generation IP routers • Distribute packet forwarding operations • Network interface cards • Processors • Route caches • Packets are transmitted once over the shared bus • Limitations: • The central routing table is a bottleneck at high-speeds • traffic dependent throughput • shared bus is still a bottleneck

39. Limitation of IP Packet Forwarding based on Route Caching • Routing changes invalidate existing cache entries and need re-establishment • The performance depends on: • a. how big the cache • b. how the cache is maintained • c. what the performance of the slow path is • Solution: • Using a forwarding database in each network interface • Benefit: • Performance, Scalability, Network resilience, and Functionality

40. Fwding Table Third Generation Routers Switched Backplane Line Card CPU Card Line Card Local Buffer Memory Local Buffer Memory Line Interface CPU Routing Table Memory Fwding Table MAC MAC Typically <50Gb/s aggregate capacity

41. Switch-based Router Architectures with Fully Distributed Processors • To avoid bottlenecks: • Processing power • Memory bandwidth • Internal bus bandwidth • Each network interface is equipped with appropriate processing power and buffer space.

42. Fourth Generation Routers/SwitchesOptics inside a router for the first time Optical links 100s of metres Switch Core Linecards 0.3 - 10Tb/s routers in development

43. Juniper TX8/T640 Alcatel 7670 RSP TX8 Avici TSR Chiaro

44. Next Gen. Backbone Network Architecture – One backbone, multiple access networks Dual Stack IPv4-IPv6 Cable Network CE router (G)MPLS based Multi-service Intelligent Packet Backbone Network Dual Stack IPv4-IPv6 Enterprise Network PE PE Router (Service POP) Service POP Residential PE DSL, FTTH, Dial CE router GGSN SGSN CE router ISP’s IPv6 IX Telecommuter Dual Stack IPv4-IPv6 DSL/FTTH/Dial access Network ISP offering Native IPv6 services • One Backbone Network • Maximizes speed, flexibility and manageability

45. Data Hdr Data Hdr IP Address Next Hop Address Table Buffer Memory ~1M prefixes Off-chip DRAM ~1M packets Off-chip DRAM Current Generation: Generic Router Architecture Header Processing Lookup IP Address Update Header Queue Packet

46. Data Data Data Data Data Data Hdr Hdr Hdr Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header 1 1 Address Table Address Table Address Table 2 2 N N Current Generation: Generic Router Architecture (IQ) Queue Packet Buffer Memory Queue Packet Buffer Memory Scheduler Queue Packet Buffer Memory

47. Data Data Data Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Address Table Address Table Address Table Current Generation: Generic Router Architecture (OQ) 1 1 Queue Packet Buffer Memory 2 2 Queue Packet Buffer Memory N N Queue Packet Buffer Memory

48. Basic Architectural Elementsof a Current Router Typical IP Router Linecard Buffer & State Memory Scheduler Buffer Mgmt & Scheduling Physical Layer Framing & Maintenance Buffered or Bufferless Fabric (e.g. crossbar, bus) Packet Processing Buffer Mgmt & Scheduling Lookup Tables Buffer & State Memory OC192c Linecard: ~10-30M gates ~2Gbits of memory ~2 square feet >$10k cost; price $100K • Backplane

49. WFQ Performance metrics • Capacity • “maximize C, s.t. volume < 2m3 and power < 5kW” • Throughput • Operators like to maximize usage of expensive long-haul links. • Controllable Delay • Some users would like predictable delay. • This is feasible with output-queueing plus weighted fair queueing (WFQ).

50. Why do we Need Faster Routers? • To prevent routers from becoming the bottleneck in the Internet. • To increase POP capacity, and to reduce cost, size and power.