IP Router Architecture: Evolution and Challenges

1. INF5050 – Protocols and Routing in Internet (Friday 5.2.2016) Subject: IP-router architecture Presented by Tor Skeie

2. This presentation is based on slides from Nick McKeown, with updates Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford.edu www.stanford.edu/~nickm Stanford High Performance Networking group: http://klamath.stanford.edu 2

3. Outline Background • What is a router? • Why do we need faster routers? • Why are they hard to build? Architectures and techniques • The evolution of router architecture. • IP address lookup. • Packet buffering. • Switching. The Future 3

4. D D R3 R1 R4 D A B E R2 C R5 Destination Next Hop F D R3 E R3 F R5 What is Routing? 4

5. R3 R1 R4 D D A D D 1 4 16 32 D Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset B E TTL Protocol Header Checksum 20 bytes Source Address R2 C R5 Destination Address Destination Next Hop F D R3 Options (if any) E R3 Data F R5 What is Routing? 5

6. POP3 POP2 POP1 D POP4 A B E POP5 POP6 C POP7 POP8 F Points of Presence (POPs) 6

7. Where High Performance Routers are Used (400 Gb/s) R2 (400 Gb/s) (2.5 Gb/s) R1 R6 R5 R4 R7 R3 R9 R10 R8 R11 R12 R14 R13 R16 R15 (400 Gb/s) (400 Gb/s) 7

8. Cisco CRS-X (CRS-X 16 slot single-shelf on picture) Juniper M320 (M160 on picture) 0.60m 0.44m Capacity: 12.8Tb/sPower: 11.2kWWeight: 723kg Capacity: 160Gb/sPower: 3.5kW 2.14m 0.88m 0.91m 0.65m What a Router Looks Like Capacity is sum of rates of linecards 8

9. Some Multi-rack Routers Juniper TX8/T640 Alcatel 7670 RSP TX8 Avici TSR Chiaro

10. Data Hdr Data Hdr IP Address Next Hop Address Table Buffer Memory Generic Router Architecture Header Processing Lookup IP Address Update Header Queue Packet 1M prefixes Off-chip DRAM 1M packets Off-chip DRAM 11

11. Data Data Data Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Address Table Address Table Address Table Data Data Hdr Hdr Data Hdr Generic Router Architecture Buffer Manager Buffer Memory Buffer Manager Buffer Memory Buffer Manager Buffer Memory 12

12. Why do we Need Faster Routers? • To prevent routers becoming the bottleneck in the Internet. • To increase POP capacity, and to reduce cost, • size and • power. 13

13. Why we Need Faster Routers 1: To prevent routers from being the bottleneck Packet processing Power Link Speed 10000 More recently transmission speed of Petabit/s has been demonstrated by Labs (multicore fiber) 1000 2x / 18 months 2x / 7 months 100 Fiber Capacity (Gbit/s) 10 1 1985 1990 1995 2000 0,1 TDM DWDM Source: SPEC95Int & David Miller, Stanford. 14

14. POP with smaller routers POP with large routers • Ports: Price >$100k, Power > 400W. • It is common for 50-60% of ports to be for interconnection. Why we Need Faster Routers 2: To reduce cost, power & complexity of POPs 15

15. Why are Fast Routers Difficult to Make? • It’s hard to keep up with Moore’s Law: • The bottleneck is memory speed. • Memory speed is not keeping up with Moore’s Law. 16

16. 1.1x / 18 months Moore’s Law 2x / 18 months Why are Fast Routers Difficult to Make?Speed of Commercial DRAM • It’s hard to keep up with Moore’s Law: • The bottleneck is memory speed. • Memory speed is not keeping up with Moore’s Law. 1.1x / 18 months • DDR4 (2015): • Speed: 3200 Mb/s • Latency: ~13 ns • Capacity: 16GB Moore’s Law 2x / 18 months 17

17. Why are Fast Routers Difficult to Make? • It’s hard to keep up with Moore’s Law: • The bottleneck is memory speed. • Memory speed is not keeping up with Moore’s Law. • Moore’s Law is too slow: • Routers need to improve faster than Moore’s Law. 18

18. Router Performance Exceeds Moore’s Law Growth in capacity of commercial routers: • Capacity 1992 ~ 2Gb/s • Capacity 1995 ~ 10Gb/s • Capacity 1998 ~ 40Gb/s • Capacity 2001 ~ 160Gb/s • Capacity 2003 ~ 640Gb/s • Capacity 2008 ~ 100Tb/s • Capacity 2013 ~ 920Tb/s Average growth rate: 2.2x / 18 months, but the last 5 years: 2.8x / 18 months. 2013:The Cisco CRS-X multishelf router has a capacity of 921.6Tb/s (1152 ports) 19

19. Outline Background • What is a router? • Why do we need faster routers? • Why are they hard to build? Architectures and techniques • The evolution of router architecture. • IP address lookup. • Packet buffering. • Switching. The Future 20

20. CPU Buffer Memory Route Table CPU Line Interface Line Interface Line Interface Memory MAC MAC MAC Typically <0.5Gb/s aggregate capacity First Generation Routers Shared Backplane Line Interface 21

21. Fwding Cache Second Generation Routers CPU Buffer Memory Route Table Line Card Line Card Line Card Buffer Memory Buffer Memory Buffer Memory Fwding Cache Fwding Cache MAC MAC MAC Typically <5Gb/s aggregate capacity 22

22. Fwding Table Third Generation Routers Switched Backplane Line Card CPU Card Line Card Local Buffer Memory Local Buffer Memory Line Interface CPU Routing Table Memory Fwding Table MAC MAC Typically <50Gb/s aggregate capacity 23

23. Fourth Generation Routers/SwitchesOptics inside a router for the first time Optical links 100s of metres Switch Core Linecards 100s Tb/s routers available/in development 24

24. Outline Background • What is a router? • Why do we need faster routers? • Why are they hard to build? Architectures and techniques • The evolution of router architecture. • IP address lookup. • Packet buffering. • Switching. The Future 25

25. Header Processing Header Processing Lookup IP Address Lookup IP Address Update Header Update Header Address Table Address Table Lookup IP Address Lookup IP Address Lookup IP Address Address Table Address Table Address Table Generic Router Architecture Buffer Manager Buffer Memory Header Processing Buffer Manager Lookup IP Address Update Header Buffer Memory Address Table Buffer Manager Buffer Memory 26

26. IP Address Lookup Why it’s thought to be hard: • It’s not an exact match: it’s a longest prefix match. • The table is large: about 590,000 entries today, and growing. • The lookup must be fast: about 2ns for a 140Gb/s line. 27

27. Longest Prefix Match is Harder than Exact Match • The destination address of an arriving packet does not carry with it the information to determine the length of the longest matching prefix • Hence, one needs to search among the space of all prefix lengths; as well as the space of all prefixes of a given length 28

28. 128.9.16.14 IP Lookups find Longest Prefixes 128.9.176.0/24 128.9.16.0/21 128.9.172.0/21 142.12.0.0/19 65.0.0.0/8 128.9.0.0/16 0 232-1 Routing lookup:Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address. 29

29. Address Tables are Large 30

30. Lookup mechanism must be simple and easy to implement • Memory access time is the bottleneck • 250Mpps × 2 lookups/pkt = 500 Mlookups/sec → 2ns per lookup Lookups Must be Fast 31

31. IP Address LookupBinary tries Example Prefixes: 0 1 a) 00001 b) 00010 0 c) 00011 d) 001 0 e) 0101 g f d f) 011 1 g) 100 h i h) 1010 e 1 i) 1100 j) 11110000 a b c j 32

32. Multi-ary trie W/k Depth = W/k Degree = 2k Stride = k bits Multi-bit Tries Binary trie W Depth = W Degree = 2 Stride = 1 bit Time ~ W/k Storage ~ NW/k * 2k-1 W = longest prefix N = #prefixes 33

33. Prefix Length Distribution Source: Geoff Huston, Oct 2001 99.5% prefixes are 24-bits or shorter 34

34. 24-8 Direct Lookup Trie 0000……0000 1111……1111 24 bits 0 224-1 8 bits 0 28-1 • When pipelined, allows one lookup per memory access. • Inefficient use of memory, though. 35

35. Outline Background • What is a router? • Why do we need faster routers? • Why are they hard to build? Architectures and techniques • The evolution of router architecture. • IP address lookup. • Packet buffering. • Switching. The Future 38

36. Line cardshostingone or moreports Bi-directional ports Bi-directional ports Arbitration/Control Conceptual architecture Non-blockingswitchingcore(s) 40

37. Input buffering Bi-directional ports Bi-directional ports Arbitration/Control Conceptual Packet Buffering Non-blockingswitchingcore(s) 41

38. Data Data Data Data Data Data Hdr Hdr Hdr Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header 1 1 Address Table Address Table Address Table 2 2 N N Arbitration Queue Packet Buffer Memory Queue Packet Buffer Memory Arbitration Queue Packet Buffer Memory 43

39. Head of Line Blocking 44

40. A Router with Input QueuesHead of Line Blocking The best that any queueing system can achieve. 45

41. The Best Performance The best that any queueing system can achieve. 46

42. Central buffer Bi-directional ports Bi-directional ports Arbitration/Control Conceptual Packet Buffering Non-blockingswitchingcore(s) 47

43. Fast Packet Buffers(http://yuba.stanford.edu/fastbuffers/) Example: 40Gb/s packet buffer Size = RTT*BW = 10Gb; 40 byte packets Write Rate, R Read Rate, R Buffer Manager 1 packet every 8 ns 1 packet every 8 ns Buffer Memory Use SRAM? + fast enough random access time, but - too low density to store 10Gb of data. Use DRAM? + high density means we can store data, but - too slow (~15ns random access time). 48

44. 54 53 52 51 50 10 9 8 7 6 5 95 94 93 92 91 90 89 88 87 86 15 14 13 12 11 10 9 8 7 6 Small ingress SRAM Small ingress SRAM cache of FIFO tails cache of FIFO heads 86 85 84 83 82 11 10 9 8 7 1 1 1 55 60 59 58 57 56 2 97 96 2 Q Q 87 88 91 90 89 DRAM Buffer Memory Packet Caches Buffer Manager SRAM 1 4 3 2 Arriving Departing 2 Packets Packets 2 1 4 3 5 Q 6 5 4 3 2 1 b>>1 packets at a time DRAM Buffer Memory 49

45. Output buffering Bi-directional ports Bi-directional ports Arbitration/Control Conceptual Packet Buffering Non-blockingswitchingcore(s) 50

46. Data Data Data Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Address Table Address Table Address Table Data Data Hdr Hdr Data Hdr Output buffering Buffer Manager Buffer Memory Buffer Manager Buffer Memory Buffer Manager Buffer Memory 51

47. Data Data Data Hdr Hdr Hdr Header Processing Header Processing Header Processing Lookup IP Address Lookup IP Address Lookup IP Address Update Header Update Header Update Header Address Table Address Table Address Table N times line rate Speed-up 1 1 Queue Packet Buffer Memory 2 2 Queue Packet Buffer Memory N times line rate N N Queue Packet Buffer Memory 54

48. Input buffering with a virtual output queue Bi-directional ports Bi-directional ports Arbitration/Control Conceptual Packet Buffering Non-blockingswitchingcore(s) 55

49. Virtual Output Queues 56

50. Matching vertex edge A matching on a graph is a subset of edges of the graph such that no two of them share a vertex in common. 57