Building Gigabit-rate Routers with the NetFPGA: EuroSys Tutorial at University of Glasgow

1. Building Gigabit-rate Routerswith the NetFPGA: EuroSys Tutorial at University of Glasgow Presented by: John W. Lockwood & G. Adam Covington (Stanford University) Andrew Moore (University of Cambridge) Monday March 31, 2008 9am-5pm Glasgow, Scotland http://NetFPGA.org

2. What is the NetFPGA? PC with NetFPGA 1GE FPGA 1GE 1GE Memory 1GE NetFPGA Board NetworkingSoftware running on a standard PC CPU Memory PCI A hardware accelerator built with Field Programmable Gate Arraydriving Gigabit network links

3. Introduction Who uses the NetFPGA • Teachers • Students • Researchers How they use the NetFPGA • To run the Router Kit • To build modular reference designs • IPv4 router • 4-port NIC • Ethernet switch, … • To create new networking systems

4. Running the Router KitUser-space development, 4x1GE line-rate forwarding OSPF BGP My Protocol user kernel Routing Table “Mirror” 1GE FPGA Fwding Table Packet Buffer 1GE 1GE 1GE 1GE IPv4 Router 1GE Memory 1GE 1GE Usage #1 CPU Memory PCI

5. Building Modular Router Modules Verilog EDA Tools (Xilinx, Mentor, etc.) NetFPGA Driver • Design • Simulate • Synthesize • Download 1GE L3 Parse L2 Parse In Q Mgmt 1GE 1GE IP Lookup Out Q Mgmt 1GE Verilog modules interconnected by FIFO interfaces Usage #2 PW-OSPF CPU Memory Java GUI Front Panel (Extensible) PCI 1GE FPGA 1GE 1GE My Block Memory 1GE

6. Creating new systems Verilog EDA Tools (Xilinx, Mentor, etc.) • Design • Simulate • Synthesize • Download NetFPGA Driver 1GE My Design (1GE MAC is soft/replaceable) 1GE 1GE 1GE Usage #3 CPU Memory PCI 1GE FPGA 1GE 1GE Memory 1GE

7. Tutorial Outline Background Basics of an IP Router (Andrew) The NetFPGA Platform (John) The Stanford Base Reference Router Demo1 : Reference Router running on the NetFPGA (Adam) Inside the NetFPGA hardware (John) Breakneck introduction to Verilog (John) Exercise 1: Build your own Reference Router (Adam) The Enhanced Reference Router Motivation: Understanding buffer size requirements in a router (Andrew) Demo 2: Observing and controlling the queue size (Adam) Exercise 2: Enhancing the Reference Router (Adam) The Life of a Packet Through the NetFPGA Hardware Datapath (Adam) Interface to software: Exceptions and Host I/O (John) Exercise 3: Drop 1 in N Packets (Adam) Concluding Remarks Using NetFPGA for research and teaching (John) Group Discussion (All) Survey (You)

8. Basic Operation of an IP Router D R3 R1 R4 D A B E R2 C R5 Destination Next Hop F D R3 E R3 F R5

9. What does a router do? R3 R1 R4 D A 1 4 16 32 D Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset B E TTL Protocol Header Checksum 20 bytes Source Address R2 C R5 Destination Address Destination Next Hop F D R3 Options (if any) E R3 Data F R5

10. What does a router do? R3 R1 R4 D A B E R2 C R5 F

11. Basic Components of an IP Router Software Hardware Management & CLI Routing Protocols Control Plane Routing Table Datapath per-packet processing Forwarding Table Switching

12. Per-packet processing in an IP Router 1. Accept packet arriving on an incoming link. 2. Lookup packet destination address in the forwarding table, to identify outgoing port(s). 3. Manipulate IP header: e.g., decrement TTL, update header checksum. 5. Buffer packet in the output queue. 6. Transmit packet onto outgoing link.

13. Generic Datapath Architecture Data Hdr Data Hdr IP Address Next Hop Forwarding Table Buffer Memory Header Processing Lookup IP Address Update Header Queue Packet

14. CIDR and Longest Prefix Matches 142.12/19 • The IP address space is broken into line segments. • Each line segment is described by a prefix. • A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment. • e.g. The prefix 128.9/16 represents the line segment containing addresses in the range: 128.9.0.0 … 128.9.255.255. 128.9.0.0 65/8 128.9/16 0 232-1 216 128.9.16.14

15. Classless Interdomain Routing (CIDR) 128.9.19/24 128.9.25/24 128.9.16/20 128.9.176/20 Most specific route = “longest matching prefix” 128.9/16 0 232-1 128.9.16.14

16. Techniques for LPM in hardware • Linear search • Direct lookup • Currently requires too much memory • Updating a prefix leads to many changes • Tries • Deterministic lookup time • Easily pipelined • But requires multiple memories/references • TCAM (Ternary CAM) • Simple and widely used • But low-density, high-power • Gradually being replaced by new algorithms

17. An IP Router on NetFPGA Software Hardware Management & CLI Linux user-level processes Routing Protocols Exception Processing Routing Table Verilog on NetFPGA PCI board Forwarding Table Switching

18. NetFPGA Router Function 4 Gigabit Ethernet ports Fully programmable FPGA hardware Low cost Open-source FPGA hardware Verilog base design Open-souce Software Drivers in C and C++

19. NetFPGA Platform Major Components • Interfaces • 4 Gigabit Ethernet Ports • PCI Host Interface • Memories • 36Mbits Static RAM • 512Mbits DDR2 Dynamic RAM • FPGA Resources • Block RAMs • Configurable Logic Block (CLBs) • Memory Mapped Registers

20. NetFPGA System User Space Linux Kernel CAD Tools Monitor Software Web & VideoServer Browser & Video Client Packet Forwarding Table PCI-e PCI VI VI VI VI NIC NetFPGA RouterHardware GE GE GE GE GE GE (nf2c0 .. 3) (eth1 .. 2)

21. NetFPGA Hardware

22. NetFPGA System Implementation • NetFPGA Blocks • Virtex-2 Pro FPGA • 4.5MB ZBT SRAM • 64MB DDR2 DRAM • PCI Host Interface • 4 Gigabit Ethernet ports • Intranet Test Ports • Dual Gigabit Ethernets on PCI-e • Internet • Gigabit Ethernet on Motherboard • Processor • Quad-Core CPU • Operating System • Linux CentOS 4.4

23. Dell 2950: Another support NetFPGA host NetFPGA properly Inserts in PCI / PCI-X slot Dell 2950 with PCI-X and PCI-Express Slots Thanks: Brian Cashman for providing machine

24. NetFPGA Lab Setup CPU x2 Dual NIC Client Eth2 : Server PCI-e GE (eth1 .. 2) Eth1 : Local host GE Server Net-FPGA Nf2c3 : Adj. Server GE PCI NetFPGA Control SW Nf2c2 : Local Host Internet Router Hardware GE Nf2c1 : Adjacent GE Nf2c0 : Adjacent GE CAD Tools

25. NetFPGA Hardware Set for Demo #1 CPU x2 NIC Video Server PCI-e PCI-e GE GE Net-FPGA GE PCI Internet Router Hardware GE GE Server deliversstreaming HD video through a chain of NetFPGA Routers GE Net-FPGA GE Internet Router Hardware GE GE GE … CPU x2 NIC PCI-e GE GE Net-FPGA GE Video Display PCI Internet Router Hardware GE GE CAD Tools GE

26. Tutorial Outline Background Basics of an IP Router (Andrew) The NetFPGA Platform (John) The Stanford Base Reference Router Demo1 : Reference Router running on the NetFPGA (Adam) Inside the NetFPGA hardware (John) Breakneck introduction to Verilog (John) Exercise 1: Build your own Reference Router (Adam) The Enhanced Reference Router Motivation: Understanding buffer size requirements in a router (Andrew) Demo 2: Observing and controlling the queue size (Adam) Exercise 2: Enhancing the Reference Router (Adam) The Life of a Packet Through the NetFPGA Hardware Datapath Interface to software Exercise 3: Drop 1 in N Packets Concluding Remarks Using NetFPGA for research and teaching (John) Group Discussion Survey

27. Topology of NetFPGA Routers Demo 1 Video Server HD Display

28. Setup for the Reference Router Each NetFPGA card has four ports Port 2 connected to Client / Server Ports 0 and 3 connected to adjacent NetFPGA cards NetFPGA NetFPGA NetFPGA Demo 1 Video Server Video Client

29. Subnet Configuration Demo 1 .1.1 .4.1 .7.1 .10.1 .13.1 .16.1 .1.2 .4.2 .7.2 .10.2 .13.2 .16.2 .3.1 .6.2 .9.2 .12.2 .15.2 .17.1 .3.2 .2.1 .6.1 .9.1 .12.1 .15.1 .30.2 .5.1 .8.1 .11.1 .14.1 .18.1 .30.1 .26.1 .23.1 .18.2 .27.2 .24.2 .21.2 .20.1 .29.1 .24.1 .21.1 .27.1 .28.2 .25.2 .22.2 .19.2 .28.1 .25.1 .22.1 .19.1 Video Server Video Client Shortest Path

30. Physical Configuration • Qnty = N * • NetFPGA • Mrg Part#: NETFPGA from DigilentInc.com • Ethernet Adapter • Intel Pro/1000 Dual-port Gigabit PCI-Express x4 NIC • Mfg Part#: EXPI9402PT from Buy.com • Category 5E Gigabit Ethernet Cables • Loopback (Red, 1’ = 30cm) • Network (Green, 3’ = 1m) • Adjacent Host (Blue, 3’ = 1m) • Configuration (Yellow, 9’ = 3m) • Additional Parts • RJ45 Cable connectors, Qnty=2

31. Cable Configuration • NetFPGA Gigabit Ethernet Interfaces • nf2c3 : Left neighbor in network • nf2c2 : Local host interface • nf2c1 : Direct connection to adjacent server • nf2c0 : Right neighbor in network • Host Ethernet Interfaces • eth1 : Local host interface • eth2 : Direct connection for adjacent client 11 12 2 3 4 9 6 7 8 5 nf2c nf2c nf2c nf2c nf2c nf2c nf2c nf2c nf2c nf2c 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 eth eth eth eth eth eth eth eth eth eth

32. Photos of Machines During Setup

33. Working IP Router Objectives Become familiar with Stanford Reference Router Observe PW-OSPF re-routing traffic around a failure Demo 1

34. Streaming Video through the NetFPGA Demo 1 • Video server • Source files /var/www/html/video • Network URL : http://192.168.Net.Host/video • Video client • Windows Media Player • Linux mplayer • Video traffic • MPEG2 HDTV (35 Mbps) • MPEG2 TV (9 Mbps) • DVI (3 Mbps) • WMF (1.7 Mbps)

35. Step 1 – Observe the Routing Tables The router is already configured and running on your machines The routing table has converged to the routing decisions with minimum number of hops Next, break a link … Demo 1

36. Step 2 - Dynamic Re-routing Break the link between video server and video client Routers re-route traffic around the broken link and video continues playing Demo 1 .1.1 .4.1 .7.1 .10.1 .13.1 .16.1 .1.2 .4.2 .7.2 .10.2 .13.2 .16.2 .3.1 .6.2 .9.2 .12.2 .15.2 .17.1 .2.1 .3.2 .15.1 .6.1 .9.1 .12.1 .30.2 .5.1 .8.1 .11.1 .14.1 .18.1 .30.1 .26.1 .23.1 .18.2 .21.2 .27.2 .24.2 .20.1 .29.1 .24.1 .21.1 .27.1 .28.2 .25.2 .22.2 .19.2 .28.1 .25.1 .22.1 .19.1

37. Tutorial Outline Background Basics of an IP Router (Andrew) The NetFPGA Platform (John) The Stanford Base Reference Router Demo1 : Reference Router running on the NetFPGA (Adam) Inside the NetFPGA hardware (John) Breakneck introduction to Verilog (John) Exercise 1: Build your own Reference Router (Adam) The Enhanced Reference Router Motivation: Understanding buffer size requirements in a router (Andrew) Demo 2: Observing and controlling the queue size (Adam) Exercise 2: Enhancing the Reference Router (Adam) The Life of a Packet Through the NetFPGA Hardware Datapath (Adam) Interface to software: Exceptions and Host I/O (John) Exercise 3: Drop 1 in N Packets (Adam) Concluding Remarks Using NetFPGA for research and teaching (John) Group Discussion (All) Survey (You)

38. Integrated Circuit Technology Full-custom Design Complementary Metal Oxide Semiconductor (CMOS) Semi-custom ASIC Design Gate array Standard cell Programmable Logic Device Programmable Array Logic Field Programmable Gate Arrays Processors

39. Look-Up Tables Combinatorial logic is stored in Look-Up Tables (LUTs) Also called Function Generators (FGs) Capacity is limited only bynumber of inputs, not complexity Delay through the LUT is constant Combinatorial Logic A B Z C D Diagram From: Xilinx, Inc

40. Each slice has four outputs Two registered outputs, two non-registered outputs Two BUFTs associated with each CLB, accessible by all 16 CLB outputs Carry logic run vertically Signals run upwards Two independent carry chains per CLB Xilinx CLB Structure LUT LUT PRE D Q CE CLR PRE D Carry Carry Q CE CLR Slice 0 Diagram From: Xilinx, Inc (Courtesy Jeff Weintraub)

41. Field Programmable Gate Arrays CLB Primitive element of FPGA Routing Module Global routing Local interconnect Macro Blocks Block Memories Microprocessor I/O Block

42. NetFPGA Block Diagram

43. Details of NetFPGA Fits into Standard PCI slot Standard Bus : 32 bits, 33 MHz Provides Interfaces for processing network packets 4 Gigabit Ethernet Ports Allows hardware-accelerated processing Implemented with Field Programmable Gate Array (FPGA) Logic

44. Tutorial Outline Background Basics of an IP Router (Andrew) The NetFPGA Platform (John) The Stanford Base Reference Router Demo1 : Reference Router running on the NetFPGA (Adam) Inside the NetFPGA hardware (John) Breakneck introduction to Verilog (John) Exercise 1: Build your own Reference Router (Adam) The Enhanced Reference Router Motivation: Understanding buffer size requirements in a router (Andrew) Demo 2: Observing and controlling the queue size (Adam) Exercise 2: Enhancing the Reference Router (Adam) The Life of a Packet Through the NetFPGA Hardware Datapath (Adam) Interface to software: Exceptions and Host I/O (John) Exercise 3: Drop 1 in N Packets (Adam) Concluding Remarks Using NetFPGA for research and teaching (John) Group Discussion (All) Survey (You)

45. Hardware Description Languages • Concurrent • By Default, Verilog statements evaluated concurrently • Express fine grain parallelism • Allows gate-level parallelism • Provides Precise Description • Eliminates ambiguity about operation • Synthesizable • Generates hardware from description

46. Verilog Data Types reg [7:0] A; // 8-bit register, MSB to LSB // (Preferred bit order for NetFPGA) reg [0:15] B; // 16-bit register, LSB to MSB B = {A[7:0],A[0:7]}; // Assignment of bits reg [31:0] Mem [0:1023]; // 1K Word Memory integer Count; // simple signed 32-bit integer integer K[1:64]; // an array of 64 integers time Start, Stop; // Two 64-bit time variables From: CSCI 320 Computer Architecture Handbook on Verilog HDL, by Dr. Daniel C. Hyde : http://eesun.free.fr/DOC/VERILOG/verilog-manual.html

47. Signal Multiplexers From:http://eesun.free.fr/DOC/VERILOG/synvlg.html • Two input multiplexer (using if / else) • reg y; • always @*   if (select)      y = a;   else      y = b; Two input multiplexer (using ternary operator ?:) wire t = (select ? a : b);

48. Larger Multiplexers Three input multiplexer reg s; always @*   begin   case (select2)       2'b00: s = a;      2'b01: s = b;      default: s = c;    endcase   end

49. Synchronous Storage Elements Din D Q Dout Clock 1 Clock Transition Clock 0 time t=0 t=1 t=2 A B C • Values change at times governed by clock • Clock • Input to circuit • Clock Event • Example: Rising edge Din t=0 • Flip/Flop • Transfers Value From Din to Dout on Clock event Clock Transition Dout A B S0 t=0

50. Finite State Machines