EtherPIPE : an Ethernet character device for network scripting

1. EtherPIPE: an Ethernet character device for network scripting Yohei Kuga (Keio University) Takeshi Matsuya (Keio University) Hiroaki Hazeyama(NAIST) Kenjiro Cho (IIJ) Osamu Nakamura (Keio University) HotSDN’13, August 16, 2013, Hong Kong, China

2. Motivation • Shell scripting is powerful to process files • We want to use it for network processing Our Goal $ cat /dev/port0 > ./capture Use Basic UNIX command Receive packets Capture packets

3. Motivation • Shell scripting is powerful to process files • We want to use it for network processing Our Goal $ cat /dev/port0 >/dev/port1 Use Basic UNIX command Receive packets Send packets

4. Motivation • Shell scripting is powerful to process files • We want to use it for network processing • It’s useful for SDN prototyping and debugging Our Goal $ ./FW.sh </dev/port0 >/dev/port0 Build a network application Receive packets Send packets

5. EtherPIPE • A character device for packet processing • It accesses the device using open(2) • And, handles packets using read(2) and write(2) Then UNIX commands can read/write packets as a file

6. Network Scripting • Lightweight network programming by shell scripts • Combining UNIX commands, redirections, pipes with EtherPIPE devices

7. EtherPIPE design: Device namespace • ‘0’ and ‘1’: Shell Interface • ASCII format. For shell scripting • ‘r0’ and ‘r1’: Raw Interface • Binary format. For high-bandwidth and complex applications

8. EtherPIPE design: Shell Interface (ASCII) 12 words 12 words 2 words 4 words • one packet per line • Device driver converts raw data to ASCII • Each byte is separated by “space” character • To make string-based UNIX commands to process packets • But the data size is 3 times larger

9. Shell Interface: Applications • Sending a packet $ echo “FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00 22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20“ >/dev/ethpipe/0 • Port mirroring $ cat /dev/ethpipe/0 | tee /dev/ethpipe/0 > /dev/ethpipe/1 • MAC address filtering $ grep ‘^FFFFFFFFFFFF’ /dev/ethpipe/0 > /dev/ethpipe/1 • VLAN untagging $ sed -e 's/8100 00 01 //' < /dev/ethpipe/0 > /dev/ethpipe/1

10. EtherPIPE design: Raw interface (Binary) • For simple capturing and replaying, and for optimized applications • Raw format includes fields for offloading • Hardware timestamp • nanosecond RX timestamp for precise capturing • Five-tuple hash for flow processing • Hash # <src and dst IP addr, proto num, src and dst port num>

11. Prototype implementations • Device driver (Shell IF) for generic NICs • For proof-of-concept • FPGA card + device driver • Dev kit: LatticeECP3 versa kit ($99, 1GE x2 and PCIe) • Supports 1000BASE-T full duplex • Shell IF device driver (working) • Raw IF device driver (under development) • FPGA logic and drivers source code are available • https://github.com/sora/ethpipe

12. Prototype implementations • Device driver (Shell IF) for generic NICs • For proof-of-concept • FPGA card + device driver • Dev kit: LatticeECP3 versa kit ($99, 1GE x2 and PCIe) • Supports 1000BASE-T full duplex • Shell IF device driver (working) • Raw IF device driver (under development) • FPGA logic and drivers source code are available • https://github.com/sora/ethpipe RX: 1GE line-rate TX: near 1GE line-rate

13. How to measure the transmit PPS $ head ./4m.pkt FFFFFFFFFFFF 0022CF63967B 8899 23 .. FFFFFFFFFFFF 0022CF63967B 8899 23 .. FFFFFFFFFFFF 0022CF63967B 8899 23 .. $ wc ./4m.pkt 4000000 196000000 676000000 4m.pkt $ time cat ./4m.pkt >/dev/ethpipe/0 real0m2.700s User0m0.004s sys 0m2.661s $ bc scale=3 4000000/2.700 1481481.481 • Measured the TX PPS of Shell IF by `wc`, `cat` and `time` • Performs the near 1GE line-rate PPS 1,481,481ppsvs. 1,488,095 pps for 64B pkts Enough for prototyping

14. Summary • We proposed “Network scripting”, new design of network IO for packet processing, and showed a prototype implementation • Allows shell scripting to handle network devices w/ ‘<, >, |’ • Simple data format for UNIX commands and scripting languages • Future work • Fix Shell and Raw IF formats • E.g,: More offloading fields (currently: RX timestamp and 5-tuple hash) • Finish Raw IF driver and its userland libraries

16. FPGA adapter design Receiving Sending

17. Performance transmit PPS of FPGA prototype: data $ head 4m.pkt FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00 22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00 22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 …

18. Host operations (tappipe.c) • To bind EtherPIPE device with OS network stack by TAP device • To use existing network commands (arp, dhclient, tcpdump, etc) • Build custom layer 2~3 functions by shell script (ping, traceroute, nmap, etc) $ ./tappipe pipe0 </dev/ethpipe/0 >/dev/ethpipe/0 & $ipaddr pipe0 $ tcpdump -ipipe0 $ dhclient –r pipe0 && dhclient pipe0 $ echo ${PING_REQUEST} > /dev/ethpipe/0

19. Performance result ofcommand-line based packet processing • Measuring the potential of Shell IF throughput by dummy driver • Read(): Dummy driver prepare the 64 Byte packet in kernel • Write(): the driver simply copies the data into a buffer in the kernel • Result • ‘Capture’ (memory copy from kernel to userland) performs over 10GE • Some command is very slow such as `cut`

20. Our focus • A typical network device file is designed for communications between hosts • A network device file has to provide all functions / APIs from Layer 2 to Layer 4 • An implementation would be as a special device • Device specific settings and methods are required to send/receive packets • We focus on a network device file for packet processing • Only provide IO of physical ethernet ports and doesn't care other functions (MTU, reassemble ,FIB, etc) • An implementation is developed as a common device file • Basic system callsare enough to send/receive packets