Packet Mangling for Fun & Profit

1. Packet Mangling for Fun & Profit A Brief Intro to Netfilter, User Mode Linux, and the Linux TCP/IP Stack

2. Topics • SplitTCP Summary (Context) • Netfilter Introduction • One buffer to rule them all: sk_buffs • User Mode Linux • Current Status

3. SplitTCP In a Nutshell • Work started by Michalis, Srikanth, and Swastik. • The idea is to add transport layer proxies to long (in terms of hop count) TCP connections. • If a link fails (due to mobility, etc.) the packet can be re-transmitted by the proxy closest to the failure. • A “demo-quality” implementation was done which made many simplifying assumptions.

4. So What? • My current task is to take that demo implementation and turn it into a general implementation. • This presentation discusses the lessons learned up to this point in that process.

5. Design Goals • Any and all modifications made to the behavior of TCP should be backwards compatible. • I.e. nodes should interoperate regardless of weather or not they’re “split tcp enabled”.) • If possible, no changes should be made to the kernel proper. It’s just better for everyone that way…

6. Enter Netfilter • Fortunately, Linux has a plugin API, called Netfilter, which allows kernel modules to hook into strategic spots in the networking stack. • Unfortunately, in the grand tradition of open source, documentation takes a back seat to implementation – so there’s a non-trivial learning curve.

7. Sideline: Plugin APIs • An example of the Delegation design pattern (GoF) – a work unit is passed through a series of cooperating steps to produce the final result. • A well-designed plugin architecture can facilitate otherwise impossible tasks and simply possible ones.

8. Sideline: Kernel Modules • Most of the linux kernel can be built as a module – a bit of code that’s loaded into (and unloaded from) the kernel dynamically. • Device drivers, file systems, even networking protocols (IPX, Appletalk) are candidates for modularization.

9. Writing a Module • Just define a few preprocessor macros (__KERNEL__, MODULE) • Include a few header files (linux/module.h, linux/version.h, linux/config.h) • And honor a small interface (module_init(), module_exit()) • You’ll have a no-op module.

10. Netfilter Overview • Initial design by Paul “Rusty” Russell • Netfilter is a series of callback functions within the network stack. The API is non-portable and appeared in linux 2.3.x • Each protocol has it’s own set of callback points. We care about IPv4.

11. Netfilter Concepts • A module expresses interest in being invoked at an arbitrary subset of the available callback points – specifying the function and the (global) priority in which it should be called. • That function is passed (among other things) a pointer to a pointer to a packet buffer ( sk_buff ** ).

12. Return Values • The netfilter function has five possible return values: • NF_ACCEPT: continue callback chain • NF_DROP: drop the packet and stop the chain • NF_STOLEN: stop the chain • NF_QUEUE: send the packet to userspace • NF_REPEAT: call the hook again

13. Netfilter Hooks in IPv 3 1 Routing Engine 4 Out In 2 5 Local Sockets

14. 1 In 4 Routing Engine Out 2 5 Local Sockets Say that Again? • 1: NF_IP_PREROUTING • any received packet which checksums OK. • 2: NF_IP_LOCAL_IN • packets destined for local sockets • 3: NF_IP_FORWARD • foreign packets being forwarded • 4: NF_IP_POST_ROUTING • any outbound packet • 5: NF_IP_LOCAL_OUT • packets originating from local sockets

15. An sk_what?? • Linux uses a structure called an sk_buff to store packet data internally. • It contains a handful of pointers to other structures as well as a packet data region. Data Tailroom Headroom Pointers

16. Sk_buff’s and you • The data area is like a stack, only you can insert at the head and the tail (deque?). • The kernel provides a handful of helper functions to manage sk_buff’s and their data areas. • The various header pointers point into the data area – which can be thought of as a serialized packet.

17. Why do We Care? • An sk_buff is built as a packet travels down the stack – each layer (TCP, IP, Ethernet) adds their own special sauce. • This means that each header is “squashed” in against the next – so while modifying existing data is relatively easy, adding new header data is a bit trickier.

18. Don’t leave me in suspense… • Basically, you make a copy of the sk_buff, and ask it to “grow” a bit during the copy. • Once you have the copy – you “slide” the IP and TCP headers backwards a bit, insert the new option bytes, and re-checksum the packet.

19. A tale of “n” checksum’s • Sounds easy, right? Remember that these sk_buffs are built one layer at a time? • There is no nice friendly function which will take a TCP sk_buff and compute all the needed checksums. • Funny thing about checksums – almost isn’t good enough.

20. Ok, insmod and <BOOM> • Remember developing on a system without memory protection and having to reboot ? • Kernel modules execute in kernel space – so no one’s watching your back. If you goof, it’s time to reboot. • User Mode Linux to the rescue

21. User Mode Linux (UML) • A kernel patch that allows running the linux kernel as a user-mode process on a linux machine. • If you crash the user-mode kernel, you just restart the process, no reboot required.

22. Where do I sign up? • Setup is (in principal) fairly easy – only it turns out that the standard distribution doesn’t have netfilter enabled. • So I re-built with the appropriate options and placed the binaries in ~swift/user-mode-linux/bin • There’s a README there – some support executables must be installed as root on your workstation.

23. That’s It? • Not quite – you also need a file system to boot this kernel off. • Good News: you can download a file system image – you don’t have to make one. • Bad News: it’s really big. (up to 700MB) • Good News: you can share one among many people • Bad News: you have to read the HOWTO.

24. Building Modules Under UML • Building a kernel module is the same under UML as under “KML” – it is important that you build it against the source used to build the target kernel. • For “our” UML build – that source is in ~swift/user-mode-linux/src

25. Current Status • A skeleton of a splittcp module (tcpproxy.c) exists. • It can inspect locally generated packets and add our newly defined PROXY option, rechecksum the packet, and send it on it’s way. • It can inspect arriving packets, check for the option, and decide if that packet should be proxied.

26. So What’s Left? • It doesn’t (yet) actually proxy the packet. • Nor does it send an acknowledgement of receipt to the upstream proxy. • There are also issues around ICMP error messages, and what should be done about them.

27. Conclusion • Once the code is “alpha” quality, I’ll commit it to the swift CVS repository for your collective viewing pleasure. • Until then, if you have questions or suggestions, see me.