GILK: A Dynamic Instrumentation Tool for the Linux Kernel

1. GILK: A Dynamic Instrumentation Tool for the Linux Kernel David J. Pearce, Paul H.J. Kelly, Tony Field and Uli Harder d.pearce@doc.ic.ac.uk Based on paper presented at 12th International Conference on Computer Performance Evaluation (TOOLS’02), London, April 2002 (Springer LNCS 2324) See http://www.doc.ic.ac.uk/~djp1/gilk.html

2. Motivation • Instrumentation: “Do programmers really know what is going on?” • Need to visualise program behaviour • Build, debug/validate performance models • Understand and fix performance problems • Exploration: users need to explore system behaviour interactively • Auditing: “How can we monitor what is going on?” • Relies on understanding of program behaviour • Minimise impact on system • Need flexible programmable tool • Operating at lowest possible level to circumvent countermeasures

3. Introduction: instrumenting the Linux kernel • GILK is a dynamic instrumentation tool for the Linux Kernel • Run-time patching of the OS kernel’s code • On the fly: no need to reboot, no need to restart servers • “Instruments” can be added and removed via a simple GUI • Or programmatically – adding and removing an instrument takes milliseconds • No need to recompile the kernel • No need to interrupt the web server etc

4. Measure, audit, intercept… • Instruments… • We provide a simple set of instruments for performance measurement, selectable from the GUI • User-definable instruments • API allows instruments to be defined by the user in C • Intercept any kernel function • Log parameters, message contents etc • Instruments can also check parameters against security policy, enforce rules, etc

5. GILK – the GUI • GUI starts with Linux kernel’s symbol table • Lists all kernel functions • System calls • Network protocol stack • File system • Device drivers • Inter-process communication • Memory management

6. GILK – An Introduction • Click on a kernel function • Dialogue offers to attach instrument to function • Various instruments • Before, after or both • Or browse control flow graph to instrument internal blocks

7. GILK – An Introduction • An instrumentation “experiment” runs automatically • Each instrument has its own launch and splashdown time • So that performance impact of instrumentation can be staggered • All instruments automatically removed on completion/exit • Instrument data is logged to experiment output file

8. Example: validate TCPdump • Uli Harder is researching statistical “self-similarity” properties of network traffic • Using standard TCPDUMP tool to obtain network timing information • Produced fair amount of data • But occasional zero or negative inter-arrival time was spotted • Are results so far wasted? • Is TCPDUMP useable or not? • Uli used GILK for comparison with TCPDUMP • GILK provides high resolution packet arrival times • TCPDUMP does generate erroneous results • But previous results valid – statistical properties unchanged

9. What We Did • Browse Linux source code and call graph • Identify high-level socket interface • Add “rdtsc” timer instrument • Is socket interface timing what we want? • Find ethernet device driver • Instrument it to time actual packet transfer Linux Network Stack { BSD Socket Layer sock_recvmsg RDTSC { Ethernet Driver RDTSC speedo_rx

10. Where was the Success? • Uli’s not a kernel programmer • Didn’t know beforehand what was needed • Time wasted trying different instrumentation • GILK is fast, and that’s important • Instruments can be moved, adjusted, relaunched interactively while the system is under the test load • Allows us to try many different instrumentations • Fits exploratory nature of debugging and modelling system behaviour

11. How it Works • GILK uses code splicing technology • Allows instrumentation of active kernel • Instrumentation performed at machine code level • Works on unmodified kernel image • No patching, restarting, recompiling etc • Instruments written in ‘C’ • Implemented as Kernel Modules • Access to full language features • Access to kernel functionality • System Requirements • Only Intel x86 Architecture supported • Kernel versions 2.0 – 2.2 supported • Extension to 2.4 is trivial

12. Instruction Boundaries { { { Kernel Function Kernel Function Code Patch Before After splice unused instrumentation Relocated sequence

13. Conclusions • GILK quickly instruments Linux Kernel • Runtime code splicing useful for instrumentation • Some development work needed – ease of use, visualisation of instrument output • Intriguing potential…. • Automatic instrumentation – bottleneck identification strategies • Security applications – • Intercept, monitor for unusual behaviour • Characterise processes which initiate unexpected behaviour • Insert authorisation policies • check client process id, application image id, IP addresses, message contents etc • Future work … • Simplify GUI, develop instrumentation strategies, visualization • Automate extraction of dynamic call graph, search for bottlenecks • Explore security policy ideas… context-dependent quality-of-service? • Later linux kernels (2.4 should be easy), SMPs, non-Intel platforms (for Solaris see Paradyn project at Wisconsin) • User-Space programs • Our Research group is now doing this for Java …