Abstracting the Content of System Call Traces

1. Abstracting the Content of System Call Traces Waseem Fadel Abdelwahab Hamou-Lhadj Department of Electrical and Computer Engineering Concordia University Montréal, QC, Canada {w_fadel, abdelw}@ece.concordia.ca

2. Objective of the Trace Abstraction and Analysis Track (Reminder) Develop trace abstraction techniques and tools to facilitate the understanding and analysis of the content of large event-based system call traces

3. Benefits • Help users understand the behavioural aspects of a system • Allow automated comparison of multiple traces based on the system behaviour and not a mere set of events • Monitor the system performance to detect service degradation • Ensure that subsequent revisions of a software system have not introduced programming errors • Compare traces of redundant servers, performing the same work in order to detect any malfunction possibly caused by a security breach

4. Proposed Approach Trace Generated from Linux Kernel (LTTng events) Trace Abstraction Algorithm Based on Pattern Matching and Filtering of Noise High-Level Trace

5. Pattern Library • We built a pattern library that contains several patterns that represent key Linux kernel operations • File, socket and process management operations • The patterns are modeled as state machines • States represent system modes (user_mode, sys_call mode, etc.) • Events consist of LTTng events

6. Patterns we have so far • We completed the pattern library to include the following operations: • File Management (Open, Read, Write, Seek, Access, File Manipulation with fcntl, Stat, Close) • Socket Management for both TCP and UDP (Create, Connect, Bind, Listen, Accept, Send, Receive, Close) • Process Management (Execution with exec and execve, Exit, Fork, Clone, Get Resource Limit, Get Time of The Day, New UName, Unlink, Read Link)

7. Filtering of Trace Noise • We define noise in an LTTng trace as any event associated with memory management, page faults, and interrupts • Are dependent on a specific kernel version • Can occur anywhere in the trace and in any order • Are treated similarly to the way utilities have been treated in related work • Associated events are treated as a set • i.e. order of occurrence of detailed events is ignored

8. Validation of Patterns • The patterns have been validated by Pierre-Marc Fournier and Mathieu Desnoyers from École Polytechnique de Montréal • Regular meetings with them have also helped in the process of understanding the markers, the system calls, and the main modes of execution • Both users agreed with the way we defined noise found in traces

9. The Linux Kernel Trace Abstraction Tool • The tool takes as input a trace generated from LTTng tracer • It applies our approach to that trace • Outputs the trace in its abstracted format • It has been developed in Java as an Eclipse plug-in

10. The Linux Kernel Trace Abstraction Tool (cont.) • The tool was designed to accept patterns defined as external XML files <?xml version="1.0" encoding="ISO-8859-1"?> <pattern name="Sample Pattern" type="HighLevelSampleConstrcut" noise="false"> <event name="syscall_entry" syscall_name="sys_sample" order="1" prev_state="IGNORE"> <current_state>SYSCALL_SAMPLE</current_state> </event> <event name="sample" order="2" prev_state="SYSCALL_SAMPLE"> <current_state>SYSCALL_SAMPLED</current_state> </event> <event name="syscall_exit" order="LAST" prev_state="SYSCALL_SAMPLED"> <current_state>USER_MODE_SAMPLED</current_state> </event> </pattern>

11. The Linux Kernel Trace Abstraction Tool (cont.)

12. The Linux Kernel Trace Abstraction Tool (cont.) • The tool consists of three parts: • The top part, through which the developer can provide the tool with the required information • The middle part, displays the trace resulting from the abstraction process • The bottom part, displays the information related to the original and abstracted traces • The following slides show the architecture and the class diagram

13. Architecture • Partitioned both horizontally and vertically • Easy to extend (to add new system calls patterns or even patterns for different calls) • Easy to maintain (to modify existing patterns or the design itself)

14. Horizontal Partitioning • Horizontal partitioning is performed by defining the main domains of the system

15. Vertical Partitioning

16. Vertical Partitioning (cont.) • Divide the system into different layers • Define the interfaces between layers • Presentation layer can be developed without affecting lower layers (Multiple GUIs can be provided for the same data) • Components from different layers can be designed, implemented and maintained independently

17. Class Diagram

18. Class Diagram (cont.) • Adding new patterns and high-level constructs can easily be done by sub-classing the appropriate classes and interfaces • Multiple implementations representing different trace formats can be applied using the same interfaces • High level constructs are easy to further abstract

19. Case Studies • We applied our approach to large traces generated while running different processes • One process was the java virtual machine which was running a distributed file server and a client • Another process was the eclipse framework • Third process was gedit • Fourth process was GIMP image editor • The final process was firefox

20. Case Studies (cont.)

21. Case Studies (cont.) SEQ(1) Socket Create: family = 2, type = 2, protocol = 0, sock = 0xd563d340, ret = 8 SEQ(1) Socket Connect: fd = 8, uservaddr = 0x80569c8, addrlen = 28, ret = 0 SEQ(2) File Stat: ip = 0xb7f78430, syscall_id = 221 [sys_fcntl64+0x0/0xb0] SEQ(1) Get Time of Day: ip = 0xb7f78430, syscall_id = 78 [sys_gettimeofday+0x0/0x80] SEQ(1) Unknown Event: Event name: pollfd, Params: fd = 8 SEQ(1) Unknown Event: Syscall name: sys_poll, Params: ip = 0xb7f78430, syscall_id = 168 [sys_poll+0x0/0xc0] SEQ(1) Socket Send: call = 9, a0 = 8 SEQ(4) Unknown Event: Event name: pollfd, Params: fd = 8 SEQ(1) Process Schedule: prev_pid = 0, next_pid = 23566, prev_state = 0 … SEQ(1) Unknown Event: Syscall name: sys_futex, Params: ip = 0xb7f78430, syscall_id = 240 [sys_futex+0x0/0x130] SEQ(1) Unknown Event: Syscall name: sys_clock_gettime, Params: ip = 0xb7f78430, syscall_id = 265 [sys_clock_gettime+0x0/0xa0] SEQ(1) Get Time of Day: ip = 0xb7f78430, syscall_id = 78 [sys_gettimeofday+0x0/0x80] SEQ(1) File Access: ip = 0xb7f78430, syscall_id = 33 [sys_access+0x0/0x30] SEQ(1) File Write: fd = 19 SEQ(1) Socket Receive: call = 10, a0 = 19 SEQ(1) Process Exit: Process Exit: pid = 23580, cpu_id = 1, state = 1, Send Signal: pid = 23610, cpu_id = 1, state = 1, Sched Try Wakeup: pid = 23972, cpu_id = 1, state = 1 A snapshot of the traces resulting from the abstraction of the JVM process

22. Remaining Challenges • Continuous improvement of the pattern library • Defining additional patterns • Dealing with new LTTng events • Using higher level constructs to further abstract the resulting traces • Improving the algorithm in terms of performance • Embedding the tool with the LTTv

23. Conclusion • We introduced techniques to abstract execution traces resulting from the Linux kernel • Our approach is based on building a pattern library that consists of patterns of the most common operations in Linux • We also defined noise patterns that result from memory management operations and page faults • We introduced an algorithm to abstract the system call traces by using the pattern library • We applied our techniques to traces generated from several processes

24. Pattern Library

25. File Management: Open & Close Open File Close File

26. File Management: Read & Write Write to File Read from File

27. Socket Management (1) Create Bind Listen

28. Socket Management (2) Connect Receive Send

29. Socket Management (3) TCP Connection

30. Process Management (1)

31. Process Management (2) Execution with exec with execve Exit Cloning

32. Other Patterns File Control Read Link Unlink Get Resource Limit New UName

33. Thank You! Questions?