Download
1 / 17
170 likes | 273 Views
Self-Propelled Instrumentation. Alex Mirgorodskiy Barton Miller {mirg,bart}@cs.wisc.edu http://www.paradyn.org/ Computer Sciences Department University of Wisconsin-Madison 1210 West Dayton Street Madison, WI 53706 USA. Motivation. Finding the causes of bugs and performance
E N D
Self-Propelled Instrumentation Alex Mirgorodskiy Barton Miller {mirg,bart}@cs.wisc.edu http://www.paradyn.org/ Computer Sciences Department University of Wisconsin-Madison 1210 West Dayton Street Madison, WI 53706 USA
Motivation Finding the causes of bugs and performance problems in production systems is hard • Systems are complex and non-transparent • Multiple components, different vendors • Anomalies are common • Intermittent • Environment-specific • Users have little debugging expertise Self-propelled Instrumentation
Vision Agent Host A Process P Host B network Process Q New family of run-time tools: • Autonomous • Detailed • Low-overhead Self-propelled Instrumentation
Applications • Diagnostics of E-commerce systems • Find bugs by following the paths of individual requests through the system of Web and Database servers • Diagnostics of Cluster and Grid systems • Work without process control facilities: Red Storm, BG/L • Start up without batch scheduler support • Diagnostics of Interactive and Real-time systems • Find the causes of latency problems in multimedia • Trace a word said by one user through a phone network • Enforcement of Security • Sandboxing and controlling execution Self-propelled Instrumentation
Self-Propelled Instrumentation: Overview • Insert an initial probe into a process • The probe starts propagating • Inserts more probes ahead of the control flow • The probes may monitor or control execution • Collect run-time data • Look for anomalies in the data Self-propelled Instrumentation
Self-Propelled Instrumentation:Key Ideas • Instrumentation without an external tool • Ultra-fast code insertion • Same mechanism for applications and the kernel • Propagation • Instrument only what is necessary • Make decisions autonomously and on the fly Self-propelled Instrumentation
Inject instrumenter.so Propagate Activate a.out 83f0: 83f1: 83f3: 8400: 8405: 8413: 8414: push mov ... call mov pop ret %ebp %esp,%ebp foo %ebp,%esp %ebp bar Patch1 call call jmp instrument(foo) foo 0x8405 jmp patch jmp 8430: 8431: 8433: 8444: 8446: 8449: 844b: 844c: push mov ... int mov xor pop ret %ebp %esp,%ebp 0x80 %ebp,%esp %eax,%eax %ebp foo Patch2 ioctl int jmp (INSTRUMENT_SYS) 0x80 0x8446 jmp /dev/instrumenter OS Kernel patch jmp Patch3 sys_call: %eax *%eax push ... call ... iret instrument(%eax) *%eax 0x6d27 call call jmp 6cf5: 6d20: 6d27: 6d49: jmp
Prototype Tool: spTracer • Collects traces of function calls and returns • Activates/deactivates at run time • Trace between two user-specified events • Can trace unmodified applications • Can trace the OS kernel code • System calls – follow control flow into the kernel • Hardware interrupts Self-propelled Instrumentation
Propagation and Trace Collection • Overwrite call sites in the code • Fast: done with local memory operations • Invoke tracing routines before and after each call A patchAB B ... call B() ... return call instrument(B) call traceEntry(B) call B call traceExit(B) jmp back ... call C() ... return jmp Self-propelled Instrumentation
Visualization: Nupshot • Time-diagram of all function calls • Nested rectangles = nested function calls Self-propelled Instrumentation
Evaluation: DVD playback study • Choppy DVD playback under Linux • Same hardware works fine under Windows Self-propelled Instrumentation
Evaluation: DVD playback study • Activation • Start on the alarm timer to skip lengthy initialization/warm-up in MPlayer • Propagation • Instrument both MPlayer and the kernel • Collect function-level traces • Deactivation • Stop tracing after 250 msec (~6 frames) Self-propelled Instrumentation
Evaluation: DVD playback study User-level trace of MPlayer • Freeze is neither in a loop nor in a system call • Also happened in other places in the code • Could not discover it with cumulative metrics (gprof) • Preemption?
Evaluation: DVD playback study User-level trace of MPlayer Kernel interrupts while MPlayer was running
Evaluation: DVD playback study Kernel interrupts under magnification • The time is spent handling interrupts from the DVD drive • The root cause – ide_input_data that reads data from the drive 4 bytes at a time • Solution – enable DMA ! Self-propelled Instrumentation
Ongoing Work • Cross process and host boundaries • Jump from one process to another • Reconstruct system-wide flows • Analyze collected data for anomalies • Diagnostics of Cluster and Grid environments • Debugging the SCore batch scheduler • Joint work with Naoya Maruyama and Satoshi Matsuoka, Tokyo Institute of Technology • Diagnostics of E-commerce environments Self-propelled Instrumentation
Conclusion • Analysis by following the control flow • Autonomous, detailed, low-overhead • Analysis across the kernel boundary • Foundation for full-system analysis • For further information: • Email mirg@cs.wisc.edu • Come see the demo: Friday, CS 7355 • Paper: Mirgorodskiy and Miller, “Autonomous analysis of interactive systems with self-propelled instrumentation”, http://www.paradyn.org Self-propelled Instrumentation
