310 likes | 535 Views
Measuring Interactive Performance with VNCplay. Nickolai Zeldovich, Ramesh Chandra Stanford University. Measuring performance. Scientific computing or server workloads: How long does it take for my program to run? Measure total running time. Measuring performance.
E N D
Measuring InteractivePerformance with VNCplay Nickolai Zeldovich, Ramesh Chandra Stanford University
Measuring performance • Scientific computing or server workloads: • How long does it take for my program to run? • Measure total running time
Measuring performance • Users of interactive applications: • I asked for something, when will I get it? • But we still measure total runtime • WinBench runs Word, PowerPoint, etc as fast as possible, and measures total runtime We are measuring the wrong thing
Case in point • Use Microsoft PowerPoint for 5 minutes • Two machines: 2.0 GHz and 300 MHz • Expect to see different performance! • Two performance metrics: • Total runtime: from start to finish • Response time: from click to output on screen
Performance Metrics Total Runtime Response Time
Users pause a lot . . . . . . 0 sec 0.1 sec 0.2 sec 5 sec Response time is noise in total runtime
Outline • Overview of VNCplay • Current approaches • How VNCplay works • Evaluation • Quantitative results • Qualitative experience • Summary
Overview of VNCplay • To compare interactive performance, we need reproducible workloads • Record interactive user session • Replay session in different environments • Compare response time between replays
Current approaches Display protocols: X11, VNC, Windows Key strokes, mouse clicks Screen updates
Current approaches . . . Input Event Time 0 sec 1 sec 5 sec 8 sec Recorder Screen updates Tools like Xnee, Visual Test, AutoIt, rfbplaymacro
Current approaches . . . Input Event Time 0 sec 1 sec 5 sec 8 sec Replayer Screen updates
What if system is slow? 0 sec 0.5 sec 1.0 sec 1.5 sec Time Need feedback of when the system responds
What should happen? 0 sec 0.5 sec 1.0 sec 1.5 sec Time Need feedback of when the system responds
Workarounds? • Current tools allow user to insert delay statements to compensate for slowness • Creates unrealistic user behavior • Tedious and error-prone
VNCplay approach • Big problem is mouse clicks • Take a screen snapshot at each mouse click during recording • Wait for the same thing during replay
VNCplay approach Input Event Time 0 sec 1 sec 5 sec 8 sec Recorder Screen updates
VNCplay approach Input Event Time 0 sec 1 sec 5 sec 8 sec Replayer Screen updates
VNCplay details • Only snapshot the screen around the cursor • Reduces trace file size • Ignores irrelevant things (e.g. system clock) • Other tricks described in the paper
Analyzing Replays System 1: Match Response Time System 2: Time
Evaluation • OpenOffice on Linux • Scaled CPU frequency from 2.0 GHz down to 300 MHz • Compare total running time and interactive response time
Other experiments • Paper has details on more experiments • Microsoft Windows, Linux • Evaluate interactive performance of different disk IO schedulers
Experience using VNCplay • Platform independence • Measured interactive performance of PowerPoint, Word, OpenOffice • Reliable replay • Replayed 5-minute PowerPoint session on a slow system, taking over an hour – works OK!
Caveats • Near-perfect reproducibility needed • VMware: start the same VM every time • In Linux, reset user account (dotfiles)
Caveats • Keyboard input can be a problem • Use mouse instead of keyboard shortcuts • Click before and after you type • Couple of tries to get a reliable session • Tooltips, pop-ups get in the way
Using VNCplay • vncplay record server:5901 trace.vnc • vncplay play server1:5901 trace.vnc out1.rfb • vncplay play server2:5901 trace.vnc out2.rfb • vncplay analyze out1.rfb out2.rfb > analyze.out • vncanalyze median analyze.out • vncanalyze cdf analyze.out > cdf.plot Use gnuplot, Excel to plot cdf.plot
Other uses • Automated testing • Make sure your application behaves the same way with a new release of Linux / Windows • GUI ``automation’’ • Automate GUI-intensive tasks that have no scriptable interface
Future directions • Integrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips) • Better keyboard input handling • Machine learning • Details in paper • If you want to help, let us know!
In conclusion … • Need to measure interactive performance • Runtime benchmarks are not appropriate • VNCplay • Portable, reliable session replay • Response time measurement
Questions? http://suif.stanford.edu/vncplay/