1 / 31

Measuring Interactive Performance with VNCplay at Stanford University

Learn how to measure performance for interactive applications efficiently using VNCplay. Understand the importance of response time in addition to total runtime benchmarks for accurate assessment. Discover the methods, tools, and future directions for evaluating interactive performance.

dbabin
Download Presentation

Measuring Interactive Performance with VNCplay at Stanford University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring InteractivePerformance with VNCplay Nickolai Zeldovich, Ramesh Chandra Stanford University

  2. Measuring performance • Scientific computing or server workloads: • How long does it take for my program to run? • Measure total running time

  3. Measuring performance • Users of interactive applications: • I asked for something, when will I get it? • But we still measure total runtime • WinBench runs Word, PowerPoint, etc as fast as possible, and measures total runtime We are measuring the wrong thing

  4. Case in point • Use Microsoft PowerPoint for 5 minutes • Two machines: 2.0 GHz and 300 MHz • Expect to see different performance! • Two performance metrics: • Total runtime: from start to finish • Response time: from click to output on screen

  5. Performance Metrics Total Runtime Response Time

  6. Users pause a lot . . . . . . 0 sec 0.1 sec 0.2 sec 5 sec Response time is noise in total runtime

  7. Outline • Overview of VNCplay • Current approaches • How VNCplay works • Evaluation • Quantitative results • Qualitative experience • Summary

  8. Overview of VNCplay • To compare interactive performance, we need reproducible workloads • Record interactive user session • Replay session in different environments • Compare response time between replays

  9. Current approaches Display protocols: X11, VNC, Windows Key strokes, mouse clicks Screen updates

  10. Current approaches . . . Input Event Time 0 sec 1 sec 5 sec 8 sec Recorder Screen updates Tools like Xnee, Visual Test, AutoIt, rfbplaymacro

  11. Current approaches . . . Input Event Time 0 sec 1 sec 5 sec 8 sec Replayer Screen updates

  12. What if system is slow? 0 sec 0.5 sec 1.0 sec 1.5 sec Time Need feedback of when the system responds

  13. What should happen? 0 sec 0.5 sec 1.0 sec 1.5 sec Time Need feedback of when the system responds

  14. Workarounds? • Current tools allow user to insert delay statements to compensate for slowness • Creates unrealistic user behavior • Tedious and error-prone

  15. VNCplay approach • Big problem is mouse clicks • Take a screen snapshot at each mouse click during recording • Wait for the same thing during replay

  16. VNCplay approach Input Event Time 0 sec 1 sec 5 sec 8 sec Recorder Screen updates

  17. VNCplay approach Input Event Time 0 sec 1 sec 5 sec 8 sec Replayer Screen updates

  18. VNCplay details • Only snapshot the screen around the cursor • Reduces trace file size • Ignores irrelevant things (e.g. system clock) • Other tricks described in the paper

  19. Analyzing Replays System 1: Match Response Time System 2: Time

  20. Evaluation • OpenOffice on Linux • Scaled CPU frequency from 2.0 GHz down to 300 MHz • Compare total running time and interactive response time

  21. Total Runtime

  22. Median Response Time

  23. Other experiments • Paper has details on more experiments • Microsoft Windows, Linux • Evaluate interactive performance of different disk IO schedulers

  24. Experience using VNCplay • Platform independence • Measured interactive performance of PowerPoint, Word, OpenOffice • Reliable replay • Replayed 5-minute PowerPoint session on a slow system, taking over an hour – works OK!

  25. Caveats • Near-perfect reproducibility needed • VMware: start the same VM every time • In Linux, reset user account (dotfiles)

  26. Caveats • Keyboard input can be a problem • Use mouse instead of keyboard shortcuts • Click before and after you type • Couple of tries to get a reliable session • Tooltips, pop-ups get in the way

  27. Using VNCplay • vncplay record server:5901 trace.vnc • vncplay play server1:5901 trace.vnc out1.rfb • vncplay play server2:5901 trace.vnc out2.rfb • vncplay analyze out1.rfb out2.rfb > analyze.out • vncanalyze median analyze.out • vncanalyze cdf analyze.out > cdf.plot Use gnuplot, Excel to plot cdf.plot

  28. Other uses • Automated testing • Make sure your application behaves the same way with a new release of Linux / Windows • GUI ``automation’’ • Automate GUI-intensive tasks that have no scriptable interface

  29. Future directions • Integrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips) • Better keyboard input handling • Machine learning • Details in paper • If you want to help, let us know!

  30. In conclusion … • Need to measure interactive performance • Runtime benchmarks are not appropriate • VNCplay • Portable, reliable session replay • Response time measurement

  31. Questions? http://suif.stanford.edu/vncplay/

More Related