1 / 26

Improving Driver Concurrency Testing

Improving Driver Concurrency Testing. James Moe and Dieter Achtelstetter Windows Devices & Storage Technologies drvtest@microsoft.com. Agenda. Concurrency and driver bugs Observations about our existing tests A tool for looking at IRP concurrency

sasha
Download Presentation

Improving Driver Concurrency Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving Driver Concurrency Testing James Moe and Dieter Achtelstetter Windows Devices & Storage Technologies drvtest@microsoft.com

  2. Agenda • Concurrency and driver bugs • Observations about our existing tests • A tool for looking at IRP concurrency • How we used that data to improve our tests • The new Windows 7 tests in depth • How you can use these tools to improve your in-house tests in respect to concurrency

  3. Concurrency and Drivers • Driver are highly concurrent • Much of a driver’s complexity derives from concurrency • Many late or post-ship bugs are related to concurrency

  4. Observations about Our Existing Tests • Tests tend to be very specific • PnP tests do PnP • I/O tests do I/O • Existing tests tend not to work well when run side by side • Unclear what concurrency scenarios they cover

  5. Introducing IRP Concurrency Coverage • Provides an additional coverage dimension to code coverage • Can identify test coverage holes • Complements other tools like code coverage, static analysis, etc.

  6. IRP Pair Coverage • A form of concurrency coverage • A pair is covered if: • The 2 IRPs making up the pair are active within a single driver stack • 1326 pairs • From 52 IRPS • Kernel rules removed 227 pairs; for example: • Only one state-changing PnP IRP active within a stack • Leaves 1099 pairs

  7. What Is an active IRP?

  8. Why Pairs? • Too many combinations of all possible IRPs • Equal distribution through the problem space • Improvements visible as percentage of coverage

  9. Data Perspectives • 2 identical pair hits are not identical • A pair hit doesn’t necessarily indicate a valuable test case • Repetitive pair hits increase the odds of valuable test cases • Low number of hits per pair indicates missing test coverage

  10. Example: Full Test Pass Metrics

  11. Example of Pair Coverage

  12. Tests Comprising the Example Test Pass

  13. Test Improvement Focus

  14. What’s New in PnPDTest – More Than Just PnP • Coupling I/O with PnP IRPs • Configurable—old test functionality is still there • IOCTLs • Read/write • Power

  15. PnPDTest - I/O and PnP • IOCTL • Generic buffered IO • Planning to integrate more DC2 functionality • Read/Write • Uses Windows Device Testing Framework (WDTF) SimpleIO • Existing WDTF SimpleIO interfaces for net, audio, video, and volume • Extensible for other device types

  16. PnPDTest – Power + I/O, Power +PnP • Uses WDTF interfaces to set system power, set timers for wake • Combine S-IRPs and resulting D-IRPs with IOCTL and Read/Write • Combine power IRPs with PnP IRPs • All three together!

  17. Device Path Exerciser (DC2), Static Tools, WLK Scripts • Work is underway to modify DC2 to play well with other tools • Goal is to get features like I/O fuzzing to run concurrently with tools like PnPDTest, Sleep Stress with I/O, etc. • Some functionality is being integrated into other tools • Static Tools (Static Analysis, PREfast for Drivers) • Look at IRP pair coverage data to find holes • Use that data to identify areas where static analysis could find bugs in untested areas • Sleep Stress with I/O, Disable_Enable_With_IO • Work on improving concurrency in these as well

  18. Old PnP Test coverage

  19. New PnP Test coverage

  20. Concurrency Coverage Tools • Enable you to do your own analysis on in-house tests • Tools will be available in the Windows 7 Beta

  21. How to Improve Concurrency Coverage Today • Combine existing tests • Simple and low tech • Look for crashes and hangs • Write your own device-specific tests • You know the behavior of your device and driver best

  22. Create Your Own Device-specific Test Cases • Combine your device-specific operations with: • PnP • Disable / Enable device • Surprise remove device • Power • Close laptop lid • Put system to sleep • Examples • I/O IRPS • Query and change device data • WMI and IOCTL IRPs

  23. Automate These Test Cases • Run our tests and your device-specific tests side-by-side • WDTF test scripts we supply in the WLK • PNPDTEST • DC2 • Write a WDTF Simple I/O plug-in for your device class • Existing WDTF scripts and PNPDTEST will use this plug-in • Enables you to easily write your own combinatorial WDTF test scripts using PnP and power

  24. Call to Action • Use the new tests in your regular day-to-day testing • Use the Concurrency Coverage tools to identify areas for additional concurrency testing • Improve your in-house tests to get better concurrency coverage

  25. Resources • WLK and WDK on the WHDC Web site http://www.microsoft.com/whdc/DevTools/WDK/WDKpkg.mspx • WDTF documentation on MSDNhttp://msdn.microsoft.com/en-us/library/aa972915.aspx • Send feedback to drvtest@microsoft.com

More Related