260 likes | 531 Views
Improving Driver Concurrency Testing. James Moe and Dieter Achtelstetter Windows Devices & Storage Technologies drvtest@microsoft.com. Agenda. Concurrency and driver bugs Observations about our existing tests A tool for looking at IRP concurrency
E N D
Improving Driver Concurrency Testing James Moe and Dieter Achtelstetter Windows Devices & Storage Technologies drvtest@microsoft.com
Agenda • Concurrency and driver bugs • Observations about our existing tests • A tool for looking at IRP concurrency • How we used that data to improve our tests • The new Windows 7 tests in depth • How you can use these tools to improve your in-house tests in respect to concurrency
Concurrency and Drivers • Driver are highly concurrent • Much of a driver’s complexity derives from concurrency • Many late or post-ship bugs are related to concurrency
Observations about Our Existing Tests • Tests tend to be very specific • PnP tests do PnP • I/O tests do I/O • Existing tests tend not to work well when run side by side • Unclear what concurrency scenarios they cover
Introducing IRP Concurrency Coverage • Provides an additional coverage dimension to code coverage • Can identify test coverage holes • Complements other tools like code coverage, static analysis, etc.
IRP Pair Coverage • A form of concurrency coverage • A pair is covered if: • The 2 IRPs making up the pair are active within a single driver stack • 1326 pairs • From 52 IRPS • Kernel rules removed 227 pairs; for example: • Only one state-changing PnP IRP active within a stack • Leaves 1099 pairs
Why Pairs? • Too many combinations of all possible IRPs • Equal distribution through the problem space • Improvements visible as percentage of coverage
Data Perspectives • 2 identical pair hits are not identical • A pair hit doesn’t necessarily indicate a valuable test case • Repetitive pair hits increase the odds of valuable test cases • Low number of hits per pair indicates missing test coverage
What’s New in PnPDTest – More Than Just PnP • Coupling I/O with PnP IRPs • Configurable—old test functionality is still there • IOCTLs • Read/write • Power
PnPDTest - I/O and PnP • IOCTL • Generic buffered IO • Planning to integrate more DC2 functionality • Read/Write • Uses Windows Device Testing Framework (WDTF) SimpleIO • Existing WDTF SimpleIO interfaces for net, audio, video, and volume • Extensible for other device types
PnPDTest – Power + I/O, Power +PnP • Uses WDTF interfaces to set system power, set timers for wake • Combine S-IRPs and resulting D-IRPs with IOCTL and Read/Write • Combine power IRPs with PnP IRPs • All three together!
Device Path Exerciser (DC2), Static Tools, WLK Scripts • Work is underway to modify DC2 to play well with other tools • Goal is to get features like I/O fuzzing to run concurrently with tools like PnPDTest, Sleep Stress with I/O, etc. • Some functionality is being integrated into other tools • Static Tools (Static Analysis, PREfast for Drivers) • Look at IRP pair coverage data to find holes • Use that data to identify areas where static analysis could find bugs in untested areas • Sleep Stress with I/O, Disable_Enable_With_IO • Work on improving concurrency in these as well
Concurrency Coverage Tools • Enable you to do your own analysis on in-house tests • Tools will be available in the Windows 7 Beta
How to Improve Concurrency Coverage Today • Combine existing tests • Simple and low tech • Look for crashes and hangs • Write your own device-specific tests • You know the behavior of your device and driver best
Create Your Own Device-specific Test Cases • Combine your device-specific operations with: • PnP • Disable / Enable device • Surprise remove device • Power • Close laptop lid • Put system to sleep • Examples • I/O IRPS • Query and change device data • WMI and IOCTL IRPs
Automate These Test Cases • Run our tests and your device-specific tests side-by-side • WDTF test scripts we supply in the WLK • PNPDTEST • DC2 • Write a WDTF Simple I/O plug-in for your device class • Existing WDTF scripts and PNPDTEST will use this plug-in • Enables you to easily write your own combinatorial WDTF test scripts using PnP and power
Call to Action • Use the new tests in your regular day-to-day testing • Use the Concurrency Coverage tools to identify areas for additional concurrency testing • Improve your in-house tests to get better concurrency coverage
Resources • WLK and WDK on the WHDC Web site http://www.microsoft.com/whdc/DevTools/WDK/WDKpkg.mspx • WDTF documentation on MSDNhttp://msdn.microsoft.com/en-us/library/aa972915.aspx • Send feedback to drvtest@microsoft.com