Best Practices For Testing Windows Drivers

1. Best Practices For Testing Windows Drivers Dieter AchtelstetterSoftware Design EngineerWindows Device Experience GroupMicrosoft Corporation

2. About This Presentation • Fulfill a request from OEM and IHV • For best practices on testing • Suggestion for • Ways to reduce support costs for you and your customers • Increasing your customer satisfaction • Finding more bugs earlier in the process, where they costa lot less to fix • This advice is based on the experience of many device testers at Microsoft • While this session attempts to cover most drivers/devices, some points may not be applicableto your driver/device

3. Agenda • Overview • Tools • Test configurations • Testing specific driver aspects • Types of testing • Resources • Call to action

4. Overview Of Topics Static tools PREfast SDV Run-time tools Driver Verifier Test configurations System Settings Driver aspects Memory PO(Power Management) Penetration/Security I/O Cancel Test strategies Test early Plan for testing PnP Setup Concurrency Testing types Stress tests Manual testing Resources DTM kits Test Framework Long haul testing Scenario testing

5. Static Analysis Tools • Tools that run against the source code • “PREfast for Drivers” • “Static Driver Verifier” • Find bugs fast • Quick and inexpensive ways to find bugs • Must pass these tests for driver signatures or logo • Run early/run often • Use these tests early in your development cycle • Resolve issues found by the tools • Run the tests for every new driver drop • For details • At WinHEC: “Static Analysis and Verification of Drivers” • Tools and documentation: WDK

6. Driver Verifier • Verifies different aspects of drivers at runtime • Built into Windows • Type verifier from the Run menu • Valuable in your regular testing • Use standard settings • Does Special pool, IRP verification, IRQL checking, etc. • Same settings are used for driver signature or logo testing • Only enable settings for drivers under test • To ensure proper testing of your driver • Get high test coverage • By running with a full test pass • You must exercise the code; otherwise, it cannot verify the driver • For details • Tools and documentation: WDK • Many new features for Windows Vista

7. Test Configuration • Vary your test environment • Don’t just test your device in a static environment – change things around • System configuration • System environments • Device settings • Make these part of your regular test pass

8. System Configuration • Different platforms: x86, x64, Itanium • System with S1 and S3 support (for PO testing) • Ensure system has WDDM video drivers • So you get hybrid sleep (S3 with Hiberfile) • Multiprocessor/multi-core machines • Only way to truly test concurrency • Hyperthreading doesn't count for this testing • Different chipsets • Checked build • Skews timing • Has additional assertions • See “Using the Checked Build” in the WDK

9. System Environments • Low memory • See “Boot Parameters to Manipulate Memory” in the WDK • Large memory • This is important for both for X86 and 64 bit • See “Will I need special hardware for memory top-down testing?”in the WDK • High CPU utilization • From something other than testing the device • Stress testing does this • Skews timing • Can uncover race conditions in your driver

10. Device Settings • Test your device with all of its settings • Not just the default settings • Only you know what these settings do • If your device uses hardware resources (IRQ, I/O Port...), change them • WDK tool “PnP Driver Test”does rebalancing • Manually in Device Manager

11. Device Configuration • Test several of the same devicein a system • Topologies • Behind a PCI-PCI bridge • Connected to a large tree of devices • For USB or 1394 device, for example • …

12. Testing Specific Driver Aspects • Aspects of driver testing that maynot usually be your focus • But tend to have high failure counts Driver aspects Memory PO Penetration/Security I/O Cancel PnP Setup Concurrency

13. Memory • Still a problem area with drivers • Corruption – testing methods • Driver Verifier with special pool setting • Static analysis tools • Code review • Stress testing • Leaks – testing methods • Code review • Manual testing • Use “PoolMon” In the WDK to monitor pool allocations • Generally, do repetitive operations, and look for continuedpool usage increases

14. Plug And Play – PnP • WDK tests • “Disable/Enable with IO” • “PnP Driver test” • Long haul testing • Set up system that tests PnP for several days • Test the driver/device in docking stations • Test undock/dock • Manual testing • Physically unplug and replug • The device • The bus/media: USB cable, net cables... • Use your device • Then disable/enable it

15. Power Management • Windows Vista = more aggressive PO management • Not just for laptops anymore • WDK tests • “Sleep Stress with IO” – cycles through sleep statesand does device I/O before and afterward • “Pwrtest” – Logs power events • Long haul testing • Set up system that tests PO for several days • Manual testing • Use the device, and then • Put the system to sleep • Hibernate the system

16. PnP And Power Management • Test both together • Don’t just test each in isolation • This is a real-world scenario • For example, often when you close thelid on a laptop, you also unplug devices • WDK tests • Common Scenario Stress with IO • Manual testing • Combine existing automated tests • Or do these manually

17. Fault Injection • Test driver code paths that are hard to hit under normal use • Error handling • Corner cases • Hard to test • You may need to add test code to the driverduring testing • Write a test filter driver • Driver Verifier has a Fault Injection setting • For USB, try Device Simulation Framework

18. Concurrency • Do different things to the driver at the same time • Analysis of our tests show they do not give very good coverage in this area • System configuration • Use multiprocessor machines • Testing • Use your device in multiple processes • Use your device and • Disable it • Shut down the system • Put the system to sleep • Change the configuration • Get data from it (WMI/IOCTL) • …

19. Penetration/Security • Be resilient against mischief • Don’t assume that only your user-mode code will be calling your device • Code review the IOCTL and WMI code path • Who should have access to your devices: Test for it • See “Creating Reliable Kernel-Mode Drivers” in WDK • Errors in buffered I/O, etc. • Test and code review for these pitfalls • Device path exerciser in WDK test – limited but useful • Run with Driver Verifier on target driver enabled • For bugs found with this tool, do a code review • Look for similar bugs in other code paths • Where bug was found, look for other types of bugs • Don’t expect this tool to find other bugs in this area for you • Use static analysis and verification tools

20. I/O Cancellation • Still a problem area with drivers • Hard to test • I/O must be currently active in the driver • Testing • Use your device, and then kill the process using it • Doesn’t guarantee you tested I/O cancellation • Set a break point or add tracing on cancellation code • Make sure it gets hit during your testing • May need to add test code to your driver • Experiment

21. Setup Packages • Test for all scenarios • OEM preinstall • During OS installation (for boot drivers) • After OS installation • OS Migration • Install your driver on Windows XP, then upgrade to Windows Vista and ensure drivers and applets work • Same for Windows Vista to Windows Vista upgrades • For example, Home Basic to Home Premium SKUs • Test with multiple packages • Create 2 packages with 2 different versions of your driver • Install version 1 • Upgrade to version 2 • Rollback to version 1 • Un-install • Tools: ChkInf

22. Testing Types Testing types Stress tests Scenario testing

23. Stress Testing • Combines tests to simulate a heavy load • Tests your driver under different system conditions • Can find bugs that can be missed in feature testing • Stress tests • WDK has a stress test • Allows you to add your device-specific test • Build your own • Testing – time and configuration • Run for at least 24 hours at a time • Run on different system configuration • Run with different device settings

24. Scenario Testing • Test your devices the way customerswill use them • Don’t just rely on synthetic tests/automation • Focus on corner cases scenarios • Mainstream scenarios tend to notbe a problem • Customers always find new waysto use your devices

25. Resources • Driver Test Manager kits – DTM • Test frameworks • Windows Device Testing Framework – WDTF • Device Simulation Framework – DSF • Driver Frameworks • Kernel-mode Driver Framework – KMDF • User-mode Driver Framework – UMDF • Windows Error Reporting

26. DTM Test Kits • Run early • When software, firmware, and hardware changes are still possible • Use these tools on a daily basis in your testing • Keep in mind that driving issues to a resolution takes time: Plan for 2-4 weeks • Many useful DTM tests for your regular testing • Use DTM automation features in your lab • Add your own tests • At WinHEC: WDK and DTM presentations • “How to Use the WDK to Develop, Sign and Test Drivers” • “Using the WDK for Windows Logo and Signature Testing” • Sign up for the WDK Beta: www.microsoft.com/whdc/driver/wdk/betawdk.mspx • See WHDC Web site: www.microsoft.com\whdc • “Index of Windows Driver Kit Tools” in the WDK

27. WDTF – Windows Device Testing Framework • Framework for building driver/device tests • Some DTM tests are built using WDTF • PnP and Power Management tests • Write your own tests • For details • At WinHEC: “Using the Windows DeviceTesting Framework” • Tools and documentation: WDK

28. DSF – Device SimulationTest Framework • Simulate hardware through software • Currently only for USB • For details • At WinHEC: “Using the Device Simulation Framework for Software Simulation of USB Devices” session • WDK • DSF USB Loop Back Device Simulation

29. Windows Driver Frameworks (WDF) • Generally, testing is same as for other drivers • Some features that might help

30. Windows Error Reporting • Database of Windows crashes • Improve experience/reduce costs • Improve your customers’ experience • Reduce your support costs – and your partners’ • Engage through Winqual • Sign up to see crash data for your drivers • Provide solutions through Windows Update • Check data regularly • For just-released products, check daily • For products on the market for a while, check monthly • For details • Winqual Web site: https://winqual.microsoft.com

31. Call To Action • Incorporate the testing points from this presentation into your regular test pass • Test early • Start in the design and prototyping phase • Take advantage of the resources listed here • Engage with Windows Error Reporting • Engage with driver testers at Microsoft • Drvtest @ microsoft.com

32. Additional Resources • WDK: http://www.microsoft.com/whdc/wdk • Testing tools on WHDC: http://www.microsoft.com/whdc/DevTools/tools/ • Driver aspects • PO: http://www.microsoft.com/whdc/system/sysperf/resumeperf.mspx • Memory: http://www.microsoft.com/whdc/driver/security/mem-alloc_tst.mspx • Setup: “Using Driver Install Frameworks (DIFx)” – WDK • Microsoft Symbol Server • All OS symbols for all releases and patches: http://www.microsoft.com/whdc/devtools/debugging/ • Send email: Windows Driver testing alias • Drvtest @ microsoft.com

33. Questions • Tomorrow’s “Ask the expert” session

34. © 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.