Testing And Tools For Print And Imaging Drivers

1. Testing And Tools ForPrint And Imaging Drivers

2. Outline • Basic testing philosophy • Where Microsoft finds bugs • What testing does Microsoft currently do • Microsoft provided test tools • Importance of Debugging • Summary

3. Basic Print and ImagingTesting Philosophy

4. Test Device Drivers • Right amount of QA before you ship • Fewer bugs for users to find • Lower support costs • Quality not just the responsibility of QA team • Need good tools for developers • Enable developers to run unit tests

5. Microsoft Investment • Microsoft is interested in the test environment to improve the overall driver ecosystem • Microsoft QA approach • Inbox driver program • Logo Program • Provide tools for drivers we never see • Driver education on tools and methods

6. Where Microsoft Finds Bugs and How They’re Triaged

7. Bug Buckets • Main areas bugs are found (highest to lowest) • Through code analysis (PREfast) • Printable regions • Basic printing features • Stress testing • Font handling • Advanced features • Setup • Communications and I/O • Microsoft supplies tools that target each bucket

8. What Happens if Microsoft Finds a Bug? • Reporting the bug • What did you see? • What were you expecting? • Exact steps to reproduce • Triage • Is it in Microsoft code? • Assign to a developer for investigation • Fix • Developer builds private for test to verify • Ship team decides whether to accept the change to the product • Check in • Resolve • Fix is verified against live build by test • Tester who opened original bug is the only person allowed close the issue

9. Microsoft Provided Test Tools

10. Microsoft Testing of Printing and Imaging Device Drivers • Tool Distributions • Winparty 2005 • Used for device and driver testing today • Used for inbox driver check in • Included in upcoming Windows Driver Kit (WDK) • WDK • Uses WTT Test Shell • Makes it very simple for anyone to run tests and collect results

11. What Architectures Are Supported? • Tools work on a variety of architectures • x86 • x64 • Itanium • Inbox check-in requires testing on all three architectures by the IHV and Microsoft • To test a 64bit operating system (OS) • Run 64-bit tools • Run 32-bit tools • Get 64-bit machines today • Required for WOW64 testing

12. Microsoft Provided Imaging Test Tools • Data transfer tool • Error Handler test tool • Filters test tool • Properties and Item Tree test tool • Imaging Driver Robustness tool • Twain Spec Compliance tool • Power Management Stress test tool • Twain Datasource Reliability/Robustness tool • WIA Navigation and Property tool • WIA Trace Viewer

13. Microsoft Provided Print Test Tools • Print Device Capabilities • Print Area • Photo Print test • Print Gremlin • Office Tests • Inbox App tests • DevMode Corruptor • Print Driver Stress • Font Test

14. Importance of Debugging

15. How Does a Debugging Environment Help? Increases stability of product Increases customer satisfaction Increases tester output Increases developer output Lowers cost of development Lowers cost of testing Lowers support costs

16. Invest in Your QA Process • Example of testing without debugging: • Tester notices applet crashes every few minutes, disregards break • Example with debugging • Tester notices applet crashes, creates debugger break • Passed on to developer, developer finds heap corruption ( developer finds code bug ) • Fixed before driver is released • Avoids support costs • Increases stability of product • Avoids customer dissatisfaction • Every single debugger break is a bug to investigate

17. Kernel vs. Local Debugging • Easy to set up:Two approaches • User mode debugging • Easy to set up • Lots of cmd windows on screen • Can’t catch kernel-mode bugs • Kernel mode debugger • Harder to set up (separate computer needed) • No visible testing machine effects • Catches kernel-mode bugs • Debugging user mode drivers can be a problem • e.g. Spooler context and loading symbols

18. Debugging Tools • PREfast for drivers • Performed mostly by developers • You need to have the commitment to fix all the errors • Zero bug count required for inbox check-in • Configurable through filters • Application Verifier, Driver Verifier, clean test systems, Pageheap, NTSD etc. • Expands scope of testing • Performed by developers and testers • Very important for quality testing

19. AppVerif vs. Verifier • Application Verifier = appverif.exe • Driver Verifier = verifier.exe • Verifier is a kernel-mode driver debugging tool • Verifer was and continues to be widely adopted to find and fix kernel mode problems • IHVs have found and fixed lots of bugs because of Verifer since Windows 2000 • AppVerif is a user-mode driver and application debugging tool • Used when testing user mode components

20. Good Test Debug Setup • Development resources to debug a break • User mode or kernel mode debugger • “Clean” test machine • Make use of imaging software • Latest debug tools on debugger and test machine • We have a new tool to make debugging set-up easier! • Winparty 2005

21. Good Test Debug Setup (con’t) • Application Verifier (user mode debugger) • Spooler • Explorer • WIA • IHV spawned process • All test tools • Driver Verifier • All installed kernel-mode components • NTSD is the Microsoft standard user-mode debugger • Enable NTSD on spooler • Enable NTSD on WIA service • Enable NTSD on all IHV-spawned processes • Debugging is for test and development

22. PREfast Overview • Static code analysis tool • Run against source code • Automated tool to find code defects • Not limited by test cases • Identify location of bug (easy to fix) • Run on source code, not binaries • Analyzes code for bugs before expensive testing begins

23. PREfast Overview (con’t) • Checks for common coding bugs • Bounds (array bounds violations) • Precedence & stack hogs • HRESULT (abuses of HRESULT type) • Divide by zero errors • Null pointers/invalid pointers It’s just a tool, not a magic bullet • Complements other tools such as Application Verifier, Pageheap, etc. • Does this tool find all the defects? No!

24. PREfast Buffer Over-run Example extern TCHAR g_szName[MAX_PATH + 1]; static TCHAR c_szServerName[] = "SERVER_NAME"; DWORD dwSize = sizeof(g_szName); TCHAR szAnsiName[MAX_NAME_LENGTH + 1]; pECB->GetVariable (pECB->ConnID, c_szServerName, szAnsiName, &dwSize); Security bulletin (and patch) MS01-023 Detected by PREfast 4.0: Fixed in Windows XP

25. Summary • Microsoft provides many printing and imaging test tools • Proper debugging saves time and money and increases customer satisfaction • Testing is not just for testers • Invest in your QA org

26. Call to Action • Use Microsoft tools and methods to improve the efficiency of your QA org • Setup proper debugging to find bugs • Work with Microsoft to improve the stability of your device drivers

27. Additional Resources • Community Sites • http://www.microsoft.com/communities/default.mspx • MSDN Developer Community Chats: • Windows Printer Driver Development http://msdn.microsoft.com/chats/transcripts/windows/windows_092104.aspx • Windows Drivers: Printer Drivers http://msdn.microsoft.com/chats/windows/windows_101602.asp • Windows Drivers Printing and Networking http://msdn.microsoft.com/chats/windows/windows_022002.asp • Web resources • WinFXhttp://msdn.microsoft.com/downloads/ • WS-Devices Profile http://msdn.microsoft.com/webservices/understanding/specs/default.aspx?pull=/library/en-us/dnglobspec/html/devprof.asp • Printer Working Group http://www.pwg.org • WHDC Printing home page: http://www.microsoft.com/whdc/device/print/default.mspx • WHDC Still Imaging / WIA home page: http://www.microsoft.com/whdc/device/stillimage/default.mspx

28. Additional Resources • E-mail • For developer print questions: prninfo @ microsoft.com • For developer scan questions: wiainfo @ microsoft.com • For developer color questions: mscolor @ microsoft.com

29. © 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.