Building high-quality Windows PCs using the Assessment and Deployment Kit

2. TOL-147T Building high-quality Windows PCs using the Assessment and Deployment Kit Sylvain Goyette Scenario Adoption PM Microsoft Corporation

3. Agenda • ADK introduction • Assessment tools scenarios • Assessment investments • Windows Assessment Console • Windows Assessment Services • Assessment execution engine • Windows Performance Toolkit (WPA/WPR)

4. Assessment and Deployment Kit

5. Assessment and Deployment Kit • Consolidate OPK toolkit and WAIK • Core deployment tools and assessment toolkit • Create, assess, and maintain quality Windows images • How we measure the PC today lacks the needed customer view • Processor speed, megahertz, gigabytes, etc. • How exactly does this relate to an individual’s experience? • The ADK system assessment tools help engineers deliver high-quality, differentiated devices that measure system quality through a customer experience • One-stop shop for all partners

6. System assessment basics • System assessment is a process that uses the ADK tools to measure and analyze a PC • Assessments are core functionality tests • Combinations of these tests provide additional measures of the entire PC experience • Quality expectations are changing • Software + hardware + Windows = PC experience • The way we measure PC quality must also change

7. Benchmarking doesn’t go far enough • Advantages • Accepted and easy to discuss by techies • Used as part of certification or comparison • Widely adopted, especially in the press • Disadvantages • Provided with little context on how benchmarks are created • A “black box” to users • Less useful in system engineering

8. System assessments • CheckLogo and driver assessments • File handling • Photo handling • Internet Explorer launch/tab create • Hybrid boot • On/off assessments (boot/shutdown/S3/S4) • Browser assessment • Media transcode performance • Metro performance • Memory footprint • First boot experience • Media streaming • WinSAT comprehensive • Battery life (and idle efficiency analysis) • MiniFilter driver performance impact (option for other assessments) • Internet browsing workload for battery life assessment • Windows Media Player performance and quality

9. Assessment tenets • System quality assessments should: • Focus on assessing the end user’s experience, not just the underlying technologies • Assess system quality in a quantifiable way • Allow for system comparisons • Identify issues and provide remediation steps wherever possible • Provide the ability to dig deeper using tools like Windows Performance Analyzer • Make collaboration easier • Be widely available

10. Windows assessment tools

11. Evolution of tools ADK assessment tools Velocity Test Suite (VTS / VOS) Velocity 2.x (VOS) Velocity 4.x Velocity 3.x (VOS) Fundamental Quality Tools Suite (FQTS) SPPTS (WLK 1.4) Velocity (VTS) Velocity 3.x (VTS) SPPTS (WLK 1.5) System Power and Performance Tools Suite (SPPTS) FQTS 2.x FQTS v3

12. Overview of problems with existing tools • Existing tools reuse solutions designed to assess quality of single computer • Introduce significant manual work, which grows as number of computers increases: • Computer preparation (image deployment and drivers installation) • Assessment execution and monitoring progress • Collection of results from computers • Assessing quality of multiple computers

13. Two ways of using assessments Single Machine and Basic Architecture Lab scenarios and Scalability

14. What You Get in the Windows ADK • Remediation tool used for analysis of performance traces WPA Windows Performance Analyzer Used to access, configure, create, save, package jobs to run on different machines. Good for running assessments on one to a few machines. Used to view results from assessment run Scenario-based tests of system quality, delivering metrics, identifying issues, and providing remediation steps wherever possible ADK Assessment and Deployment Kit WAC Windows Assessment Console Assessments WAS Windows Assessment Services Used for larger environments with the potential of running assessments on hundreds of computer

15. Three key concepts • Jobs run, not assessments • Jobs contain one or more assessments • Jobs contain the parameters that are used to configure individual assessments (as well as a few job parameters) • All jobs run in administrator mode • It’s a requirement of the Assessment Execution Engine (AXE) • It’s the assessment that has the “power” • The GUI and platform provide a test harness to expose things the assessment wants to expose—it’s the assessment that does (and should do) most of the heavy lifting, analysis, and subject matter expertise

16. demo Assessments, jobs, workloads, and the Windows Assessment Console

17. Assessment infrastructure Windows Assessment Service (WAS and WASC) – lab environment Discover Compose Execute Interpret Monitor Diagnose Manage • Alignment with WAC on the four primary goals • Extends concept to distributed environment with multiple assets (machines, images, assessments) • Adds capability to manage assets, monitor progress on multiple jobs and assets, and facilitate centralized diagnosis

18. Assessment Execution Engine • AXE core components: • AXE.exe • AL.exe • Etc. • Small set of binaries to support “platform” • XML-driven definition and configuration • Assessment manifest defines the assessment, including parameters, metrics, thresholds, etc. • Job manifest defines the job (which assessments to run and parameter values) • Results file defines the output • Each “subsequent” XML contains the previous files in their entirety • Ability to launch any code/script/process • Assessment resources: • asmt.exe • Assessmentmanifest.xml • Axecore.dll • Other resources (including samples)

19. Integration goals • Modular design provides multiple ways to extend and varying levels of integration • Provide both interactive and automated tools • APIs and schema will be made available publicly • Both native and managed execution APIs for running assessments • Assessment, job, and results XML schema definitions • Advanced managed administrative APIs for creating XML documents We used these same APIs and schemas to produce the tools and assessments delivered in the kit.

20. Ways to integrate and extend • Convert or build your own assessments • Automate execution in your existing infrastructure • Import the results • Create jobs and packages

21. Windows Assessment Services (WAS)

22. Windows Assessment Services • Infrastructure to support lab environments • Automation to support large scale deployment and assessment of many machines • Results database allows for system comparison, analysis, and reporting

23. Automation for the lab environment • Computer preparation • Using Windows Deployment Services (WDS) and customized WinPE image • Assessment execution • Using Windows Remote Management (WinRM) to copy assessment execution engine + assessments to computers and run them • Collection of results from computers • Using WinRM to copy results to lab execution server and preprocess them • Supports multiple computer scenarios

24. End-to-end workflow

25. What WAS is not? • Assessment execution in the cloud / over the Internet. It is designed to work within a single subnet. • Windows Hardware Certification framework • Testing computers on manufacturing floor • Large scale testing (there is an limit of 300 computers) • Testing computers that are already shipped and are in use by end users • Debugging / fixing issues that are found by assessments • Issue management

26. WAS results database • Provide an optional platform for aggregated results collection based on the AXE assessment results schema • Focus on results collection for trending, comparison, and business reporting needs across large amounts of results data • By contrast, WAC is focused on engineering views of assessment results for immediate analysis and triaging purposes

27. Windows Performance Toolkit (WPT)

28. Benefits of WPT • Enables follow up on issues raised via ADK performance assessments • Enables holistic performance analysis • All processes/threads • All key system resources • User and kernel mode • Provides a way to analyze what customer actually experiences • Catch the problem as it happens • Capture-Anywhere-Decode-Anywhere • Very low overhead versus alternatives • Region of interest profiling • You get both what and when

29. Windows Performance Recorder and Analyzer • Tools for: • Trace capture (WPR) • Trace analysis(WPA) • Based on ETW (event tracing for Windows) • Used within Microsoft to build Windows • Core code reused in-box • Documentation on MSDN • Part of ADK and SDK in Windows 8

30. High-level changes • Windows Performance Analyzerreplaces xperview • Windows Performance Recorder replaces xbootmgr • Also replaces xperf trace capture functionality • xperf command line actions remain in WPT

31. xperf in Windows 7

32. Benefits of WPA in Windows 8 • Broaden the target audience to mainstream developers • Leverage familiar UI experiences • Enable quick ramp up • Reduce “noise” to focus on the “right” issues

33. WPA in Windows 8 Detailed Graphs Graph Explorer Graph Legend Grouping Columns Aggregating Columns Graphing Columns Table Blue Bar Gold Bar

34. Call to action • Evaluate the ADK assessment tools (file bugs, send feedback) until Beta • Work with your local contact to start planning your implementation • Integrate ADK assessments into your test processes to optimize user experience • Training and evangelization will be communicated • POC for adoption: adkcontact@microsoft.com • Requests and Timeline

35. Further reading and documentation Event Site: • http://channel9.msdn.com/Events Dev Center: • Windows Performance Analysis MSDN documentation: • WPA and WPR • Event Tracing for Windows • Assessment and Deployment Kit • Assessment Execution Engine Videos • Channel9 – BUILD ADK Videos Books: • Windows Internals by Mark Russinovich, David Solomon, and Alex Ionescu Reference: • ADK_TechnicalReference.chm in the ADK

36. Related sessions from //build conference Go to - www.buildwindows.com/sessions and search for these numbered sessions. • [HW-141T]Reducing the Memory Footprint of Drivers and Apps • [HW-147T] Building High Quality Windows PCs using the Assessment and Deployment Kit • [HW-148P] Scaling and Extending Windows Assessments to Improve System Quality • [HW-149P] Improving Battery Life and Energy Efficiency • [HW-459T] Delivering Great Performance and Battery Life • [HW-59T] Improving Performance with the Windows Performance Toolkit • [HW-915P] Introduction to Assessments • [HW-916P] Introduction to the Boot Performance Assessment • [HW-917P] Introduction to the IE Launch Assessment • [HW-920P] Introduction to the Streaming Assessment

37. Related sessions from //build conference (cont.) Go to - www.buildwindows.com/sessions and search for these numbered sessions. • [HW-921P]Introduction to ADK Custom Jobs • [HW-922P]Capturing and Analyzing Performance Traces • [HW-925P]Customizing WPA Trace Views • [HW-926P]Introduction to the New WPA User Interface • [HW-927P] Understanding Pool Usage using WPA • [HW-928P] Understanding Heap Data using WPA • [HW-975P] Customizing ADK Assessment Reports • [HW-976P] Packaging Assessments for Use on a Second Computer • [HW-977P] Understanding VirtualAlloc Usage using WPA

38. Thank You! For questions, please visit me in the Speakers Connection area following this session.

39. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.