Pushing the Limits of Windows

1. SESSION CODE: WCL402 Pushing the Limits of Windows Mark Russinovich Technical Fellow Microsoft Corporation *Portions derived from David Solomon’s Windows Internals Seminar

2. About Me • Technical Fellow, Microsoft • Co-founder and Chief Software Architect of Winternals Software • Co-author of Windows Internals 4th and 5th Edition and Inside Windows 2000 3rd Edition with David Solomon • Author of TechNet Sysinternals • Home of blog and forums • Contributing Editor TechNet Magazine, Windows IT Pro Magazine • Ph.D. in Computer Engineering

3. Pushing the Limits • Resource exhaustion is obviously not a good thing • At the minimum it causes service outage • At worst it can cause data loss or even corruption • As a Windows systems administrator or support engineer you need to know the limits • Capacity planning • Systems monitoring • Troubleshooting

4. What You’ll Learn • This session explains core Windows kernel limits, where they come from, and how to monitor them • You’ll learn: • Find the cause of a handle leak • Standard terms for memory values • How to spot a process memory leak • How to size the page file • How Windows uses physical memory • Determine if you have enough physical memory • I’ll use SysinternalsTestlimitto stress resources:http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

5. Outline • Handles • Virtual Memory • Physical Memory

6. Handles and Objects • Windows defines objects to represent operating system resources • There are 42 in Windows 7 • Use SysinternalsWinobj to view defined objects • When a process wants to interactwith a resource, it opens it • That creates a handle in the process’ handle table • Handle entry points at the object and records granted access • Subsequent operations reference the handle Object Handle Table Entry Pointer Accesses

7. Handle Limits • A process handle table can grow to about 16 million handles • Handle tables are stored in paged pool, so that can also be a limiter • 32-bit Windows: 8 bytes per entry • 64-bit Windows: 16 bytes per entry

8. Tracking Handle Leaks • A handle leaker will show a growing handle count over time • Task Manager and Process Explorer show handle count columns • Process Explorer also shows it in process properties dialog • Process Explorer uses difference highlighting to show newly opened and closed handles

9. Tracking Handle Call Stacks • Windows can capture call stacks at the time of handle open and close • Use Application Verifier to track from process start • Free download from Microsoft • Use Windbg to enable and disable dynamically • !htrace –enable, !htrace –disable

10. Viewing Handle Call Stacks • Use Windbg to view stacks • Use !htrace (no parameters) to show all recorded stacks • Use !htrace –snapshot and !htrace –diff to see differences

11. Outline • Handles • Virtual Memory • Physical Memory

12. Address Space Limits • One virtual memory limit is address space • Process can run out of space to allocate virtual memory • Typically only an issue for 32-bit processes • Another is the system commit limit

13. 32-bit x86 Address Space • 32-bits = 2^32 = 4 GB 4GT Address Space (boot.ini: /3GB or /USERVA) (BCD: increasuserva) Default 2 GB User process space 3 GB User process space 2 GB System Space 1 GB System Space

14. 64-bit Address Spaces • 64-bits = 2^64 = 17,179,869,184 GB • 32-bit process on 64-bit Windows that’s Large Address Space Aware = 4 GB x64 (AMD64& Intel 64) IA-64 8192 GB (8 TB) User process space 7152 GB (7 TB) User process space ~8192 GB (8 TB) System Space ~8192 GB (8 TB) System Space

15. Virtual Memory Types • Process Committed: contents are backed by a file on disk (data file, image or paging file) • Address space breakdown • Committed: • Shareable (e.g. EXE, DLL, shared memory, other memory mapped files) • Private (e.g. process heap) • Uncommitted: • Reserved (not yet committed) • Free (not yet defined) • Performance counters available: • Private Bytes – private virtual memory • Virtual Bytes – total of shareable+private+reserved • No separate counters for Shareable or Reserved or Free

16. Viewing Address Space Breakdown • Task Manager only lets you see private bytes • Before Vista: column called “VM Size” • Vista and later: column called “Commit Size” • Process Explorer shows both virtual size and private bytes • Experiment: • Add 2 columns to process list: • Virtual Size • Private Bytes • Run Testlimit twice • Testlimit -r • Testlimit -m • Note: if on 64-bit Windows, Testlimit can grow to 4GB

17. Tracking Process Commit Usage • Most virtual memory problems are due to a process leaking private committed memory • Heap, GC heap, language heaps (CRT) • Also fragmentation can be an issue • Address space can effectively be exhausted prematurely • Basic performance counters don’t provide enough information to troubleshoot

18. Viewing Processes with VMMap • VMMap shows detailed breakdown of process address space: • Private process memory • Private (VirtualAlloc) • Heap and GC Heap • Stack • Copy-on-write • Shareable process memory • Shareable – shareable memory • Mapped File – memory mapped files and images • Page table – page table pages • Unknown – page table pages not in working set, copy-on-write pages, VAD charges • Fragmentation is visible by selecting Options->Show Free Regions, selecting the Free type, and sorting by size

19. VMMap Differencing • Press F5 to refresh the view • VMMap keeps all snapshots • Use the timeline to select snapshots to compare

20. The System Commit Limit • Another virtual memory limit is total virtual memory: system commit limit • System committed is virtual memory that must be backed either by physical memory or stored in the paging file • Process private memory • Pagefile-backed shared memory • System Commit Limit: • Sum of (most of) physical memory and paging files RAM C:\Pagefile.sys D:\Pagefile.sys System Commit Limit

21. Viewing the System Commit Limit • Process Explorer tracks the current commit limit in the System Information window: • Prior to Vista, Task Manager shows it as “PF Usage”: • Task Manager on Vista and higher doesn’t show a graph: Win7 Vista

22. Exhausting the System Commit Limit • 64-bit Testlimit64 -m (committed memory) will exhaust the system commit limit before its address space: • You can increase the System commit limit by adding RAM or increasing the pagefilesize

23. Sizing the Paging File • Should be based on workload usage of committed virtual memory • Many recommendations use a formula based on RAM (1.5x, 2x, etc.) • Actually, the more RAM, the smaller the paging file needed • So how does the system size the paging file with the default System Managed? • Minimum: 1.5x RAM if RAM < 1 GB; RAM otherwise • Maximum: 3x RAM or 4 GB, whichever is larger

24. Determining a Workload’s System Commit Requirement • Look at peak system commitafter workload has run • Pre-Vista: Task Manager • Vista+: Process Explorer • Apply a formula to that to give buffer (1.5x or 2x) • Make sure it’s big enough to hold a crash dump

25. Outline • Handles • Virtual Memory • Physical Memory

26. Windows Physical Memory Limits • The amount of physical memory Windows will use depends on: • SKU licensing • 32-bit vs 64-bit • Tested systems • Kernel address space • See “Memory Limits for Windows Releases” for a complete list: • http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx

27. Physical Memory on 64-bit Systems • Kernel address space limits it to 2 PB (Petabytes) • x64 page table structures limit physical memory to 48-bits physical addressing (256 TB)

28. 64-Bit Server • Current Windows Server limits: • Windows Server 2003: 16 GB (Standard) – 2 TB (Datacenter) • Windows Server 2008: 32 GB (Standard) – 2 TB (Datacenter) • Windows Server 2008R2: 8 GB (Foundation) – 2 TB (Datacenter) • For Datacenter Edition, the limit is based on SKU and hardware testing • We won’t support what we can’t test • Limits have gone up even for a particular release of Windows because of access to larger machines • Datacenter Edition maxes out all licensing, including memory

29. 2 Terabytes • The Windows architecture can support much more, but as of now, the largest system we’ve tested is 2 TB (Itanium):

30. 32-Bit Server • Windows 32-bit Server limits: • Windows Server 2003: 2 GB (Web) – 128 GB (Datacenter) • Windows Server 2008: 4 GB (Standard) – 64 GB (Datacenter) • Windows Server 2008 R2: n/a • The 32-bit Server limit is based on the kernel address space • PFN entry is 28-bytes so 128 GB requires 890 MB • Limits other system code and data • Also why limit is lower when booted 4GT (/3GB) User Address Space System PFN Database

31. Client Systems • 64-bit client limits solely based on licensing: • 512 MB in Windows XP Starter • 128 GB in Windows Vista Ultimate • 192 GB in Windows 7 Ultimate • 32-bit client limit is 4 GB • Limited by potential driver incompatibility • And usable amount might be less than 4 GB…

32. Drivers and Physical Memory • To be safe, client limits physical memory addresses to 4 GB • Some client device drivers still truncate addresses > 4 GB • Client systems with > 4 GB are recent • Servers are safe with > 4 GB • Have had more than 4 GB for longer • Devices are more mainstream • Policy allows only WHQL-signed drivers 0 0x00032000 0.2 GB DMADestination Device Data 4 GB 4.2 GB Target 0x00000001 0x00032000 6 GB

33. Device Memory 0 • Device memory can push physical memory above 4 GB • Firmware maps all device memory below 4GB • For compatibility with older operating systems • Biggest device memory consumer is video • Result is that some physical memory may lie above 4 GB • Watch out for Shared Video Memory (UMA) Video RAM 4 GB Inaccessible RAM

34. Working Sets • Working set: all the physical pages “owned” by a process • Pre-Vista: Task Manager’s “Mem Usage” • Vista+: Task Manager’s “Memory – Working Set” • Process Explorer’s “Working Set” • System has its own working set (broken into multiple in Windows 7) Notepad Word Explorer System Available

35. Working Set Lists • A process always starts with just enough working set to describe the address space • It then incurs page faults when referencing a page that isn’t in its working set • Many page faults may be resolved from memory (from other working sets, the standby or Modified page lists) • The Memory Manager can take away pages and reassign them (trimming) • Note: the amount of physical memory assigned to a process is invisible to the process • It simply affects the performance of the process newer pages older pages

36. Working Set Breakdown • Consists of 2 types of pages: • Shareable (of which some may be shared) • Private • Four performance counters available: • Working Set Shareable • Working Set Shared (subset of shareable that are currently shared) • Working Set Private • Working Set Size (total of WS Shareable+Private) • Note: adding this up for each process overcounts shared pages • Caveats • Working set does not include trimmed memory that is still cached • Working set shareable if it’s not shared really should be viewed as “private”

37. Viewing Working Set with Process Explorer • Process Explorer shows all the performance counters • Virtual Bytes • Private Bytes • WS Shareable Bytes • WS Shared Bytes • WS Private Bytes • Run Testlimit: • Testlimit –r 100 • Testlimit –m 100 • Testlimit –d 100 • Note how working set numbers don’t at all represent the process virtual memory usage

38. Viewing the Working Set with VMMap • Vmmap shows working set size of each component of address space

39. Physical Memory Lists • System keeps track of pages with lists: • Working sets (process and system) • Standby page list: file cache) • Zero page list: free memory that has been zeroed • Free page list: paged that have been freed • Modified page list: pages that must be saved before being repurposed • Lists are implemented by entries in the “PFN database” • Each page of physical memory has an entry in the database • The Memory Manager watches paging activity to determine how to assign memory

40. Available Memory • Available memory is the physical memory that can be repurposed without a disk I/O • Page fault without disk I/O: soft fault • Page fault with disk I/O: hard fault • Three memory lists add up to available memory: • Standby List: cached file pages • Free Page List: pages that have been explicitly freed or freed at process termination • Zero Page List: free memory that has been zeroed

41. Viewing the Paging Lists with Task Manager • XP/2003: • Available = Standby + Zero + Free • System Cache = Standby + Modified + System Working Set • Vista/Server 2008: • Replaced Available with Free • Free + Zero list • System Cache relabeled Cached • Windows 7/Server 2008 R2 • Available put back

42. Viewing Memory Lists • In Windows 7, the Resource Monitor shows the sizes of the memory lists: • Process Explorer also shows them

43. Viewing Memory Usage with Rammap • In addition to showing size of paging lists, shows usage breakdown: • Process private • Mapped file • Shared memory • Page tables • Paged pool • Nonpaged pool • System PTE • Session private • Metafile • AWE • Driver locked • Kernel stack

44. Do You Have Enough Memory? • There’s no sure-fire rule or counter to tell if you if you have enough memory • The general rule: available memory remains generally low • Use Perfmon to monitor available memory over time • Use Process Explorer, or on Vista and later Task Manager, to monitor physical memory usage • Use Process Explorer or Task Manager to see instantaneous value • Watch in Process Monitor for excessive reads from paging file

45. Other Limits Covered in My Blog • Paged and nonpaged pool: system virtual memory buffers • Processes and threads • GDI and USER objects: window manager objects

46. Conclusion • Knowing system limits can help you avoid resource exhaustion • By knowing how to track resource usage you can identify potential problems and root cause them • Come to my last session:CLI315 Case of the Unexplained... Windows Troubleshooting • Today at 5:00

47. What is the Springboard Series? The Springboard Series IT pro experience offers dynamic content and structured guidance across the adoption lifecycle • Inside of Microsoft we are • A turnkey IT pro engagement platform for depth and breadth • The program to mobilize MS marketing and field to focus on desktop OS IT pros • To the IT pro, our goal is • Be the definitive resource for Desktop IT pros • Open, honest; show don’t tell • Information at right time, right level across Adoption Lifecycle DISCOVER EXPLORE PILOT DEPLOY MANAGE How does it change my work? How do I maintain and optimize? Is it worth the pain? Is our environment ready? Is the organization ready? Weekly, Monthly and Quarterly Rhythm of Topical Content Springboard Technical Experts Panel Event Support and Resources Straight-talk Monthly Feature Articles and Overview Guides one-Windows TechCenter in 10 languages TalkingAboutWindows Video Blogs Virtual Roundtable Events Visit the Springboard Series on TechNet at www.microsoft.com/springboard

48. Required Slide Resources Learning • Sessions On-Demand & Community • Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers • http://microsoft.com/technet • http://microsoft.com/msdn

49. Required Slide Complete an evaluation on CommNet and enter to win!

50. Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st http://northamerica.msteched.com/registration You can also register at the North America 2011 kiosk located at registrationJoin us in Atlanta next year