1 / 57

The Case of the Unexplained…

The Case of the Unexplained…. Mark Russinovich Technical Fellow Windows Azure. WCA-B306. Outline. Introduction Sluggish Performance Error Messages Application Crashes Blue Screens. Case of the Unexplained…. This is the 2013 version of the “case of the unexplained” talk series

chase
Download Presentation

The Case of the Unexplained…

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Case of the Unexplained… Mark Russinovich Technical Fellow Windows Azure WCA-B306

  2. Outline Introduction Sluggish Performance Error Messages Application Crashes Blue Screens

  3. Case of the Unexplained… • This is the 2013 version of the “case of the unexplained” talk series • Previous versions covered different cases • Can view webcast on Sysinternals->Mark’s webcasts • Based on real case studies • Some of these have been written up on my blog

  4. Troubleshooting • Most applications do a poor job of reporting unexpected errors • Locked, missing or corrupt files • Missing or corrupt registry data • Permissions problems • Errors manifest in several different ways • Misleading error messages • Crashes or hangs

  5. Purpose of Talk • Show you how to solve these classes of problems by peering beneath the surface • Interpreting process, file and registry activity • Interpreting call stacks • You’ll learn tools and techniques to help you solve seemingly unsolvable problems

  6. Tools We’ll Use • Sysinternals: www.microsoft.com/technet/sysinternals (\\redmond\files\SYSINTERNALS\LBI\Latest) • Process Explorer – process/thread viewer • Process Monitor – file/registry/process/thread tracing • Procdump – process memory dumper • Autoruns – displays all autostart locations • SigCheck – shows file version information • PsExec – execute processes remotely or in the system account • TcpView – shows TCP/IP endpoints • Strings – dumps printable strings in any file • Zoomit – presentation tool I’m using • Microsoft downloads: • Debugging Tools for Windows: Windbg application and kernel debugger: www.microsoft.com/whdc/devtools/debugging (//dbg)

  7. The Sysinternals Administrator’s Reference • The official guide to the Sysinternals tools • Covers every tool, every feature, with tips • Written by markruss and aaronmar • Full chapters on the major tools: • Process Explorer • Process Monitor • Autoruns • Other chapters by tool group • Security, process, AD, desktop, …

  8. Outline Sluggish Performance Error Messages Application Crashes Blue Screens

  9. The Case of the Sluggish Explorer • Explorer was very slow to display the contents of local files • Captured a Process Monitor trace

  10. Process Monitor • Process Monitor is a real-time file, registry, process and thread monitor • When in doubt, run Process Monitor! • It will often show you the cause for error messages • It many times tells you what is causing sluggish performance

  11. Process Monitor Enhancements: Bookmarks • Bookmarking enables you to save markers in the trace: • Use F6 to find the next one, Shift+F6 to search up

  12. The Case of the Sluggish Explorer (Cont) • Noticed numerous references to network path with NAME NOT FOUND:

  13. The Case of the Sluggish Explorer (Cont) • Checked account mapping and saw that home drive was network share: • Confirmed in Process Explorer current directory was the network share:

  14. The Case of the Sluggish Explorer: Solved • Opened command prompt, killed Explorer, and restarted from local directory: no delays • Contacted IT and had them change home directory: problem solved

  15. Process Explorer • Process Explorer is a Task Manager replacement • You can literally replace Task Manager with Options->Replace Task Manager • Hide-when-minimized to always have it handy • Hover the mouse to see a tooltip showing the process consuming the most CPU • Open System Information graph to see CPU usage history • Graphs are time stamped with hover showing biggest consumer at point in time • Also includes other activity such as I/O, kernel memory limits

  16. Process Explorer v15: GPU Monitoring and Windows 8 • Captures GPU utilization and memory usage • System-wide • Per-Process

  17. Process Explorer v15.2 • Process timelines • Autostart locations

  18. Process Explorer v15.3 • Process view heat map columns • Bit-matching dump files • Disk counters fixed for Windows 8

  19. The Case of the Spiking ICQ Process • System was sluggish • Opened Process Explorer and saw Instant messaging application consuming CPU:

  20. Processes and Threads • A process represents an instance of a running program • Address space • Resources (e.g., open handles) • Security profile (token) • A thread is an execution context within a process • Unit of scheduling (threads run, processes don’t run) • All threads in a process share the same per-process address space • The System process is the default home for kernel mode system threads • Functions in OS and some drivers that need to run as real threads • E.g., need to run concurrently with other system activity, wait on timers, perform background “housekeeping” work • Other host processes: svchost, Iexplore, mmc, dllhost

  21. Viewing Threads • Task Manager doesn’t show thread details within a process • Process Explorer does on “Threads” tab • Displays thread details such as ID, CPU usage, start time, state, priority • Start address is where the thread began running (not where it is now) • Click Module to get details on module containing thread start address

  22. Thread Start Functions and Symbol Information • Process Explorer can map the addresses within a module to the names of functions • This can help identify which component within a process is responsible for CPU usage • Configure Process Explorer’s symbol engine: • Download the latest Debugging Tools for Windows from Microsoft (free) • Use dbghelp.dll from the Debugging Tools • Point at the Microsoft public symbol server (or internal symbol server if you have access)

  23. The Case of the Spiking ICQ Process (Cont) • Looked at threads tab • Start address of spiking threads were generic:

  24. Viewing Call Stacks • Click Stack on the Threads tab to view a thread’s call stack • Note that start address on Threads tab is different than first function shown in stack • This is because all threads created by Windows programs start in a library function in Kernel32.dll which calls the programmed start address

  25. The Case of the Spiking ICQ Process (Cont) • Looked at thread stack and saw video DLL:

  26. The Case of the Spiking ICQ Process (Cont) • Opened DLL view and saw that each time issue occurred, four video DLLs were loaded: • Noted that DLLs were not loaded on program startup • Opened Add/Remove programs, but saw no software from Viscom

  27. Autoruns • Shows every place in the system that can be configured to run something at boot & logon • Standard Run keys and Startup folders • Shell, userinit • Services and drivers • Tasks • Winlogonnotifica`tions • Explorer and IE addins (toolbars, Browser Helper Objects, …) • More and ever growing… • Each startup category has its own tab and all items display on the Everything tab • Startup name, image description, company and path

  28. Identifying Malware Autostarts • Zoom-in on add-ons (including malware) by selecting these filter options: • Verify Code Signatures • Hide Microsoft Entries • Select an item to see more in the lower window • Online search unknown images • Double-click on an item to look at where its configured in the Registry or file system • Has other features: • Can also show empty locations (informational only) • Includes compare functionality • Includes equivalent command-line version, Autorunsc.exe

  29. The Case of the Spiking ICQ Process: Solved • Opened Autoruns and disabled all Viscom DLLs: • Problem solved

  30. The Case of the Company 15-Minute Logons • Logons were taking 5-15 minutes across a company for one OS image • The other OS image had normal logon time • Execs started to complain • Admins tried everything: • Examined event viewer • Captured Wireshark traces • Checked for malware • Finally, another admin that had seen “Case of the Unexplained” decided to help: • Captured a Process Monitor boot trace

  31. The Case of the Company 15-Minute Logons (Cont) • Looked through process tree and noticed Framepkg taking four minutes: • Command-line showed it was McAfee A/V reinstallation

  32. The Case of the Company 15-Minute Logons: Solved • Question was: why was script reinstalling A/V? • Contacted server admins: • They discovered that Active Directory OU for OS image had GPO • GPO should have been removed from domain after Win7 rollout • Removed GPO: problem solved

  33. Outline Sluggish Performance Error Messages Application Crashes Blue Screens

  34. The Case of the File in Use Error • Tried deleting PowerPoint deck I had been editing, but got an error: • No sign of PowerPoint running,not even in Task Manager’s applications tab

  35. The Case of the File in Use Error: Solved • Did a Handle Search in Process Explorer • Found hidden PowerPoint process: • Terminated process: problem solved

  36. The Case of the Mistaken Windows-to-Go • User tried to use the Windows 8 AppStore, but got an error message: • Had installed Windows on a Maxtor SATA 2 disk, but for some reason Windows thought it was a portable USB disk

  37. The Case of the Mistaken Windows-to-Go: Solved • Captured a Process Monitor trace when trying to access the store and scanned the log • Found this reference: • Deleted the PortableOperatingSystem value: problem solved

  38. The Case of the Default Win8 RDP Lockscreen Background • Set Windows 8 lockscreento custom background • When RDP’ing into the system, default lockscreen showed • Needed to find out where default was coming from • Captured a Process monitor trace of locking system and RDP’ing into system

  39. The Case of the Default Win8 RDP Lockscreen Background (Cont) • Set a filter to “Path Begins With .jpg”: Lock RDP

  40. The Case of the Default Win8 RDP Lockscreen Background: Solved • Had to replace system version of lockscreen background • Launched command prompt as system using PsExec: • Copied background over RDP lockscreen file: • Problem solved

  41. Outline Sluggish Performance Error Messages Application Crashes Blue Screens

  42. The Case of the Crashing IE Favorites • IE was crashing every time Favorites was clicked • Ran IE8 without Add-ons • Still Crashed • Upgraded to IE9 • Still Crashed • Ran IE9 without Add-ons • Still Crashed

  43. The Case of the Crashing IE Favorites (Cont) • Used Process Explorer to get the PID of iexplore.exe:

  44. The Case of the Crashing IE Favorites (Cont) • Used ProcDump to capture a unhandled exception (-e) mini dump

  45. The Case of the Crashing IE Favorites (Cont) • Changed to the Exception Context Record (.ecxr) and looked at the stack (k) – looked like it was sorting

  46. The Case of the Crashing IE Favorites (Cont) • Suspected one favorite that had bullets: • Moved all of the Favorites to another folder • Still Crashed

  47. The Case of the Crashing IE Favorites (Cont) • Captured a Process Monitor trace • Set a filter for IE, ‘Action’ is SUCCESS and ‘Path’ contains ‘Favorites’, and Registry

  48. The Case of the Crashing IE Favorites (cont.) • Saw Order REG_BINARY value read from HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MenuOrder\Favorites:

  49. The Case of the Crashing IE Favorites: Solved • Used “Jump to” context menu to open Regedit • Renamed ‘Order’ value to ‘xOrder’ • Restarted IE: problem solved • Moved the favorites back • Looked at RegEdit, new ‘Order’ value made

  50. Outline Application Hangs Sluggish Performance Error Messages Blue Screens

More Related