Introducing AppSight 5 Application Support System with Black Box Software Technology

1. Introducing AppSight 5Application Support Systemwith Black Box Software Technology Shlomo Wygodny VP Product Engineering

2. AppSight 5 – Highlights • Full support of .NET applications • Code level recording of managed (and unmanaged) code • .NET in Application Performance and application configuration • Broader and deeper problem resolution • Analysis of application memory problems (DNA) • Capture and comparison of IIS and Internet Explorer configuration • File attachments to Black Box logs • Recording of Active Directory calls • Better usability • Unified system/code Black Box • Automatic tuning of recording profile in run-time • 1st level support module • New AppSight Manager console

3. .NET – system level recording • Recording of ASP.NET Execution • Processing ASPX (instead of Start/Stop ASP processing) • Filter log by ASPX • Recording of XML Web Services • Web Service method calls and responses (Client and Server side) • Protocol operations: Get URI/URL, create request, set Soap message. • Follow Transaction between Client and Server.

4. AppSight 4.5.NET - Remoting Client side Server side • Recording of Remoting • Remote method calls and returns. • Remote object creation. • Create and register channel, connect to socket, etc. • Follow Transaction between client and server

5. .NET – system level recording (cont.) • Recording of ADO.NET (database) Calls • Database connections • Queries (Create/Set/Run) • Datasets • Providers (OLE-DB, ODBC, SQL, ODP) • Recording of COM Calls (Interop) • Runtime Callable Wrapper (RCW) • .NET calls C COM object • COM Callable Wrapper (CCW) • .NET COM objects

6. .NET – system level recording (cont.) • .NET exceptions • Handled / unhandled exceptions • System / User-defined exceptions • Recording access to Config files • Recording of Assemblies • Compile File or Unit • Load/unload Assembly

7. Recording of Internal Logging • Records .NET Trace class functions (used for internal logging) • Used for understanding application context ticker = Trim(cmbticker.Text) Trace.WriteLine("Ticker value is '" & ticker & "'")

8. Web Services performance Analysis • Presents performance data of Web services calls

9. Application Configuration • Records access to .NET application configuration files • Also viewed and compared in the Application Configuration dialog

10. Sample Recording

11. .NET system level Demo

12. .NET Code level recording • Record: • Clients • WEB applications (ASPX – “Code behind”) • WEB Services (ASCX), etc.

13. .NET Code level • Record: • Function calls and exit • Source lines • This, Parameters & return value • Exception • Stack • Record combined Managed and Unmanaged code • (C#/VB.NET and C/C++/VB) • Work with or without PDB’s • PDB required for source location • To record lines in functions, the application needs to be recorded from start-up • For .NET only!

14. .NET Code Level Demo

15. Capture and Comparison of IIS and IE Configuration • Captures IIS and Internet Explorer settings • Compares configuration to simplify root cause identification

16. Black Box Log Attachments • Files attached to Black Box logs and opened from AppSight/System • Defined in the recording profile • Absolute or relative path • “C:\Windows\MyApp\MyFile.txt” or “%WINDIR%\MyApp\MyFile.txt” Files specified in recording profile dialog in AppSight/System Files opened from AppSight/System Log Attachments dialog

17. Unified Black Box • One Black Box executable records at user, system and code level • One recording profile (.RPR) includes all recording settings • System-level settings selected in AppSight/System, code-level in AppSight/Code • One Black Box log includes full recorded data • Viewed in AppSight/System and AppSight/Code

18. Unified Black Box (cont.) • Code and System recording now uses the same AppSight repository • Code level Black Box now supports: • Remote recording • Remote install & Run • Recorded sessions management • User code-level functions can be displayed in AppSight/System • Create combined RPR: • Start in AppSight/System and move to /Code • Or vice versa (using BBX/Merge RPR menu item)

19. Unified BBX in AppSight/Code • Opening RPR with system elements in AppSight/Code: • New Recording – equivalent to Restart BBX • New Connect To… dialog to select a session

20. .NET Code Level Demo

21. Auto-Tuning • User-defined maximum penalty (e.g. 10%, 200%) • Functions are removed from recording automatically in run-time to ensure desired performance impact level • Functions can be marked as non-removable.

22. .NET Code Level Demo

23. Saving Logs • Adding comments to log • Saving filters along Black Box logs • Saving just the filtered part of the log • [Demo]

24. Specifying RPR/ASL to BBX • Command line Interface for the Black Box: BlackBox /RPR “c:\...\X.rpr” /ASL “…” • Registry keys for the Black Box: [HKEY_LOCAL_MACHINE\SOFTWARE\Identify\AppSight 5.0 Black Box Service\Common]"Current Log Path"="...""Current Rpr Path"="...““Last Connected Server”=”…”"WorkWithServer"=dword:00000001 • Service BBX can now run in Standalone mode • When ASL is specified, the log is saved upon exiting the BBX. • E.g. by Net Stop

25. Licensing • The license file includes the name of the site/group to which it is licensed. • Appears in flash screen and Help/About • Feature: ApsSite • Previous licenses will not work for version 5.0 • AppSight 5.0 license will not support 4.51 • No Black Box licensing • To be added in a post-release version

26. AppSight Level 1 Viewer • Targeted at level 1 support staff • Extremely easy to use • Plays back the visual part of a Black Box log (user movie) • Displays system and browser configuration data

27. Other Enhancements • New GUI for defining recording profiles • Selective recording of surrogate processes (DLLHOST/W3WP) • COM object list refresh • Alerts on COM/COM+ method calls • Alert to call a function from DLL • Used to dump open handles, dump memory, etc. • Show all network packet. …And many other fixes

28. Match EXE/PDB improvements • Search path for PDB and modules: • “Locate PDB option

29. Troubleshooting AppSight 5.0 • Introducing a new mechanism: AppSight Log • Allows tracing of the recording mechanism (recorded application). • Execution tracing, not just stack dump. • ASD/PTB files created with each run • Under the IdentifyTrace subdirectory • Filenames like 3860_01_Jan_04_11_07_42.asd • Deleted automatically. • To use: Run once AslLogConverter.exe from: C:\Program Files\Identify\AppSight5.0\Bin • Whenever a crash occurs, dbl-click ASD to create ASL • ASL is password protected: sd • Attach ASL to bug report.

30. Application Memory (MC) Capture and Analysis • Detects memory allocations which were not freed • Synchronized with the detailed execution pane • Enables drill down to code-level

31. .NET Code Level Demo

32. Memory Analysis is a known market Rational Purify Compuware BoundsChecker dmalloc (freeware) Stratos MemCheck ZeroFault Parasoft Insure MicroQuill HeapAgent mpatrol Memory Validator

33. MC - Our Advantage Not a development tool – used for deployed apps • Top level view pointing to the problematic module • Does not require PDB’s, debugger etc. • Code level view – show allocation line • BBX/console architecture allows catching problems on a remote location.

34. MC – What we do not do • Memory management tools for developers do other things which are off AppSight target: • Read / Write access through illegal pointers • Overwrite / underwrite of heap/stack/static objects • Read / Writes into freed heap objects • Good analysis is very slow • AppSight can help if it causes immediate crash • Reads of uninitialized memory • Free of unallocatred memory • Double frees • Improved memory management

35. MC – target platforms • Used for C/C++ applications – clients or WEB • Not .NET • Not VB • Almost

36. MByte Time Memory Consumption in Details • Sample target: server application • After some initialization memory usage should be flat over time: • Whatever may cause constant increase over time in memory usage is problematic. • This may be either memory leak or regular memory consumption.

37. Memory Consumption and Memory Leak • Memory usage in computer program involves: • Allocation (e.g. using New, malloc etc.) – returns address • Using the allocated memory to store data • Deallocation (e.g. using destructors, free, etc.) – of a given address. • A potential problem is every memory piece which is: • Allocated after initialization; • Not deallocated after a short while (e.g. transaction time). • Every such Memory Consumption (MC) case is counted for the AppSight graph. • The time to deallocate is a parameter to the memory consumption system. • Memory Leak is a subset of MC case in which it is not possible to deallocate the memory • Because its address was lost. • Memory leak is almost always a bug.

38. MC - example • In T1 no allocations will be reported • Because they are “new”. • In T2 only A2 will be reported • Because A1 was freed • In T3 A2 will be reported again • Because it wasn’t freed yet Allocation A1 Allocation A2 T0 T1 T2 T3 Deallocation A1

39. (1) is used for the graph (2) is used for escalation to AppSight Code Memory consumption (cont.) • Every 3 seconds the following information is sent: • MC per DLL • Top 5 MC’s • Or (if leaks were detected) Leaked memory

40. MC Graph MC Stacks Memory Consumption Graph • MC graphs are displayed per DLL: • Whenever a leak is detected, the respective graph segment is black: • For the graph we need to know: Which DLL allocated the memory? • E.g. in the case: • F1 (D1.DLL) ->f2 (D2.DLL) -> Malloc (Sys1.DLL) -> RTLheapAlloc (Sys2.DLL)

41. MC Graph MC Stacks MC - Finding the culprit DLL Sample case: main (D1.EXE) ->f3 (D2.DLL) -> Malloc (Sys1.DLL) -> RTLheapAlloc (Sys2.DLL) < Prev alloc main (D1.EXE) ->f2 (D2.DLL) -> Malloc (Sys1.DLL) -> RTLheapAlloc (Sys2.DLL) < current alloc • The recommended analysis method is XXXXX

42. MC Graph MC Stacks MC - What is loop allocation Main () => F1 () => F2 (){ for (i=0; i<100; i++) { item.next = create_item(data); item = item.next; } => create_item(data) { return new item(data); } • The stack is identical from main till F2 • The DLL of f2 will be considered the one which made the allocation.

43. MC Graph MC Stacks MC – Collecting stacks • Every Memory Consumption (MC) is collected with its stack. • MC is collected per stack: • For example: • F1 -> F2 -> F3 -> allocation (10 bytes) • F1 -> F3 -> allocation (15 bytes) • F1 -> F2 -> F3 -> allocation (20 bytes) • The algorithm collects: • F1 -> F2 -> F3 -> allocation (30 bytes) • F1 -> F3 -> allocation (15 bytes)

44. MC Graph MC Stacks MC – Sending stacks • If leaks were detected, their stack is sent. • Up to 5 leaks per sending. • The corresponding graph segment is marked in black. • If no leaks were detected, the stack of the largest 5 Memory Consumptions are sent. • Note the same stack can be sent a few times. • Stacks are only used while escalating to AppSight/Code.

45. MC - Known deficiencies • In order to find leaks, it is needed to search for the black segments. • In 5.0, no way to get a list of all leaks. • Leak detection “jumps” CPU usage every 3 seconds • In 5.0, no way to disable it in MC is on. • MC info always transferred continuously • The only way to clear MC info is to restart BBX • Note MC info may be irrelevant after stop/start recording