Windows Azure Diagnostics Logging and Monitoring in the Cloud

PDC09-SVC15 Windows Azure DiagnosticsLogging and Monitoring in the Cloud Matthew Kerner Program Manager, Windows Azure

Diagnostics: Single Server vs. the Cloud Single Server Cloud Dynamic Environment Multi-instance, elastic capacity Distributed transactions Local Access Infeasible Many nodes Distributed, scaled-out data Service Upgrades • Static Environment • Single well-known instance • Traceable local transactions • Local Access Feasible • All in one TS session • Data & tools co-located • In-Place Changes

Windows Azure Diagnostics • SDK component providing distributed monitoring & data collection for cloud apps • Support Standard Diagnostics APIs • Cloud-Friendly • Manage multiple role instances centrally • Scalable • Built on Windows Azure Storage & used by scale-out Windows Azure platform components • Developer In Control • What to collect & when to collect it

Hello World Demo Windows Azure Diagnostics

Windows Azure Diagnostics Configuration Role Instance Role Data collection (traces, logs, crash dumps) Quota enforcement Diagnostic Monitor Local directory storage Windows Data Sources IIS Logs & Failed Request Logs Perf Counters Windows Event Logs

Windows Azure Diagnostics Request upload Role Instance Windows Azure Storage Role Diagnostic Monitor Local directory storage Windows Data Sources Scheduled or on-demand upload

Windows Azure Diagnostics Development Fabric Windows Azure Hosted Service

Windows Azure Diagnostics Development Fabric Windows Azure Hosted Service Diagnostic Manager Desktop Diag Application Controller Code Configure

Activate Windows Azure Diagnostics Generate Data Enable Local Buffering Transfer to Windows Azure Storage How-To

Sample: Activate WA Diagnostics • // This is done for you automatically by • // Windows Azure Tools for Visual Studio • // Add a reference to Microsoft.WindowsAzure.Diagnostics • usingMicrosoft.WindowsAzure.Diagnostics; • // Activate diagnostics in the role's OnStart() method • publicoverrideboolOnStart() • { • // Use the connection string contained in the • // application configuration setting named • // "DiagnosticsConnectionString” • // If the value of this setting is • // "UseDevelopmentStorage=true" then will use dev stg • DiagnosticMonitor.Start("DiagnosticsConnectionString"); • ... • }

Sample: Web.Config Changes • <!– • This is automatically inserted by VS.The listener routes • System.Diagnostics.Trace messages to • Windows Azure Diagnostics. • --> • <system.diagnostics> • <trace> • <listeners> • <addtype="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" name="AzureDiagnostics"> • <filtertype="" /> • </add> • </listeners> • </trace> • </system.diagnostics>

Sample: Generate Diagnostics Data • stringmyRoleInstanceName = • RoleEnvironment.CurrentRoleInstance.Id; • // Trace with standard .Net tracing APIs • System.Diagnostics.Trace.TraceInformation( • "Informational trace from " + myRoleInstanceName); • // Capture full crash dumps • CrashDumps.EnableCollection(true); • // Capture mini crash dumps • CrashDumps.EnableCollection(false);

Sample: Enable Local Data Buffering • // Managed traces, IIS logs, failed request logs, • // crashdumps and WA diags internal logs are buffered • // in local storage by default. Other data sources must be • // added explicitly • DiagnosticMonitorConfigurationdiagConfig = • DiagnosticMonitor.GetDefaultInitialConfiguration(); • // Add performance counter monitoring • PerformanceCounterConfigurationprocTimeConfig = new • PerformanceCounterConfiguration(); • // Run typeperf.exe /q to query for counter names • procTimeConfig.CounterSpecifier = • @"\Processor(*)\% Processor Time"; • procTimeConfig.SampleRate = System.TimeSpan.FromSeconds(1.0); • diagConfig.PerformanceCounters.DataSources.Add(procTimeConfig); • // Continued on next slide...

Sample: Enable Local Data Buffering • // Continued from previous slide... • // Add event collection from the Windows Event Log • // Syntax: <Channel>!<xpath query> • // http://msdn.microsoft.com/en-us/library/dd996910(VS.85).aspx • diagConfig.WindowsEventLog.DataSources.Add("System!*"); • // Restart diagnostics with this custom local buffering • // configuration • DiagnosticMonitor.Start( • "DiagnosticsConnectionString", • diagConfig);

Sample: Web.Config Changes •  • <system.webServer> • <tracing> • <traceFailedRequests> • <addpath="*"> • <traceAreas> • <addprovider="ASP"verbosity="Verbose" /> • <addprovider="ASPNET" • areas="Infrastructure,Module,Page,AppService" • verbosity="Verbose" /> • <addprovider="ISAPI Extension"verbosity="Verbose"/> • <addprovider="WWW Server"verbosity="Verbose"/> • </traceAreas> • <failureDefinitionsstatusCodes="200-599"/> • </add> • </traceFailedRequests> • </tracing> • </system.webServer>

Sample: Scheduled Data Transfer • // Start off with the default initial configuration • DiagnosticMonitorConfiguration dc = • DiagnosticMonitor.GetDefaultInitialConfiguration(); • dc.WindowsEventLog.DataSources.Add("Application!*"); • dc.WindowsEventLog.ScheduledTransferPeriod = • System.TimeSpan.FromMinutes(5.0); • DiagnosticMonitor.Start("DiagnosticsConnectionString", dc);

Sample: On-Demand Data Transfer • // On-Demand transfer of buffered files. • // This code can live in the role, or on the desktop, • // or even in another service. • varddm = newDeploymentDiagnosticManager( • storageAccount, • deploymentID); • varridm = ddm.GetRoleInstanceDiagnosticManager( • roleName, • roleInstanceName); • vardataBuffersToTransfer = DataBufferName.Logs; • OnDemandTransferOptionstransferOptions = • newOnDemandTransferOptions(); • transferOptions.From = DateTime.MinValue; • transferOptions.To = DateTime.UtcNow; • transferOptions.LogLevelFilter = LogLevel.Critical; • GuidrequestID = ridm.BeginOnDemandTransfer( • dataBuffersToTransfer, • transferOptions);

Storage Considerations • Standard WA Storage costs apply for transactions, storage & bandwidth • Data Retention • Local buffers are aged out by the Diagnostic Monitor according to configurable quotas • You control data retention for data in table/blob storage • Query Performance on Tabular Data • Partitioned by high-order bits of the tick count • Query by time is efficient • Filter by verbosity level at transfer time

Feature Summary • Local data buffering • Configurable trace, perf counter, Windows event log, IIS log & file buffering • Local buffer quota management • Query & modify config from the cloud or from the desktop per role instance • Transfer to WA Storage • Scheduled & on-demand • Filter by data type, verbosity & time range • Transfer completion notification • Query & modify from the cloud and from the desktop per role instance • Under the hood • Role runs in Performance Log Users group • Coming Soon: IIS Logs generated in role’s local data directory

Common Diagnostic Tasks • Performance measurement • Resource usage • Troubleshooting and debugging • Problem detection • Quality of Service Metrics • Capacity planning • Traffic analysis (users, views, peak times) • Billing • Auditing

Performance Measurement Resource UsageTroubleshooting Demo Windows Azure Diagnostics

http://diags.cloudapp.net Web Role Survey My Utility Classes PowerShell scripts Desktop Cloud Diagnostic Manager Diagnostic Monitor Diags DiagController.exe Survey Results Blob Controller Table My Utility Classes Windows Azure Storage

Q&A

Windows Azure Diagnostics • SDK component providing distributed monitoring & data collection for cloud apps • Standard diagnostics APIs • Cloud-Friendly and Scalable • Developer In Control • Available now in the WA SDK

Flip Camera/BlueTrack Mouse

YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation forms online at MicrosoftPDC.com

Learn More On Channel 9 • Expand your PDC experience through Channel 9 • Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses channel9.msdn.com/learn Built by Developers for Developers….

Windows Azure Diagnostics Logging and Monitoring in the Cloud