Performance Management on Linux : The HP OpenView way

1. Performance Management on Linux : The HP OpenView way Dr. R. Krishnan OpenView Performance HP, Bangalore

2. agenda • the challenge of performance management on Linux • the openview solution

3. response Time fixed parameters infrastructure component workload output variable parameters performance fundamentals • performance (response time) is a function of workload vs. available resources (capacity) characterized by: • fixed parameters: • disk architecture • variable parameters: • memory and cpu • application • partitioning trend

4.      Web Server (Connect) ’ ’ NAS Request NAS Process   Firewall LAN LAN LAN x’  ’’’ ’ x’ App Server (Process) response time DB Server LAN LAN 7 ’ 6 ’ ” 9 8 ”   Client Buffer T1 Line LAN ” understanding response time • multiplicity of workloads (application dependent) • multiplicity of processing capacities • global and component response times

5. some common performance problems hub database server users/ customers application servers legacy system hub web servers firewall routers load balancer routers ISP1 ISP2 End-user connection issues Internet Bottlenecks ISP Performance and Peering Load Balancer Problems Errors Faulty Web Server within a cluster Application Server Issues Devices Issues Server HW & OS Issues Content Issues General • insufficient memory • incompatible service packs and DLLs • excessive queuing requests • too many secure HTTPS connections Network • inadequate internet pipe • hidden bottlenecks between website and ISP (peering) • faulty hops (misdirected traffic and lost packets • mis-configured software and incompatible hardware Web Server • insufficient memory • poor web server design • high CPU usage App Server • poor cache management and high CPU usage • lack of memory • poor session management • poor database tuning DB Server • inefficient indexing • fragmentation • out-of-date statistics • faulty application design

6. the problem of IT silos I knew we shouldn’t have upgraded the database last weekend I hope that memory leak hasn’t reappeared So the application is crawling to its knees and you are all telling me everything’s fine?!! Can you bring up those reports that show that the network is ok.

7. one transaction users/customers web servers application servers database servers many infrastructure components disciplines required service management • commit to service levels • monitor compliance • communicate to business operational monitoring • service level objectives • customer experience • infrastructure elements performance problem resolution • bottleneck location • problem diagnosis resource optimization • utilization trends/baselines • hot spot analysis • forecasting and planning

8. processes required • Performance management is the art of resolving current performance bottlenecks. • A reactive process that tends to correct design flaws or infrastructure faults and failures. It requires a quasi-real-time analysis of problems. • Capacity management is the art of predicting future performance bottlenecks. • A proactive process that relies on past experience to forecast potential problems. It is founded on statistical analysis.

9. Service Manager Performance Planner Operator roles required Darn that problem is tough to reproduce. I wish I had put in better debugging When do we need to migrate to that T-1 based Frame Relay? Can our servers handle the Christmas rush? Application Developer Are we reaching the service levels we set for my customers? I don’t think the problem is in the network. It’s probably in the database. There are a lot of performance degradations today. I’ll have to escalate these to Bob. Performance Expert

10. The OpenView solution for Linux

11. internet Webportal Webportal DSL, T1,T3, ATM, Sonet infrastructure accesstier edge routers routing switches NOCconsole NOCconsole NOCconsole authentication, DNS,intrusion detect, VPNweb cache 1st level firewall network devices load balancingswitches webtier help deskasset mgmt.change mgmt. config mgmt. web servers trapd, syslog, logfiles web page storage(NAS) mgmtservers 2nd level firewall internet services integrations and gateways software agents and probes applicationtier switches applicationservers systems & servers mgmt data warehouses files(NAS) switches apps & DBs databasetier failoverservers databaseSQL servers historicalreporting storage storage areanetwork(SAN) OV architecture for a typical data center

12. hp OpenView service desk SLM Config Change service information portal events + inventory service model hp OpenView operations service navigator Reporting: (Perf, SLA, Availability, etc) DATA perf, event, config Event Correlation hp OpenView reporter NNM Perf Admin Ops data templates, events, config events configuration, templates, and action requests data events + config Polling, Threshold checks SNMP alarms hp OpenView internet services Server ops agent + SPIs Ops Perf Admin Perf. data Config Results O/S Services Applications OVIS probe Database Transaction tests Systems Network Elements OpenView Architecture • collect – messages sent from agents • process – central storage of events and config data • presentation of events for problem resolution and forward to trouble ticketing systems • act – operator or automated actions • SMART Plug-ins – application mgnt • service view • actively monitor common internet services and protocols through simulation • actively monitor custom web-based transactions e.g. shopping cart through recorder/playback • passively monitors popular web servers, web application servers, e-commerce servers, firewalls • collect - SNMP traps, application and system logfiles, customer variables, etc… • process - filter, prioritise, group and correlate events • act – buffer messages, forward messages or perform automatic actions • call management • incident management • problem management • change management • work management • service level management • complete and live view of network • proactive problem isolation • event correlation (connector down, scheduled maintenance, repeated events and pairwise correlation) • clear reporting • manage global networks

13. where performance management fits adaptive management performance assurance cc service management customer experience customer portal & reporting transactionanalysis SLAs and service models IT Process definition and automation service management Infrastructure management provisioning usage performance fault network services systems &servers application &web services data & storage

14. operational monitoring with openview your customer or end-user top-down customer experience monitoring event management resource management infrastructure monitoring your computing environment bottom-up

15. diagnosing network problems • network node manager intelligent diagnostics • openview problem diagnosis • openview performance insight for networks

16. diagnosing system problems • openview performance manager and agent • hp openview glanceplus • top tools and compaq insight manager

17. diagnosing application problems • hp openview transaction analyzer • application response measurement (ARM) • hp openview performance manager • hp openview reporter • 3rd party integrations (e.g. Oracle Enterprise Manager Diagnostics and Tuning Packs)

18. OpenView Reporter OpenView Performance Manager automated data collection, consolidation, and web-based reporting from multiple OV sources OpenView Performance Agent real-time performance monitoring and diagnostics NETWORKS SYSTEMS INTERNET APPS DATABASES OpenView Performance management stack web-based analysis, planning and resource management central management system system performance and transaction management repository and data provider OpenView GlancePlus managed node …provides the flexibility and depth required for comprehensive infrastructure management

19. HP OpenView GlancePlus • ~1000 metrics • 1 sec interval • Global, App, Process data • Application Response Measurements (ARM) • System tables

20. Easy to use web GUI to provides distributed near real-time access to multiple data sources from heterogeneous systems OV Performance Manager 4.0

21. diagnosing slo alarms e2e response OVIS internet web server app server database network segment legacy app Transaction Analyzer ISP desktop firewall web page DNS DHCP.. JSP ASP O/S EJB COM SQL path main frame Keynote Performance Manager 3rd party OVTA OVIS Transaction Analyzer Transaction Analyzer Transaction Analyzer 3rd party process switch router Performance Manager GlancePlus Performance Insight Problem Diagnosis

22. infrastructuredrill-down OVTA OVTA get_stock_trad • it’s the application! • which sub-transaction is it? client communication 1 check_credit Web svr move fund lookup_bond communication 2 buy_commodity application svr 1 network drill-down OVPD client/server drill-down OVPD communication 3 hop1 path1 hop2 app svr 2 path2 hop3 communication 4 path3 hop4 path4 hop5 database svr 1 path5 • where’s the problem? • which of my IT silos is the culprit? • it’s the network! • what’s the most popular path? • where’s the slow path? • it’s path 3! • which device is it? drilling down to the source OVIS • what is the customer experience?

23. application response measurement • industry standard interface • method for easily monitoring service levels in standalone or distributed applications • simple API to instrument source code to track transactions • SDK that ships with OpenView Performance Agent and Glance products • facility used by several other OpenView products START what is ARM? ARM business application STOP management agents & applications network business application ARM COMPLETED response time transaction status measurements drilldown client to server

24. thank you!