1 / 24

Ecotools Case Study : Database Monitoring at BNP Paribas

Ecotools Case Study : Database Monitoring at BNP Paribas. Dennis Adams BNP Paribas. What’s in a name : Paribas Capital Markets. Now part of BNP Paribas Investment Banking € Euro-Denominated Bonds. Equities, Bonds and Derivatives Trading Systems in most major financial centres

dillon
Download Presentation

Ecotools Case Study : Database Monitoring at BNP Paribas

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ecotools Case Study : Database Monitoring at BNP Paribas Dennis Adams BNP Paribas

  2. What’s in a name : Paribas Capital Markets Now part of BNP Paribas Investment Banking € Euro-Denominated Bonds. Equities, Bonds and Derivatives Trading Systems in most major financial centres Infrastructure in territories and central London Data Centre. Finance and Reporting in Paris.

  3. Databases and OS Types in Production Tokyo, Singapore, Hong Kong, Paris, London, New York

  4. Our Responsibility: Data Management Group Managing DBMS Servers Based in London, remote Support to territories (out-of-hours callouts !) Liaise with London Operations 24h/day Other Teams Systems Management - hardware & OS Networks - LAN & WAN Application Teams - Application Support

  5. Objectives when purchasing Ecotools Monitoring of Sybase DBMS detail down to SQL statement Ingres as an additional requirement Reliable WAN based Event Alerting Links to TNG for Central Alerting Extract trend data for capacity planning

  6. How we use Ecotools GUI Monitored Agent Machines Alerts Command Line Control Files for Scenarios Ecotools Repository

  7. How we use Ecotools Single Central Solaris Console (V6.2.1) Continuous running 24 X 7 Minimal use of the Ecotools GUI Scenarios controlled from ECOCLI start/stop UNIX shell scripts. Alerts picked up by “tail” of ecotools log summary on daily in-house web page

  8. Configuring Domains • Domain = Logical grouping of Servers • Unit of “discovery” within Ecotools • Monitor “All Servers” together

  9. Configuring Domains • Group by Business Unit / DBMS Type / Territory ? • Lots of small Domains ? • speed up “discovery” • Few no of large Domains ? • easier to get to individual machines • COMPROMISE: DOMAINS OF MAX 25 • ORGANISED BY APPLICATION TYPE (ALMOST)

  10. Creating Scenarios • Scenario = Basic Collection Task • One or more agents (cache hit, log size) • One or more machines (“All Servers”) • Time Interval (“10 Minutes”) • Either using the GUI or from .CTL file • CREATE USING GUI, SAVE AS A CTL TEMPLATE... SED/AWK • START FROM UNIX SHELL

  11. Managing Scenario Control Files • Lots of Small Control files ? • One UNIX impf process per scenario • a single machine can have 15 “request”s • 80 *15 impf processes on console (!!!) • ps -ef | more • Few no of Large Control files ? • Unix VI ctl file when new machine added. • Lots of machines per ctl file makes restart a problem

  12. Managing Control Files • EVERY MACHINE HAS IT’S OWN CONTROL FILES (MAXIMUM OF 3) • Use SED string substitution to created from standard Templates • Ecotools Startup logic... • For file in *.ctl • do • ecocli • eco: run $file • done

  13. GUI Stability Issues • GUI “hangs” when Alerts arrive. • Determined by ECOCLI_BLINKALARM • ECOCLI_ALARMINTERVAL =1200 default = 2*CliLogInterval • The GUI Crashes ! • Corrupts master imdb - lose all data • Better in version 6.2.1 (latest patches) • SWITCH OFF ECOCLI_BLINKALARM, USE UNIX SCRIPT TO TAIL LOG FILE

  14. Managing Ecotools “Views” • View = Repository for performance data • consists of indexed/ flat file directory • Need X months data - capacity planning • Can keep views going for several weeks • but “imdb” = 12MB, .datalog = 1.8GB. • CREATE NEW VIEW EVERY MONTH • SET “NO OF DATA POINTS=1500” IN CONTROL FILES (OR LOSE DATA)

  15. Trend Data for Capacity Planning • Several Month’s data required • Start a new GUI for every single view ?? • SOLUTION : extract data to CSV files for reading into database • There is no supported utility to do this. • SHELL SCRIPTS - GREP / SED / AWK • UNSUPPORTED HACK !

  16. Unicenter/TNG Integration • In-house implementation calling KSH script from Ecotools Scenario Language • Can be implemented as “MgrAction” or “AgtAction” • MgrAction = run on Solaris Console • GMRPY_CDBSTOP errors - not scalable • AgtAction = run on Agent Machine • NOT YET PROVEN

  17. Summary : Ecotools Experience so far Evaluation of version 4.n in 1997 Monitoring of Euro changeover in 1998 Y2K tracking and Reporting 1999 Version 6.1 chosen as Y2K version Soon replaced by 6.2.1 with patches. Currently Monitoring nearly 80 Production Systems from London.

  18. Looking at Ecotools V.7 • Looks impressive on paper • SQL server storing collected data • Genuine 3-layer architecture • Better User interface • Concerns: • Support for Legacy Operating Systems • Dec UNIX versions • Ability to create own “shell script” agents. • Earlier Versions of Sybase

  19. Ecotools Future at BNP Paribas • Evaluate V7 for monitoring NT/Sybase Production Systems. • Need UNIX shell agents for NT console • Objective: move to centralised NT console - eventually • Keyword: STABILITY

  20. Success Stories • Sybase Settlement System - Tokyo • Reserved log space low • caused by backup server failing and transaction logs not being properly dumped. Restarted backup server • Potential system hang averted • Delayed Settlement = We get Fined

  21. Success Stories • Ingres Equity Derivative Trading - New York • Large Table Approaching Ingres Architectural Limit of 2GB • Planned outage to remodify table into multiple locations to allow it to grow • Equity Trading Outage averted • Keep the Traders Trading

  22. Success Stories • Ingres Forex Trading - Paris • Ingres Error Log reported potential UNIX Filesystem Error one evening • Alerted Paris Unix Team, who decided to ignore the warning • System Crashed overnight • But at least we could say “I told you so”

  23. Final Proof • Accepted within DMG Group • Integrated with our way of working • We are now a more Pro-Active Team • Looks good on the CV, provided you call it... • E-COTOOLS • Lunch Time ?

More Related