1 / 20

Catalin Cirstoiu (IT/PSS/ED) Costin Grigoras, Latchezar Betev (PH/AIP)

Monitoring, accounting and automated decision support for the ALICE experiment based on the MonALISA framework. Catalin Cirstoiu (IT/PSS/ED) Costin Grigoras, Latchezar Betev (PH/AIP) EGEE User Forum, May 9 th -11 th 2007, Manchester. Contents. Monitoring requirements MonALISA overview

brittany-graves
Download Presentation

Catalin Cirstoiu (IT/PSS/ED) Costin Grigoras, Latchezar Betev (PH/AIP)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Monitoring, accounting and automated decision support for the ALICE experiment based on the MonALISA framework Catalin Cirstoiu (IT/PSS/ED) Costin Grigoras, Latchezar Betev (PH/AIP) EGEE User Forum, May 9th-11th 2007, Manchester

  2. Contents • Monitoring requirements • MonALISA overview • Application monitoring • Monitoring architecture in AliEn • Jobs monitoring • Traffic monitoring • Services monitoring • Nodes monitoring • Actions framework • Feature snapshots Catalin.Cirstoiu@cern.ch

  3. Monitoring Requirements • Global view of the entire distributed system • Non-intrusive • Accurate • Providing • Near real-time information • Long-term history of aggregated data • On key parameters like • System status • Resource usage • Helping with • Correlating events • System debugging • Generating reports • Taking automated actions based on the monitored data Catalin.Cirstoiu@cern.ch

  4. Postgres MySQL Data Store Lookup Service Lookup Service Data Cache Service & DB Web Service WSDL SOAP Registration Discovery WS Client (other service) Data (via ML Proxy) Predicates & Agents Configuration Control (SSL) Applications Java Client (other service) Agents Filters Data Modules MonALISA Overview • MonALISA is a Dynamic, Distributed Service Architecture capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems. Catalin.Cirstoiu@cern.ch

  5. dynamic reloading Config Servlet App. Monitoring MonALISA hosts UDP/XDR UDP/XDR UDP/XDR APPLICATION Time;IP;procID MonitoringData MonitoringData MonitoringData parameter1: value MonALISA Service parameter2: value ApMon ... APPLICATION MonALISA Service ApMon App. Monitoring No Lost Packages Mbps_out:0.52 System Monitoring ApMon configuration generated automatically by a servlet / CGI script Status: reading load1:0.24 ApMon Config processes: 97 MB_inout: 562.4 pages_in:83 ApMon – Application Monitoring • Lightweight library of APIs (C, C++, Java, Perl, Python) that can be used to send any information to MonALISA Services • High comm. performance • Flexible • Accounting • Sys Mon Catalin.Cirstoiu@cern.ch

  6. AliEn CE AliEn CE Cluster Monitor Cluster Monitor AliEn IS AliEn Optimizers AliEn Job Agent AliEn Job Agent AliEn Brokers ApMon ApMon AliEn TQ ApMon ApMon ApMon ApMon AliEn SE AliEn SE ApMon ApMon ApMon ApMon MySQL Servers ApMon ApMon ApMon CastorGrid Scripts AliEn Job Agent AliEn Job Agent AliEn Job Agent AliEn Job Agent ApMon ApMon ApMon ApMon ApMon API Services ApMon MonALISA LCG Site MonALISA @CERN MonALISA @Site Monitoring Architecture in AliEn job slots net In/out run time cpu time free space processes load jobs status vsz sockets rss migrated mbytes active sessions Aggregated Data nr. of files open files Queued JobAgents job status MonaLisa Repository Alerts cpu ksi2k Actions Long History DB disk used MyProxy status http://pcalimonitor.cern.ch:8889/ LCG Tools Catalin.Cirstoiu@cern.ch

  7. Job Status Monitoring • Global summaries • For each/all conditions • For each/all sites • For each/all users • Running & cumulative • Error status • From job agents • From central services • Multiple views • Real-time map • Integrated pie charts • Long history plots Catalin.Cirstoiu@cern.ch

  8. Real-time map Catalin.Cirstoiu@cern.ch

  9. Integrated Pie Charts Catalin.Cirstoiu@cern.ch

  10. History Plots, Annotations Catalin.Cirstoiu@cern.ch

  11. Job Resource Usage Monitoring • Cumulative parameters • CPU Time & CPU KSI2K • Wall time & Wall KSI2K • Read & written files • Input & output traffic (xrootd) • Running parameters • Resident memory • Virtual memory • Open files • Workdir size • Disk usage • CPU usage • Aggregated per site Catalin.Cirstoiu@cern.ch

  12. Job Network Traffic Monitoring • Based on the xrootd transfer from every job • Aggregated statistics for • Sites (incoming, outgoing, site to site, internal) • Storage Elements (incoming, outgoing) • Of • Read and written files • Transferred MB/s Catalin.Cirstoiu@cern.ch

  13. Individual Job Tracking • Based on AliEn shell cmds. • top, ps, spy, jobinfo, masterjob • Using the GUI ML Client • Status, resource usage, per job Catalin.Cirstoiu@cern.ch

  14. AliEn & LCG Services Monitoring • AliEn services • Periodically checked • PID check + SOAP call • Simple functional tests • SE space usage • Efficiency • LCG environment and tools • Integrating the VoBOX tests previously run by ML within the SAM framework • Proxy lifetime, gsiscp, LCG CE/SE, Job submission, BDII, Local catalog, software area etc. • Error messages in case of failure • Efficiency • ML Alerts are used for problems notification • . Catalin.Cirstoiu@cern.ch

  15. FTD/FTS Monitoring • Status of the transfers • Transfer rates • Success/failures • Efficiency via ARDA Experiment Dashboard Catalin.Cirstoiu@cern.ch

  16. VOBox/Head Node Monitoring • Machine parameters, real-time & history • Load, memory & swap usage, processes, sockets Catalin.Cirstoiu@cern.ch

  17. Traffic Jobs Hosts Apps ML Service Actions based on global information ML Repository Actions based on local information Temperature Humidity A/C Power … ML Service Sensors Local decisions Global decisions Actions Framework • Based on monitoring information, actions can be taken in • ML Service • ML Repository • Actions can be triggered by • Values above/below given thresholds • Absence/presence of values • Correlation between multiple values • Possible actions types • Alerts • e-mail • Instant messaging • RSS Feeds • External commands • Event logging Catalin.Cirstoiu@cern.ch

  18. Alerts and Actions MySQL daemon is automatically restarted when it runs out of memory Trigger: threshold on VSZ memory usage ALICE Production jobs queue is automatically kept full by the automatic resubmission Trigger: threshold on the number of aliprod waiting jobs Administrators are kept up-to-date on the services’ status Trigger: presence/absence of monitored information Catalin.Cirstoiu@cern.ch

  19. Summary • The MonALISA framework is used as a primary monitoring tool for the ALICE Grid since 2004 • Presently the system is used for monitoring of all (identified) services, jobs and network parameters necessary for the Grid operation and debugging • The number of concurrently monitored and stored parameters today is ~ 300.000 in 75 ML Services • The add-on tools for automatic events notification allow for more efficient reaction to problems • The framework design and flexibility answers all requirements for a monitoring system • The accumulated information allows to construct and implement automated decision making algorithms, thus increasing further the efficiency of the Grid operations Catalin.Cirstoiu@cern.ch

  20. Thank You Questions? http://alien.cern.chhttp://monalisa.caltech.edu Catalin.Cirstoiu@cern.ch

More Related