1 / 16

T3mon s tatus

This project focuses on creating detailed installation and configuration guides, reinstallation procedures, and monitoring plug-ins documentation. Pilot sites validate the documentation, with successful XRootD monitoring at IHEP and JINR DNLP. Sharing on Twiki at CERN T3mon-site dataflow. Technologies used include PostgreSQL, Python, Ganglia, and Web UI for visualization.

hectore
Download Presentation

T3mon s tatus

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. T3mon status ArtemPetrosyan, DanilaOleynik DNG section meeting, 05.07.11

  2. Testbed at JINR • Multicore nodes, virtualization • 5 clusters • XRootD • PROOF • PBS • OGE/SGE • Condor • Lustre • Ganglia • XRootD, PBS, OGE/SGE, Lustre • Nagios • XRootD • JobMonarch • PBS • OGE/SGE ATLAS TIM

  3. Load simulation • XRootD • User login/log out, file access • PBS • Test jobs • OGE/SGE • Test jobs • PROOF • Test physics analysis jobs ATLAS TIM

  4. Validation • Installation, create documentation which assembles references to installation and configuration instructions for particular batch and storage solutions and documentation for monitoring plug-ins • Reinstallation basing on the prepared documentation • Validation of documentation by pilot sites • Successful XRootD monitoring implementation at IHEP and JINR DNLP • Documentation sharing • Twiki at CERN ATLAS TIM

  5. T3mon-site dataflow ATLAS TIM 31.05.11 5

  6. Technologies ATLAS TIM • PostgreSQL • JobMonarch backend • Have to use MySQL as a temporary backend • Python • ATLAS DDM, Dashboard development language • Ganglia • RRD for storage • Web UI for visualization 31.05.11 6

  7. XRootD dataflow ATLAS TIM 31.05.11 7

  8. XRootD add-on • Summary database structure is ready • Multithread application • Read detailed monitoring stream • Publish into Ganglia and database backend • Metrics • User login/logout • File access • File transfer • Status • Reader is ready • Writer is in development ATLAS TIM

  9. PROOF dataflow ATLAS TIM 31.05.11 9

  10. PROOF add-on • Summary database structure is ready • Normalized structure • Triggers are used for data normalization • Metrics • User • Start-end time • CPU • Wall time • Dataset name • Number of files in the dataset • Number of events • Number of workers • Status • Data is being collected in the database • Publisher into Ganglia is in development • Open issues • PostgreSQL connectivity ATLAS TIM

  11. T3mon-site summary - done + - in progress ATLAS TIM

  12. T3mon-global • Dashboard – common collector and presenter of Tier3 sites global monitoring • Metrics compatibility • Common technologies • Metrics for global monitoring defined and collected on local sites • Job processing • Data transfers • Data access • Messaging System for the Grid (MSG) based on ActiveMQ is used as a message bus ATLAS TIM

  13. T3mon data flow ATLAS TIM

  14. Job processing metrics with PROOF ATLAS TIM

  15. Data transfers metrics • Defined in Dashboard for FTS: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGTransferMonitoring • Should cover list of metrics given by xRootd ATLAS TIM

  16. Todo list • Finalize Tier3 site development • xRootd, PROOF solutions • Ensure robust handling of the monitoring agents • Documentation and testing • Develop producers for transferring data to the global level ATLAS TIM

More Related