1 / 12

Progress on Transfers Dashboard Applications Alexandre Beche

Progress on Transfers Dashboard Applications Alexandre Beche. Landscape of Data Transfer monitoring application(*). WLCG Dashboard. AAA WEB API. FAX WEB API. WLCG WEB API. XRootD integration. Timeline. XRootD detailled dashboard. WLCG DB. AAA DB (CMS). FAX DB (ATLAS). EOS

miyo
Download Presentation

Progress on Transfers Dashboard Applications Alexandre Beche

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Progress on Transfers Dashboard Applications AlexandreBeche

  2. Landscape of Data Transfer monitoring application(*) WLCG Dashboard AAA WEB API FAX WEB API WLCG WEB API XRootD integration Timeline XRootD detailled dashboard WLCG DB AAA DB (CMS) FAX DB (ATLAS) EOS Integration ALICE integration FAX EOSA FTS ALICE AAA EOSC (*) WLCG and XRootD Dashboards, not ATLAS DDM AMQ IT-SDC-MI, AlexandreBeche

  3. A Federated approach for DT monitoring All data retrieved through web API and aggregated together FED EOS is now in the global picture AAA WEB API FAX WEB API FTS WEB API FTS DB AAA DB (CMS) FAX DB (ATLAS) No more duplicationin RAW Data FAX EOSA FTS ALICE AAA EOSC AMQ IT-SDC-MI, AlexandreBeche

  4. XRootD Monitoring Dashboard CMS ATLAS • News: • Optimization in the DB • Workflow fully instrumented • Fully puppetized (thanks Pablo) http://dashb-atlas-xrootd-transfers.cern.ch http://dashb-cms-xrootd-transfers.cern.ch Two instances Openstack cluster CHEP13 - Alexandre Beche

  5. Monitoring & Automatic recovery procedure N : Notification R : Automatic restart IT-SDC-MI, AlexandreBeche

  6. FTS Dashboard • FTS specific view • Lot of new requirements • Production ready http://dashb-fts-transfers.cern.ch • *New* application • Better performance • No more XRootD data • Consolidated dataflow • Common tool used IT-SDC-MI, AlexandreBeche

  7. WLCG Transfers Dashboard • Fully puppetized • New style LB alias http://dashb-wlcg-transfers-new.cern.ch Top level User Interface No Database, only REST API Multi-threaded call IT-SDC-MI, AlexandreBeche

  8. Current status on migration to the new infrastrucutre Everything is in openstack … … fully puppetized (thanks again Pablo) … … and robust (automatic procedure) IT-SDC-MI, AlexandreBeche

  9. Short term Roadmap • Development of new functionality is frozen • Priority is on deployment in production • Finish the migration to new prod brokers • 3 missing FTS servers • Decommission old machine • Dashboard 24 & 72  Done • Dashboard 58, 63 & 71  Soon (couple of weeks) IT-SDC-MI, AlexandreBeche

  10. Long term Roadmap • Rewrite the topology package • New requirements • Does not fit new model • FTS server to site resolution (GLUE) • Consolidate the collector package • Collector monitoring using simplevisor • Move all the LB aliases to the new system • … (long list) IT-SDC-MI, AlexandreBeche

  11. Side comments • Faster releasing cycle for XRootDdashboards • Current version of WLCG TD is 1 year old • Single DB (no INT / PROD yet) ? • Too complex ? • Improve the development cycle for data transfers applications • Agile-like development would be good • Will start with XRootD but scope should be bigger IT-SDC-MI, AlexandreBeche

  12. Conclusion Thanks for your attention • Data management monitoring application are moving forward • Big refactoring is almost done • Minimal maintenance effort (theoretically) • Time for new development • Lot of lesson has been learned • Consuming, puppet structure, LB alias IT-SDC-MI, AlexandreBeche

More Related