GridMon

1. GridMon EGEE JRA4 R-GMA workshop Thursday 22nd July 2004 University College London Mark Leese

2. Contents Very simple (like me ;-) • GridMon: • where it has come from • how it works • what it does • where it is going • Conclusion • Lots of questions

3. Project: brief UK e-Science approved project proposal to… “…design and deploy an infrastructure for network performance monitoring within the UK e-Science community.” This became… • kit of monitoring tools to be deployed at each e-Science centre, focusing on: • publication to Grid middleware • visualisation for humans • end-to-end performance (what users see)

4. GridMon: progress • Work began in June 2002. • Resulting toolkit called “GridMon” (everyone needs a good name ;-) • Building on work of EDG WP7, initial presence of toolkit established at 12 original UK e-Science Centres (Cambridge, Daresbury, RAL…) • As always, some problems – but being debugged Visualisation for humans

5. GridMon: review • Well received by people who use it • Experience feeding into other UK projects • Fed back into EDG • Interest growing, e.g. UK HEP groups, and UK National Grid Service (NGS) • In some form or another will likely be part of proposed UK GOC So, good foundations laid But how does it work and what’s next?

6. Publication Service IperfER PingER UDPmon Monitor Node Grid middleware MiperfER bbcp/ftp www.visualisation Installed on dedicated & similar nodes at each centre MESH 30 mins How does it work? data stored locally .txt files Linux Simple Architecture

7. Where next? • New funding from UK e-Science until March 2005 • You’ve seen visualisation for humans, so next is publication to middleware: • Back in June 2002, LDAP looked the way to go • Then in response to WP7, R-GMA looked the best way (middleware would use R-GMA queries to get data) • Now we (and fortunately most of the World) are looking at web and Grid services • Will use GGF NM-WG schemas for data requests and publication of corresponding results: • In GridMon’s case you can request only historic data, no predictions or on-demand/future tests (at least not yet) • For simplicity, graph plotting and publication to middleware will have their own interfaces into the data • But first, we must select a new data storage method, because the current .txt files would prove to a be problem: • .txt files are good for the human visualisation • Writing into an ascii text file is relatively quick and easy • Minimal programming required • The performance graphs plot continuous data for a particular period, so no need to “jump around” within a dataset • .txt files bad for powerful web service • Don’t easily support complex queries, like “give me all measurements from XY for the last week, where RTT ≥ 100mS” • So investigation will have to be made into alternative choices, which are likely to be RRDTool (see next slide) or RDBMS, or both.

8. RRD Tool? • Final year undergraduate student (Chong Hei Cheng) produced prototype enhanced graph plotter for GridMon • Allows plots of data from all tools on the same graph • Uses “Round Robin Database Tool” to do this • System to store and display time series data • Produces plots like MRTG’s graphing functions • Takes .txt for time period, inserts into RRD, then plots • Better if data is stored in RRD when it is first collected. Don’t know if human visualisation is of interest to JRA4 – perhaps for GOC?

9. test request (request schema) tests results (publication schema) www.ggf.org Fast GGF advert • GGF Network Measurements Working Group (NM-WG) are defining XML schemas for requesting tests and historic data, and publishing network measurements • Aims: to standardise communication, and… • …use XML, for web services and OGSI/WSRF model • Simple use case… • All request & result messages can be formatted using standardised schemas = truly powerful combination • How data is stored and then retrieved to be sent as XML is irrelevant (could be LDAP, R-GMA, .txt….) it is obviously hidden from the requestor! Network Monitoring Service

10. Grid/Net Operations GOC/NOC Admin Software Grid Applications Network domain X Network domain Y Network domain Z Automated Test Systems Grid Middleware Network Monitoring Service Network Monitoring Service Network Monitoring Service Other Network Services Other Network Services Other Network Services Another fast advert • GGF GHPN-RG (Grid High Performance Networking) looking at network services. • Diagram shows (numerous and varied) potential clients of network monitoring service: • All this GGF work is funded by GridPP2

11. Web service: how? • How is web service likely to be realised? • Apache Tomcat & Axis providing web service interface to Java programs that query database/datastore • Simple example shown at 2003 UK e-Science All Hands Meeting • “Proper” version will need to be much more complicated and robust • In RelaxNG format from OASIS, not W3C’s WXS • More readable • More modular • Will use Document/Literal SOAP binding • allows validation of messages against the schemas • RPC/Encoded binding (most common/default) makes validation difficult or impossible

12. Conclusion • GridMon: • UK infrastructure with existing graph plotting functionality • graph plotting will be enhanced • Web services interface for middleware etc. to be added, using GGF NM-WG schemas • Migration of datastore to more powerful technology to be investigated Questions m.j.leese@dl.ac.ukhttp://gridmon.dl.ac.uk/ ? ? ? ? ?