1 / 18

R-GMA

R-GMA. Producers and Archiver Paul Mealor. gsiwuftp. Overview. edg-netmon-archiver. edg-gridftplog2rgma. Log. Archiver API. P. Producer API. MySQL DB. Here be servlets. edg-netmon-rgma-info. P. Producer API. P. A. C. DBP. edg-pinger-timeping. edg-netmon2rgma. Producer API.

kasper-sanders
Download Presentation

R-GMA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. R-GMA Producers and Archiver Paul Mealor

  2. gsiwuftp Overview edg-netmon-archiver edg-gridftplog2rgma Log Archiver API P Producer API MySQL DB Here be servlets edg-netmon-rgma-info P Producer API P A C DBP edg-pinger-timeping edg-netmon2rgma Producer API FIFO

  3. edg-netmon2rgma • Maintains producers for the tools • Tools’ scripts are rerun for each measurement batch • Program running APIs must be persistent • Files • /opt/edg/sbin/edg-netmon2rgma.pl • /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo • /etc/rc.d/init.d/edg-netmon2rgmad • Configuration • RGMA client configuration

  4. Tool/netmon2rgma interaction PingER Normal log Web interface • addmetric pinger rtt 10 NMIdSource=pc18.hep.ucl.ac.uk tool=pinger edg-netmon2rgma If producer doesn’t exists If bufferSize < count * 4 / 3 • CircularBufferProducer_new(“NetworkRTT”, • “WHERE NMIdSource=‘pc18.hep.ucl.ac.uk’ AND tool=‘pinger’”); • CircularBufferProducer_setRemoteBufferSize($producer,10*2);

  5. Tool/netmon2rgma interaction PingER Normal log Web interface • measurement pinger rtt pc18.hep.ucl.ac.uk edgnm.gridpp.rl.ac.uk pinger 100 57 edg-netmon2rgma • CircularBufferProducer_insert($producer, • ”INSERT INTO NetworkRTT • (NMIdSource, NMIdDestination, tool, packetSize, value) • VALUES • (pc18.hep.ucl.ac.uk, edgnm.gridpp.rl.ac.uk, pinger, 100, 57)”);

  6. edg-netmon2rgma • Possibly should rewrite in Java • Problem in C (Perl) API results in memory leak and heavy performance hit • Current solution is to restart the daemon with a cron job • /etc/cron.d/edg-netmon2rgma-restart • But java lacks the text-handling of Perl • Would have to have java on the NM

  7. Tools modifications • Generally 2 new functions • initialise_rgma • Writes initialisation directives to the FIFO • i.e. fixed columns, metric names, expected number of entries • Netmon2rgma creates producers, changes buffer sizes when necessary • produce_it • Writes the results of the measurements • No extra configuration • Assumes /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo

  8. edg-netmon-rgma-info • Publishes NetworkSE, NetworkCE tables • Republishes every 12 hours (i.e. keepalive) • Files • /opt/edg/sbin/edg-netmon-rgma-info.pl • /opt/edg/etc/edg-netmon-rgma-info.conf • Local NM, associated SEs and CEs • /etc/rc.d/init.d/edg-netmon-rgma-infod • RGMA client configuration

  9. edg-ftlog2rgma • Publishes file-transfer logs (i.e. GridFTP) • Files • /opt/edg/sbin/edg-gridftplog2rgma.pl • /etc/rc.d/init.d/edg-gridftplog2rgmad • Reads from GridFTP log • /var/log/gsiwuftpd.log • Publishes new entries when they are added • RGMA client configuration

  10. edg-netmon-archiver • Program • /opt/edg/share/edg-netmon-archiver.jar • /opt/edg/sbin/edg-netmon-archiver • runs the Java code • /opt/edg/bin/edg-netmon-archiver-setup • Configures MySQL database; requires mysql root password • What to do in LCFG? • /opt/edg/etc/edg-netmon-archiver.conf • List of tables to archiver plus predicates for each • /etc/rc.d/init.d/edg-netmon-archiverd • Also RGMA client configuration

  11. edg-netmon-archiver • Ironically this might be the best package to write in C or Perl • Archiver API is all but unaffected by C API problem • But might run on R-GMA servlet machine • Interaction with edg-netmon-info • Archives every row that edg-netmon-info produces • Perhaps need to automatically remove duplicates (via mysql)

  12. Package progress • Tools, edg-netmon2rgma, edg-netmon-rgma-info • 99% • Latest tweaks still need to be uploaded • edg-ftlog2rgma • 75% • rc script, autobuild process • edg-netmon-archiver • 95% • need to upload, tag, check build process (ant)

  13. Testing • Tools on pc18.hep.ucl.ac.uk • edg-netmon2rgmad on pc18.hep.ucl.ac.uk • “User servlets” on pc23.hep.ucl.ac.uk • Registry and Schema servlets on pc23.hep.ucl.ac.uk • Something always goes wrong when I try to connect to the WP3 Registry at infocat.gridpp.rl.ac.uk • >1200 RTT/loss measurements archived by Tuesday • >130 TCP throughput measurements to localhost :)

  14. GridFTP logging in MDS Paul Mealor

  15. Status • Information provider • Will be in EDG 2 on SE • Publishes the most recent information for each host pair • MDS being phased out in EDG 2 • GIIS structure replaced by R-GMA • Will be all R-GMA later

  16. Details • Publishes achieved throughput • Filesize, time etc as parameters • Publishes most recent transfer between each host pair • Specifies source and host pair, plus, if known, source and host NM • NM names taken from config file as with other netmon info providers • Uses same schema as other netmon • But doesn’t use the full tree structure

  17. Schema dn: NMMeasureId=$source/$dest,$opt{rootdn} objectClass: NetworkMeasurement NMMeasureId: $source/$dest SourceHost: $source SourceNE: $sourceNE DestHost: $dest DestNE: $destNE MetricName: gridftpthroughput MetricValue: $throughput MetricUnit: bit/s MonitorTool: GridFTP MetricTime: $timestamp Parameter: time: $etime Parameter: size: $size Parameter: buffersize: $buffer Parameter: streams: $streams Parameter: stripes: $stripes createTimestamp: $timestamp modifyTimestamp: $timestamp

  18. END

More Related