1 / 20

Descriptive Data Analysis of File Transfer Data

Descriptive Data Analysis of File Transfer Data. Sudarshan Srinivasan Victor Hazlewood Gregory D. Peterson. Objective. Understanding the GridFTP log transfer data we have at NICS. Analyze the data and identify areas of potential improvement.

annona
Download Presentation

Descriptive Data Analysis of File Transfer Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Descriptive Data Analysis of File Transfer Data Sudarshan Srinivasan Victor Hazlewood Gregory D. Peterson

  2. Objective Understanding the GridFTP log transfer data we have at NICS. Analyze the data and identify areas of potential improvement. Perform predictive analysis to improve efficiency. Apply knowledge to XSEDE service providers.

  3. NICS GridFTP Infrastructure

  4. GridFTP Logging Gridftp data transfer protocol version 5.2.2. Two types of logging: "usage" logging and "log_transfer" logging (enabled in 5.2.2). Prior to 5.2.2 endpoint IP address data was filled with 0.0.0.0. Thanks to the Globus folks for fixing this bug!

  5. Transfer Logs NICS uses a PostgreSQL database for storing transfer log data. Two new tables: n_gridftp_usage and n_gridftp_usage_detail. n_gridftp_usage: quick lookup of aggregate monthly GridFTP usage information. n_gridftp_usage_detail: Detailed records of each data transfer. Log data includes: starttime, endtime, nbytes, user, filename, source and destination end points.

  6. Log Data Collection • Data from each GridFTP server is copied to log files to a central NFS location. • Each month we run a processing script on the log files that checks for errors in the log entry. • Following this, we run a script to load the log files into database table. • We chose transfer log data for the year 2013 for this analysis. DATE=20130401132041.657463 HOST=datamover1.nics.utk.edu PROG=globus-gridftp-server NL_EVNT=FTP_INFO START=2013041132041.534646 USER=username NBYTES=1048576 VOLUME=/ STREAMS=1 STRIPS=1 DEST=[192.249.6.164] TYPE=RETR CODE=226

  7. Log Data Analysis Two variables were identified: number of transfers and total amount of data transferred. Data transfer rate based on starttime, endtime and nbytes. Monthly visual comparison of data coming into and going out of NICS from everywhere. Intra XSEDE site number of transfers and data transferred coming into and going out of NICS. Bucketing of transfer data based on transfer size (ts). R statistical computing language was used to plot all histograms and graphs.

  8. Basic Statistics for the year 2013

  9. Number of transfers and amount transferred for the year 2013 Number of transfers (in millions) Total = 83.54 millions Mean Number of transfers (in millions) Total amount transferred (in TB) Total = 1235.7millions Total amount transferred (in TB) Month

  10. Percentage of transfers vs Transfer size for the year 2013 Total transfers: 67160380 Percentage of transfers Transfers size (ts)

  11. Transfer speed for top 500 transfers with transfer size > 1GB gbps Month

  12. Monthly comparison between number of transfers coming into and going out of NICS for year 2013 Total number of transfers (in millions) Month

  13. Monthly comparison between total amount of data coming into and going out of NICS for year 2013 Total amount of data moved (in TB) Month

  14. Transfer data buckets for November 2013 All transfers for November 2013, 1MB < ts < 64GBTotal transfers: 1431385 All transfers for November 2013, ts > 64GBTotal transfers: 25 All transfers for November 2013, ts < 1MBTotal transfers: 749747 All transfers for November 2013Total transfers: 2181157 Percentage of transfers Percentage of transfers Percentage of transfers Percentage of transfers Transfer size (ts) Transfer size (ts) Transfer size (ts) Transfer size (ts)

  15. Intra XSEDE Sites and Abbreviation

  16. Intra XSEDE site data coming into NICS TACC IU PSC OSG SDSC NCAR NICS/GaTech Number of transfers (in thousands) Total amount transferred (in TB) Month

  17. Intra XSEDE site data going out of NICS TACC IU PSC OSG SDSC NCAR NICS/GaTech Number of transfers (in thousands) Total amount transferred (in TB) Month

  18. Intra XSEDE site data coming into and going out of NICS together TACC IU PSC OSG SDSC NCAR NICS/GaTech Number of transfers (in thousands) Total amount transferred (in TB) Month

  19. Future Work • Currently in progress: • Moving from using PostgreSQL database to loading data completely in memory in a separate machine. • Using Apache Spark for fast large-scale data processing. • Combining SQL, streaming, and complex analytics. • Using advanced data mining and machine learning algorithms provided in libraries in Python. • Next Step: • Analyze by combing job data, filesystem data, and archive data for analysis. • Visualize data flow within XSEDE network on a geographical map.

  20. Thank You! Questions?

More Related