1 / 14

Bulk Data Transfer Based on Effective Transfer Rate

Bulk Data Transfer Based on Effective Transfer Rate. 2011_4_20 Gao Xiang. Bulk Data Transfer. Application Optical circuit-switched networks offer end-to-end dedicated connection with high-capacity bandwidth(e.g., GE, 10GE) Distributed Scientific Computing Cloud Computing

abra-ramirez
Download Presentation

Bulk Data Transfer Based on Effective Transfer Rate

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bulk Data Transfer Based on Effective Transfer Rate 2011_4_20 Gao Xiang

  2. Bulk Data Transfer Application Optical circuit-switched networks offer end-to-end dedicated connection with high-capacity bandwidth(e.g., GE, 10GE) Distributed Scientific Computing Cloud Computing Problem in High Bandwidth & Delay Network Protocol: TCP vs UDP TCP: Additive Increase Multiplicative Decrease UDP: Non-connection-oriented unreliable service End-systems: Bottleneck NIC & Storage System

  3. File Transfer File In Host A File In Host B File Stream File Stream Application Application TCP or IP Packet TCP or IP Packet Socket Buffer Socket Buffer IP Packet IP Packet NIC NIC 1 2 3 4 5 6 7 8 Network Transfer

  4. Packet Loss Packet loss occurs mainly at the end-system instead of the network Packet loss in NIC Packet loss in socket buffer

  5. Bottleneck of Storage System Except for real-time streaming, transferred data are generally stored in end-systems Storage Medium: CD (DVD), Hard Disk, Flash Hard Disk: Winchester Structure, low speed, large capacity Flash: SSD (solid-state drive),high speed, small capacity, short life Compromise: RAID (Redundant Array of Independent Disk) RAID-5 in our server Write Speed: 239 MB/s Read Speed: 277 MB/s

  6. Effective Transfer Rate Effective Transfer Rate (ETR) is the best transfer rate but may not be the highest one. For example, if sending rate is so fast that the receiver cannot accept all the data correctly as soon as possible, causing the sender needs to resend the lost data. The problem is how to determine ETR High speed means high packet loss rate Low speed means low packet loss rate but longer transfer time For example, a 10GB file to transfer. 1 At high transfer rate 100 MB/s, packet loss rate 20%, the total time is 12s; At low transfer rate 80 MB/s, packet loss rate 0%, the total time is 12.5s; 2 At high transfer rate 100 MB/s, packet loss rate 50%, the total time is 15s; At low transfer rate 80 MB/s, packet loss rate 0%, the total time is 12.5s;

  7. ETR with Bottleneck of SS ETR can’t be above the read speed of hard disk. ETR can be above the write speed of hard disk by using big buffer in RAM R_recv* T_recv= R_disk* T_disk= Size_file If R_recv>R_disk, then T_recv<T_disk When arrive T_recv, the written data is R_disk* T_recv The unwritten data is (R_disk - R_disk) * T_recv, and it’s the buffer size for improving ETR So ETR=R_disk/(1-m/f)

  8. Structure of Reliable UDP Error Control Adding a sequence number at the head of every packet could help the receiver to notice the disorder or packet loss Rate Control Interval between packets: R=P/I, where P is packet size and I is the interval time. We can using CPU clock cycles to improve the accuracy Client Server TCP PLM RXM RCM, etc TCP File Packet Stream UDP UDP

  9. Structure of Programs Client Server Retransfer Retransfer Retransfer Thread Control Thread Network Rate Control Receiving Thread Sending Thread Buffer Buffer Rate Control Thread Rate Control File Thread File Thread File File

  10. Experiment Result Throughput under Different Buffer Size Transfer Time under Different File Size

  11. ETR with Packet Loss Rate High packet loss rate means a lot of packets need to be retransferred and it wastes the bandwidth resource Tested by Iperf, at rate about 6000Mbps, the lost rate is about 50%! At period k, let f be the remaining file size, l be the lost rate with sending rate r, the time to finish the task is t_finish(k)=f*(1+l)/r If period k t_finish(k)>t_finish(k-1)-t_period, ETR won’t be increased

  12. ETR Control Algorithm f(remaining file size) ,b(remaining buffer size),ETR, P(received packet num), L (packet loss num), R_read(max read rate), R_disk(average write rate), G(good send packet size) 1 ETR(k) = R_disk/(1-b/f) ; 2 G(k-1) = P(k-1)-l(l-1); 3 if(ETR(k) < ETR(k-1)) ETR(k) = ETR(k-1); 4 if(G(k-1) < G(k-2)) ETR(k) = ETR(k-2); 5 if(ETR(k) > R_read) ETR(k) = R_read; 6 return ETR(k);

  13. Discuss 1 Packet Loss Rate is random and unstable. 2 ETR definition 3 Mathematical Model

  14. Thanks

More Related