Redundant Parallel File Transfer with Anticipative Adjustment Mechanism in Data Grids

1. Redundant Parallel File Transfer with Anticipative Adjustment Mechanism in Data Grids Chao-Tung Yang, Yao-Chun Chi, Chun-Pin Fu, High-Performance Computing Laboratory Department of Computer Science and Information Engineering Tunghai University, Taichung City, 40704, Taiwan R.O.C. Presented by 張肇烜

2. Outline • Introduction • Co-Allocation Mechanism • System Design And Implementation • Related Works • Experimental Results And Analyses • System Components • Conclusion

3. Introduction • Most Data Grid applications execute simultaneously and access large numbers of data files in the Grid environment. • In Data Grid environment, large data sets are replicated and distributed to multiple servers.

4. Introduction (cont.) • Downloading large datasets from any replica locations may result in varied performance rates. • The downloading speed is limited by the bandwidth traffic congestion in connecting the server and the client.

5. Introduction (cont.) • The methods uses to improve the downloading speed: • Replica selection techniques. • Co-allocation architecture. • Anticipative Recursive-Adjustment Co-allocation.

6. Co-Allocation Mechanism • We used the grid middleware Globus Toolkit as the data grid infrastructure. • One of its primary components is MDS. • And it uses GridFTP. • Replica Location Service (RLS)

7. Co-Allocation Mechanism (cont.)

8. System Design And Implementation

9. System Design And Implementation (cont.)

10. System Design And Implementation (cont.) • A new section of a file to be allocated is first defined. • SEj=(unassignedFileSize+Total UnfinishedFileSize)*a (0<a<1) • SEj denotes the section j such that 1<=j<=k

11. System Design And Implementation (cont.) • In the next step, SEj is divided into several blocks and assigned to n servers.

12. System Design And Implementation (cont.) • A faster channel finishes its assigned data blocks at real finished time RTj may later or earlier than expected time Tj, TSi denotes the actually transfer size at the real finished time RTj is:

13. System Design And Implementation (cont.)

14. Related Works • Brute-Force Co-Allocation • History-based Co-Allocation • Conservative Load Balancing • Aggressive Load Balancing • Co-Allocation Scheme with Duplicate Assignments (DCDA)

15. Experimental Results and Analyses • Network variation between client and servers.

16. Experimental Results and Analyses (cont.) • Completion time of different methods.

17. System Components

18. System Components (cont.) • This portal is to simplify the operation of replicas management and file transfer. • Replica Selection Service • Anticipative Recursive-Adjustment Co-Allocation • One-way Replica Consistency Service • Dynamic Maintenance Service

19. Conclusion • Dynamic co-allocation scheme had led to performance improvement, but it has some shortcoming. • The goal is to reduce the total idle time on awaiting the slowest server.