1 / 10

Parallel File System Simulator

Parallel File System Simulator. tests. . tests. . In order to test the Parallel File System (PFS) scheduling algorithms in a light-weighted approach, we have developed the PFS simulator.

camila
Download Presentation

Parallel File System Simulator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Parallel File System Simulator tests. tests. In order to test the Parallel File System (PFS) scheduling algorithms in a light-weighted approach, we have developed the PFS simulator. We use discrete event simulator and accurate disk modeling software to construct an agile parallel file system simulator. This simulator is capable of simulating enough details while having an acceptable simulation time. HEC FSIO 2009

  2. Existing PFS simulators • The IMPIOUS simulator, by E Molina-Estolano, al. et[1]. • It does not model the metadata server. • The scheduler modules are lacking, so scheduling algorithms are hard to model. • The simulator developed in PVFS improvement paper by Carns P. H., al. et[2]. • It over simulates the network, which extends the simulation time. • It uses real PVFS in simulation, which introduces too much details, while not flexible. HEC FSIO 2009

  3. Simulator Details • We use the discrete event simulation library OMNeT++ 4.0 to simulate the network. • It is capable of simulating the network topology with bandwidth and delay. • We use Disksim to simulate the data server disks. • Disksim accurately estimates the time for data transactions on the physical disks. • Disksim allows users to extract disk characteristics from real disks. HEC FSIO 2009

  4. Simulator Architecture trace trace trace Client Client Client queue queue queue Local FS Local FS Local FS Disk Disk Disk Stripping Strategy New request arrives. output Metadata Server Query file layout. Metadata Server Simulated Network Send the request to data server. Scheduling Algorithm Data Servers OMNeT++ Dispatch the job. Socket Connections Send the results back. Disksims instances HEC FSIO 2009

  5. Implementation of Scheduling Algorithms /* SFQ algorithm, at each data server */ systime = 0 waitQ.initiate() while(!simulation_end) { if reqArrive(), then: R = getReq() R.start_tag = min { R.getPrevReq().finish_tag, systime } R.finish_tag = R.start_tag + R.cost / R.getFlow().weight pushReq(R, waitQ) if diskHasSlot(), then: R = popReqwithMinStartTag(waitQ) systime = R.start_tag dispatch(R) } The request from client arrives at the data server. Disksim tells OMNet++ that it still has free slot. OMNet++ dispatches the request to Disksim. Similarly, we have also implemented FIFO, Distributed SFQ[3], MinSFQ[4] algorithms, etc. HEC FSIO 2009

  6. Simulation 1: Weighted Scheduling • 2 groups of clients, 16 each. • 4 Data servers and 1 Metadata server. • Parallel Virtual File System(v2) is modeled. • All files striped to all servers. Stripe size: 256KB. • The trace files are generated by IOR. • Each client has 100MB checkpoint-write operation. • SFQ with different weight assignments have been simulated. • The simulated weight ratios are 1:1, 2:1 and 10:1. HEC FSIO 2009

  7. Throughput Results HEC FSIO 2009

  8. Simulation 2: Scheduling Fairness • Same as Simulation 1, except that the two groups do not have the same resources: • Group 1’s files are stripped to all servers [1, 2, 3, 4]. • Group 2’s files are stripped to 3 servers [1, 2, 3]. • We add Distributed SFQ algorithm[3], in which all servers share the scheduling information. • We are able to show that if the workloads are not evenly distributed, the Distributed SFQ algorithm achieves better fairness than FIFO and SFQ. HEC FSIO 2009

  9. Throughput Results HEC FSIO 2009

  10. References [1] E Molina-Estolano, C Maltzahn, J Bent and S A Brandt, “Building a parallel file system simulator”, Journal of Physics: Conference Series 180 (2009) 012050. [2] Carns P H, Ligon W B, al. et. “Using Server-to-Server Communication in Parallel File Systems to Simplify Consistency and Improve Performance”, Proceedings of the 4th Annual Linux Showcase and Conference (Atlanta, GA) pp 317-327. [3] Yin Wang and Arif Merchant, “Proportional Share Scheduling for Distributed Storage Systems”, File and Storage Technologies (FAST’07), San Jose, CA, February 2007. [4] W. Jin, J. S. Chase and J. Kaur, “Interposed Proportional Sharing For A Storage Service Utility”, SIGMETRICS, E. G. C. Jr., Z. Liu, and A. Merchant, Eds. ACM, 2004, pp. 37-48. HEC FSIO 2009

More Related