PVFS: A Parallel File System for Linux Clusters

1. PVFS: A Parallel File System for Linux Clusters Christina M Patrick CSE 598D

2. Overview of Different Types of File Systems CSE 598D

3. Need for Parallel FS’s (PVFS) • Each node in a Cluster of Parallel Computers has its own I/O subsystem, considerable I/O bandwidth, and system software. • Cater to applications having large file I/O requirements. CSE 598D

4. Functions of a Parallel FS • Allow data stored in a single file to be physically distributed among I/O resources in the cluster. • Provide a mechanism for each task in a parallel application to access distinct subsets of the data. CSE 598D

5. Limitations/Disadvantages of a Parallel FS • Lack of high performance synchronization mechanisms. • Lack of centralized control for partitioning, locating, and distributing I/O activities and agents. • Overheads associated with distributing and collecting data. • Severe mismatch between physical and logical views. CSE 598D

6. Unique Features of PVFS • Decouples data striping and file partitioning in a distributed environment. • Uses a streams-based approach. • Provides support for collective access operations. CSE 598D

7. PVFS was designed To Provide • Cluster wide consistent name space • Enables user controlled striping of data across disks on different I/O nodes • Existing binaries to operate on PVFS files without the need for recompiling CSE 598D

8. Above Limitations Motivate us towards … • Use of collective I/O primitives which specify I/O operations for an entire group of application tasks at once. • Centers on joint configuration of the parallel file system and a parallel application. CSE 598D

9. PVFS Design and Implementation • Designed to provide parallel access to files spread across multiple (homogeneous) machines and/or disks. CSE 598D

10. PVFS Design and Implementation … CSE 598D

11. Overview of Communication Client Manager Daemon I/O Daemon Ask for file location via TCP (thru client lib) Location of I/O nodes on which data is stored Time Read/Write Read/Write CSE 598D

12. Striping and Partitioning in PVFS stripe offset (so) offset from the beginning of the file where striping starts, stripe size (ss) indicates how many sequential bytes of the file are allocated to a stripe fragment, number of fragments (nf) which indicates how many stripe fragments are in a stripe, and node set which is the ordered set of nf I/O nodes that will hold the file's stripes. CSE 598D

13. Striping and Partitioning in PVFS A stripe location (sl) is an offset from the beginning of a stripe segment and only has meaning relative to a specific stripe segment number (sn), which is the ordinal number (counting from zero) of the I/O node in the node set that contains the given stripe segment. We can map a stripe location and segment number to a file location (fl) as follows: We can map a file location to a stripe location and segment number given the file location is not less than the stripe offset: CSE 598D

14. Partitioning Parameters Logical file partitions are defined with partition parameters as follows: partition offset (po) is the offset from the beginning of the file where the partition begins, group size (gs) indicates how many sequential bytes of the file are allocated to a group of the partition, and stride (sd) is the distance from the beginning of one group in the partition to the beginning of the next. CSE 598D

15. Partitioning Parameters We can map an application’s partition location (pl) to a file location as follows: We can map a file location to an application's partition location as follows: CSE 598D

16. Partitioning Parameters A typical I/O request consists of a contiguous segment of data from the logical partition of a given length and beginning at a given partition offset. When the IOD processes I/O requests, it is unaware of the partitioning of the file; the IOD is only aware of the striping. Thus, the PVFS library call casts each request in terms of logical file locations. Therefore, the parameters that define an I/O request are as follows (Figure 5): request location (rl) - file location of the start of the request. first size (fs) - size of the starting partial block group size (gs) - the size of each block group count (gc) - the number of whole blocks stride (sd) - the stride of the blocks last size (ls) - the size of the ending partial block CSE 598D

17. I/O Request Processing • When an IOD receives a request: • It converts the request parameters into a job structure which consists of a job header and a list of access structures. • Each access structure is transferred from the stripe segment to the application via a TCP/IP socket. • The library routines also construct a job structure similar to the IODs. This structure is used in much the same manner as on the IOD, to multiplex data received from multiple IODs. CSE 598D

18. Collective Request Support A collective request is identical to other requests, except that request parameters for all application tasks are sent to the IODs at one time by a single master task that is responsible for synchronizing the request for the other application tasks. The job data structure is designed so that the network stream and request type (read or write) can be specified on a per-access basis so that coordinated collective requests can be performed. CSE 598D

19. PVFS Manager and Metadata CSE 598D

20. Metadata CSE 598D

21. Where to Get More Information • http://www.pvfs.org/ • http://www.solarisinternals.com/si/reading/vnode.pdf • http://www.opensolaris.org/os/community/ufs/ • Solaris Internals by Jim Mauro (http://www.amazon.com/Solaris-Internals-Core-Kernel-Architecture/dp/0130224960) CSE 598D