Linux Virtual File System

Linux Virtual File System Robert Ledford Leif Wickland CS518 Fall 2004 I/O, I/O, It's off to disk I go-o-o, A bit or byte to read or write, I/O, I/O, I/O...

Overview • What is a file system? • Historical view of file systems • Another layer of indirection • Do I have to? • File systems have layers • How is it done? • Sign me up

What is a file system? • Speaking broadly, a file system is the logical means for an operating system to store and retrieve data on the computers hard disks, be they local drives, network-available volumes, or exported shares in a storage area network (SAN)

What is a file system? • There is some ambiguity in the term “file system”. The term can be used to mean any of the following: - The type of a file system refers to a specific implementation such as ext2, reiserfs or nfs, each implementation contains the methods and data structures that an operating system uses to keep track of files on a disk or partition

What is a file system? - An instance of a file system refers to a file system type residing at a location such as /dev/hda4 - Additionally a file system can refer to the methods and data structures that an operating system uses to keep track of files on a disk or partition

What is a file system? • Linux keeps regular files and directories on block devices such as disks • A Linux installation may have several physical disk units, each containing one or more file system types • Partitioning a disk into several file system instances makes it easier for administrators to manage the data stored there

What is a file system? Overhead view sector track cylinder The same track on each platter in a disk makes a cylinder; partitions are groups of contiguous cylinders Disk blocks are composed of one or more contiguous sectors

What is a file system? • Why have multiple partitions? • Encapsulate your data: - Since file system corruption is local to a partition, you stand to lose only some of your data if an accident occurs

What is a file system? • Increase disk space efficiency: - You can format partitions with varying block sizes, depending on your usage - If your data is in a large number of small files (less than 1k) and your partition uses 4k sized blocks, you are wasting 3k for every file - In general, you waste on average one half of a block for every file, so matching block size to the average size of your files is important if you have many files

What is a file system? • Limit data growth: - Runaway processes or maniacal users can consume so much disk space that the operating system no longer has room on the hard drive for its bookkeeping operations - This can lead to disaster. By segregating space, you ensure that things other than the operating system die when allocated disk space is exhausted

What is a file system? • Partitioning tools and utilities: - fdisk rledford@leonard > sudo fdisk -l /dev/had Disk /dev/hda: 255 heads, 63 sectors, 4863 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 26 208813+ 83 Linux /dev/hda2 27 1070 8385930 83 Linux /dev/hda3 1071 1853 6289447+ 83 Linux /dev/hda4 1854 4863 24177825 f Win95 Ext'd (LBA) /dev/hda5 1854 2375 4192933+ 83 Linux /dev/hda6 2376 2897 4192933+ 83 Linux /dev/hda7 2898 3028 1052226 82 Linux swap /dev/hda8 3029 4863 14739606 83 Linux - parted: GNU partition editor

What is a file system? • Review Questions - Where does Linux keep regular file types? • On block devices such as disks. - On average, how much of a block is wasted for every file? • On average ½ of a block is wasted for every file.

What is a file system? • Review: • File system instances reside on partitions • Partitioning is a means to divide a single hard drive into many logical drives • A partition is a contiguous set of blocks on a drive that are treated as an independent disk • A partition table is an index that relates sections of the hard drive to partitions

What is a file system? Entire Disk Master Boot Record Partition Table Disk Partitions Boot Block Super Block Inode List Data Blocks A Possible File System Instance Layout

What is a file system? What is a file system? • The central structural concepts of a file system type are: - Boot Block - Super Block - Inode List - Data Block Boot Block Super Block Inode List Data Blocks

What is a file system? • Boot Block: - Occupies the beginning of a file system - Typically residing at the first sector, it may also contain the bootstrap code that is read into the machine at boot time - Although only one boot block is required to boot the system, every file system may contain a boot block Boot Block

What is a file system? • Super Block: - Describes the state of a file system - How large it is - How many files it can store - Where to find free space in the file system - Additional data that assists the file management system with operating on the file system Boot Block Super Block

What is a file system? • Super Block: - Duplicate copies of the super block may reside through out the file system in case the super block is corrupted Boot Block Super Block

What is a file system? • Inode List: - An inode is the internal representation of a file contains the description of the disk layout of the file data - file owner - permissions - The inode list contains all of the inodes present in an instance of a file system Boot Block Super Block Inode List

What is a file system? • Data Blocks: - Contain the file data in the file system - Additional administrative data - An allocated data block can belong to one and only one file in the file system Boot Block Super Block Inode List Data Blocks

What is a file system? • On a Linux system, a user or user program sees a single file hierarchy, rooted at / • Every file and directory can trace its origins on a tree back to the root directory

What is a file system? / bin boot dev etc home lib mnt proc root sbin tmp usr var

What is a file system? • Review Questions - Where is the boot strap code located • In the Boot Block. - What contains information about a file’s owner and the file’s permissions? • The inode. - What is the index that relates sections of the hard drive to partitions? • The Partition Table.

What is a file system? • A file system implements the basic operations to manipulate files and directories • These basic operations include: - Opening of files - Closing of files - Creation of directories - Listing of the contents of directories - Removal of files from a directory

What is a file system? • The kernel deals on a logical level with file systems rather than with disks • The separate file systems that the system may use are not accessed by device identifiers • Instead they are combined into a single hierarchical tree structure that represents the file systems as one whole single entity

What is a file system? • So what is a file system? - A file system is a set of abstract data types that are implemented for the storage, hierarchical organization, manipulation, navigation, access, and retrieval of data

Overview • What is a file system? • Historical view of file systems • File systems have layers • Do I have to? • Another layer of indirection • How is it done? • Sign me up

Historical view of file systems • Managing storage has long been a prominent role for operating systems. • This role was so important to the MS-DOS OS that it was named after that function. • DOS stands for Disk Operating System. • Created in 1980.

Historical view of file systems • The file system hasn’t changed much since the 1960s. • A research paper was presented in 1965 describing “A General-Purpose File System For Secondary Storage.” • Laid out the notion of a hierarchal file system much as is used today.

Historical view of file systems • File system features in 1965 paper • Files • Directories • Links • Access permissions • Create, access and modify times • Path nomenclature: Directory:directory:file • All devices mount into a unified hierarchy • Backing up • Everybody knew it was a good idea back then, too.

Historical view of file systems • This type of file system was implemented in Multics. • Unix was created as an “emasculated” version of Multics. • Project started as gaming system in 1969. • The designers of Unix had worked on Multics and brought to Unix a Multics-style file system.

chdir: change directory chmod: change permission chown: change owner chroot: change root close: close a file creat: create a file dup: copy file descriptor link: add a file reference lseek: set open file cursor mknod: make a special file mount: graft in a file system open: open a file system pipe: create a pipe read: read from a file stat: get file status umount: opposite of mount unlink: delete file write: write to a file Historical view of file systems File System API in Unix System V, c. 1983

Historical view of file systems • Tannenbaum wrote Minix, a pedantic version of Unix, in 1987. • Of course, it included a Unix-style file API.

Historical view of file systems • Linus Torvalds introduced Linux in 1991. • He developed Linux on Minix. • Consequently it was convenient for the OSes to share a file system and file API. • Thus, Linux inherited the same style file system as presented in the 1965 paper. • Today Linux supports a superset of the file system features available in Unix System V.

Historical view of file systems • Review Questions • In what year was the paper released that described the file system design that is the ancestor of Linux’ file system? • 1965 • Yes, that’s about 40 years ago

Another layer of indirection • Multics, Unix, Minux, and Linux originally supported only one file system. • They only understood one type of layout on disk for directories and files. • Because of its origins, Linux initially supported just the Minix file system. • Limited to small partitions and short filenames • However, it wasn’t long before people wanted more from their file systems

Another layer of indirection User Program • The problem: • Linux was implemented like this. Minix FS Interface Hard Drive • To add support for another file system in a similar manner was unsavory and didn’t scale. • User program must call a separate API for each type of file system. User Program Other FS Minix FS Hard Drive A Hard Drive B

Another layer of indirection • “Any problem in computer science can be solved with another layer of indirection.” • David Wheeler (chief programmer for the EDSAC project in the early 1950s)

Another layer of indirection • The solution was to add a layer of indirection to the file system stack. • In Linux this layer is called the virtual file system (VFS). • User programs access any file system through a consistent API. • All File Systems implement an API which is called by the VFS. User Program Virtual File System Other FS Minix FS Hard Drive A Hard Drive B

Another layer of indirection • The VFS is • Another layer of indirection • A file system- and device-agnostic layer of the operating system • A consistent API for user applications to access storage independent of the underlying device or type of file system

Task 1 Task 2 Task n … user space kernel space VIRTUAL FILE SYSTEM minix ext2 msdos proc Buffer Cache device driver for hard disk device driver for floppy disk Linux Kernel software hardware Hard Disk Floppy Disk Robbed from http://www.cs.usfca.edu/~cruse/cs326/lesson22.ppt

Do I have to? • Is a VFS worth doing? • What do you think?

Cons Harms system performance Another layer of indirection Adds to the size of the system because more code must be written A conceptually simpler system Pros Enables using multiple file systems Facilitates research Makes the computer more useful My Linux box has ext2, ext3, FAT32, and NTFS partitions mounted Facilitates code reuse Simplifies implementation Do I have to?

Do I have to? • Is a VFS worth doing? • Ultimately, the answer is yes for general purpose operating systems. • All modern commercial operating systems do. • What would you do if you had to design an OS’ file system? Would you use a VFS?

File systems have layers • Like onions and ogres… • In this context “file system” means the software stack that extends from the user application to the hardware.

File systems have layers • The file system of Unix System V has three layers • File/Directory API • Inodes • Buffers User Program File/Directory API Inodes File system Buffers Storage Device

Linux Virtual File System