Lecture 15 File Systems

1. Lecture 15 File Systems xlanchen@06/03/2005

2. Contents • Windows 2000 File System Formats • NTFS Design Goals and Features • File System Driver Architecture • NTFS File System Driver • NTFS On-Disk Structure Understanding the Inside of Windows2000

3. Windows 2000 File System Formats • CDFS (CD-ROM File System) • 1988, read-only formatting standard for CD-ROM media • UDF • FAT12, FAT16, and FAT32 • NTFS • the native file system format of Windows 2000 Understanding the Inside of Windows2000

4. FAT series (12, 16, 32) • FAT format organization • Example: file allocation table Understanding the Inside of Windows2000

5. NTFS Design Goals • Recoverability • Security • Data Redundancy • Fault Tolerance Understanding the Inside of Windows2000

6. Multiple data streams Unicode-based names General indexing facility Dynamic bad-cluster remapping Hard links and junctions Compression and sparse files Change logging Per-user volume quotas Link tracking Encryption POSIX support Defragmentation NTFS Features Understanding the Inside of Windows2000

7. NTFS files • NTFS file = set of ($attribute-name, data) pairs • Std attributes include: • FileName, flags, data, MSDOS-name, ACL, other… • Filename up to 255 characters • Values (data) held in MFT entry if possible • Otherwise: • Data attribute = set of (start-VCN, start-LCN, #clusters) • Allows sequence of VCNs to be discovered • Provides VCN->LCN->cluster mapping • “attribute list” attribute added if MFT entry too small • Points to (first) overflow MFT rec for mappings Understanding the Inside of Windows2000

8. NTFS Directories • An index of filenames • Index blocks organized as balanced (b+) tree • Tree pointer gives (VCN,LCN) of next block • Index entries contain: • File Reference Number, plus • Size, timestamp, etc (for directory browsing) (saves reading MFT rec to find file attributes) • NT4 indexes on filename only • NT5 indexes on other file attributes also Understanding the Inside of Windows2000

9. File System Driver (FSD) Architecture • FSDs manage file system formats • Kernel mode • Two different types of FSD (2K) • Local FSDs • Remote FSDs Understanding the Inside of Windows2000

10. Local FSDs • A local FSD must register with the I/O manager • Local FSDs include: • Ntfs.sys, Fastfat.sys, Udfs.sys, Cdfs.sys, and the Raw FSD (integrated in Ntoskrnl.exe). Understanding the Inside of Windows2000

11. Local FSD & other concept of the OS • Boot sector • I/O manager • volume parameter block (VPB) • Storage device  file system device • Cache manager Understanding the Inside of Windows2000

12. Remote FSDs • Remote FSDs consist of two components: • a client and a server • Client-side remote FSD (2K: LANMan Redirector) • allows applications to access remote files and directories • Accepts I/O request & translates into network commands • Server-side FSD (2K: LANMan Server) • Listens and fulfills the command Understanding the Inside of Windows2000

13. Remote FSD operation Understanding the Inside of Windows2000

14. EXPERIMENT • Viewing the List of Registered File Systems Understanding the Inside of Windows2000

15. File system operation • Two ways • Directly, file I/O functions • Indirectly, file mapping • An FSD can be invoked through several paths • Explicit file I/O • From the memory manager's modified page writer • Indirectly from the cache manager's lazy writer • Indirectly from the cache manager's read-ahead thread • From the memory manager's page fault handler Understanding the Inside of Windows2000

16. Components involved in file system I/O Understanding the Inside of Windows2000

17. NTFS FSD • Components of the Windows 2000 I/O system Layered drivers Understanding the Inside of Windows2000

18. NTFS and related components Understanding the Inside of Windows2000

19. Log File Service (LFS) • NTFS provides file system recoverability by means of a transaction-processing technique called logging • LFS is a series of kernel-mode routines inside the NTFS driver Understanding the Inside of Windows2000

20. NTFS data structures Understanding the Inside of Windows2000

21. NTFS On-Disk Structure • Volumes • Clusters • Directories • The storage of actual file data and attribute information • NTFS data compression Understanding the Inside of Windows2000

22. Volumes • A logical partition on a disk • A disk can have one volume or several • stores all file system data as ordinary files, such as • Bitmaps、directories、system bootstrap Understanding the Inside of Windows2000

23. Clusters • Cluster size established when formatting • Also called cluster factor • 2n sectors • Cluster vs. sector • NTFS is independent from physical sector sizes • LCN, logical cluster numbers • the numbering of all clusters of the volume • VCN, virtual cluster numbers • Numberthe clusters belonging to a particular file Understanding the Inside of Windows2000

24. Master File Table (MFT) • All data is contained in files, including metadata • Metadata • the data structures used to locate and retrieve files, • the bootstrap data, • the bitmap that records the allocation state of the entire volume • Easy to locate and maintain • Each can be protected by a security descriptor Understanding the Inside of Windows2000

25. Master File Table (MFT) • MFT, the heart • Implemented as an array of file records • File records, fixed size, 1KB • Logically, contains one record for each file including the MFT itself. • Metadata files (with name prefixed with “$”) • $Mft Understanding the Inside of Windows2000

26. Metadata files in MFT Understanding the Inside of Windows2000

27. Mount a volume with MFT • find the physical disk address of the MFT from the boot sector • Find information inside the file record of MFT • Open more metadata file • Perform the file system recovery operation • Open other metadata file Understanding the Inside of Windows2000

28. Other metadata files • log file ($LogFile) • root directory ("\") • bitmap file ($Bitmap) • security file ($Secure) • boot file ($Boot) • bad-cluster file ($BadClus) • extensions ($Extend), a metadata directory • object identifier file ($ObjId), the quota file ($Quota), the change journal file ($UsnJrnl), and the reparse point file ($Reparse). Understanding the Inside of Windows2000

29. File record vs. File • Normally, 1:1 • May n:1 • If a file has a large number of attributes • or becomes highly fragmented • First one called base file record • stores the locations of the others • Others extended file record Understanding the Inside of Windows2000

30. File Reference Numbers • A file on an NTFS volume is identified a file reference • 64-bit • File number, index to the file's file record position in the MFT • Sequence number, the reused times of an MFT file record position Understanding the Inside of Windows2000

31. File Records • File, a collection of attribute/value pairs • Filename • time stamp information • unnamed data attribute • additional named data attributes Understanding the Inside of Windows2000

32. Understanding the Inside of Windows2000

33. Understanding the Inside of Windows2000

34. Attribute streams • Each file attribute is stored as a separate stream of bytes within a file • Streams is the unit of file operation • create, delete, read and write • Attribute type code • The file attributes in an MFT record are ordered by type codes • Attribute • Type code: value: optional name Understanding the Inside of Windows2000

35. Filenames • NTFS, <=255 characters Understanding the Inside of Windows2000

36. Resident and Nonresident Attributes • resident attribute • the value of an attribute is stored directly in the MFT • Example: Understanding the Inside of Windows2000

37. Several attributes are defined as always being resident • The standard information • Index root attributes Understanding the Inside of Windows2000

38. Resident attribute header and value • Example: filename attribute • Only need once disk accessing Understanding the Inside of Windows2000

39. MFT file record for a small directory • If a particular attribute is too large • Clusters outside MFT is allocated, called run • If the value size grows, more runs is allocated • This is nonresident attributes Understanding the Inside of Windows2000

40. Resident or nonresident • Determined by the file system • Location transparent to the process • Example: • MFT file record for a large file with two data runs Understanding the Inside of Windows2000

41. A large directory can also have nonresident attributes • Example: MFT file record for a large directory with a nonresident filename index Understanding the Inside of Windows2000

42. Keeping track of the runs • VCN-to-LCN mapping pairs • VCN • LCN • Example: • VCNs & LCNs for a nonresident data attribute Understanding the Inside of Windows2000

43. VCN-to-LCN mappings Understanding the Inside of Windows2000

44. Indexing • A file directory is simply an index of filenames • Example: • Filename index for a volume's root directory Understanding the Inside of Windows2000

45. Index root attribute • For large directories: • Index buffer + B+ tree • With index root attribute contains the first level of B+ tree • File index entry: • The file reference in the MFT • Time stamp • File size information Understanding the Inside of Windows2000

46. The index allocation attribute • For index buffer runs: VCNLCN Understanding the Inside of Windows2000

47. Data Compression and Sparse Files • NTFS supports compression • per-file, per-directory, or per-volume • Related func • GetVolumeInformation • GetCompressedFileSize • DeviceIoControl Understanding the Inside of Windows2000

48. Compressing Sparse Data • Sparse data is often large but contains only a small amount of nonzero data relative to its size • Runs of a noncompressed file and the related MFT record Understanding the Inside of Windows2000

49. Compression technique • to remove long strings of zeros from the file • NTFS allocates space only for runs that contain nonzero data • So certain ranges of the file's VCNs have no disk allocations Understanding the Inside of Windows2000

50. example • Runs of a compressed file containing sparse data 16~31 64~127 Understanding the Inside of Windows2000