ecs251 Spring 2011 : Operating System #1: File

1. ecs251 Spring 2011:Operating System#1: File Dr. S. Felix Wu Computer Science Department University of California, Davis http://www.facebook.com/group.php?gid=29670204725 http://cyrus.cs.ucdavis.edu/~wu/ecs251 ecs251, spring 2011

2. About the Instructor • S. Felix Wu • Facebook group: ecs251 • wu@cs.ucdavis.edu (but, please use SoEmail) • Office: 2109 Watershed • Phone: 530-754-7070 • Office Hours: • 1-2 p.m. on Tuesday and Friday • by appointment ecs251, spring 2011

3. ecs251, spring 2011

4. As we don’t have an TA • Please utilize the Facebook discussion board as much as possible. • Some of my graduate students will help! ecs251, spring 2011

5. ecs251, spring 2011

6. This quarter…. • File: from the basic to the “most” future advance • OS architecture: kernel, services, VM, distributed, timing, commitment/consensus, recovery, real-time, P2P, cloud • Advanced topics: social informatics and computing, mobility/geography, information search ecs251, spring 2011

7. Textbook Seven papers have been assigned/announced on the FB group. ecs251, spring 2011

8. Prerequisites • Programming Languages: C and assembly • Some undergraduate training in OSes • E.g., ecs150 • Please talk to me if you have any concern. ecs251, spring 2011

9. Course Requirements • 30%: Homework Assignments • 3~5 Assignments • 30%: Midterm (around early May) • 40%: Final Project • Teamwork: 1~3 students • Topics: We will discuss this later. • Final report or demo (slides) ecs251, spring 2011

10. File  Disk • separate the disk into blocks • separate the file into blocks as well • “paging” from file to disk blocks: 4 - 7- 2- 10- 12 How to represent the file?? How to link these 5 pages together?? ecs251, spring 2011

11. Bit Torrent pieces • 1 big file (X Gigabytes) with a number of pieces (5%) already in (and sharing with others). • How much disk space do we need at this moment? ecs251, spring 2011

12. One Logical File  Physical Disk Blocks efficient representation & access File - Pieces/Blocks Free Pieces/Blocks ecs251, spring 2011

13. File  Disk blocks 0 file block 0 file block 1 file block 2 file block 3 file block 4 4 7 2 10 12 • What are the disadvantages? • disk access can be slow for “random access”. • How big is each block? 2^X bytes? 2^X+8 bytes? ecs251, spring 2011

14. One Logical File  Physical Disk Blocks efficient representation & access File - Pieces/Blocks Free Pieces/Blocks ecs251, spring 2011

15. An i-node A file ??? entries in one disk block ecs251, spring 2011

16. 125 struct ufs2_dinode { 126 u_int16_t di_mode; /* 0: IFMT, permissions; see below. */ 127 int16_t di_nlink; /* 2: File link count. */ 128 u_int32_t di_uid; /* 4: File owner. */ 129 u_int32_t di_gid; /* 8: File group. */ 130 u_int32_t di_blksize; /* 12: Inode blocksize. */ 131 u_int64_t di_size; /* 16: File byte count. */ 132 u_int64_t di_blocks; /* 24: Bytes actually held. */ 133 ufs_time_t di_atime; /* 32: Last access time. */ 134 ufs_time_t di_mtime; /* 40: Last modified time. */ 135 ufs_time_t di_ctime; /* 48: Last inode change time. */ 136 ufs_time_t di_birthtime; /* 56: Inode creation time. */ 137 int32_t di_mtimensec; /* 64: Last modified time. */ 138 int32_t di_atimensec; /* 68: Last access time. */ 139 int32_t di_ctimensec; /* 72: Last inode change time. */ 140 int32_t di_birthnsec; /* 76: Inode creation time. */ 141 int32_t di_gen; /* 80: Generation number. */ 142 u_int32_t di_kernflags; /* 84: Kernel flags. */ 143 u_int32_t di_flags; /* 88: Status flags (chflags). */ 144 int32_t di_extsize; /* 92: External attributes block. */ 145 ufs2_daddr_t di_extb[NXADDR];/* 96: External attributes block. */ 146 ufs2_daddr_t di_db[NDADDR]; /* 112: Direct disk blocks. */ 147 ufs2_daddr_t di_ib[NIADDR]; /* 208: Indirect disk blocks. */ 148 int64_t di_spare[3]; /* 232: Reserved; currently unused */ 149 }; ecs251, spring 2011

17. 166 struct ufs1_dinode { 167 u_int16_t di_mode; /* 0: IFMT, permissions; see below. */ 168 int16_t di_nlink; /* 2: File link count. */ 169 union { 170 u_int16_t oldids[2]; /* 4: Ffs: old user and group ids. */ 171 } di_u; 172 u_int64_t di_size; /* 8: File byte count. */ 173 int32_t di_atime; /* 16: Last access time. */ 174 int32_t di_atimensec; /* 20: Last access time. */ 175 int32_t di_mtime; /* 24: Last modified time. */ 176 int32_t di_mtimensec; /* 28: Last modified time. */ 177 int32_t di_ctime; /* 32: Last inode change time. */ 178 int32_t di_ctimensec; /* 36: Last inode change time. */ 179 ufs1_daddr_t di_db[NDADDR]; /* 40: Direct disk blocks. */ 180 ufs1_daddr_t di_ib[NIADDR]; /* 88: Indirect disk blocks. */ 181 u_int32_t di_flags; /* 100: Status flags (chflags). */ 182 int32_t di_blocks; /* 104: Blocks actually held. */ 183 int32_t di_gen; /* 108: Generation number. */ 184 u_int32_t di_uid; /* 112: File owner. */ 185 u_int32_t di_gid; /* 116: File group. */ 186 int32_t di_spare[2]; /* 120: Reserved; currently unused */ 187 }; ecs251, spring 2011

18. #include <stdio.h> #include <stdlib.h> int main (void) { FILE *f1 = fopen("./sss.txt", "w"); int i; for (i = 0; i < 1000; i++) { fseek(f1, rand(), SEEK_SET); fprintf(f1, "%d%d%d%d", rand(), rand(), rand(), rand()); if (i % 100 == 0) sleep(1); } fflush(f1); } # ./t # ls –l ./sss.txt ecs251, spring 2011

19. ecs251, spring 2011

20. ecs251, spring 2011

21. ecs251, spring 2011

22. di_size vs. di_blocks • Logical • Physical • fstat • du ecs251, spring 2011

23. One Logical File  Physical Disk Blocks efficient representation & access ecs251, spring 2011

24. An i-node A file ??? entries in one disk block Typical: each block 1K ecs251, spring 2011

25. An i-node A file ??? entries in one disk block Typical: each block 1K ecs251, spring 2011

26. i-node • How many disk blocks can a FS have? • How many levels of i-node indirection will be necessary to store a file of 2G bytes? (I.e., 0, 1, 2 or 3) • What is the largest possible file size in i-node? • What is the size of the i-node itself for a file of 10GB with only 512 MB downloaded? ecs251, spring 2011

27. Answers • How many disk blocks can a FS have? • 264 or 232: Pointer (to blocks) size is 8/4 bytes. • How many levels of i-node indirection will be necessary to store a file of 2G (231) bytes? (I.e., 0, 1, 2 or 3) • 12*210 + 28 * 210 + 28 *28 *2 10 +28 *28 *28 *2 10 >? 231 • What is the largest possible file size in i-node? • 12*210 + 28 * 210 + 28 *28 *2 10 +28 *28 *28 *2 10 • 264 –1 • 232 * 210 You need to consider three issues and find the minimum! ecs251, spring 2011

28. i-node • How many disk blocks can a FS have? • How many levels of i-node indirection will be necessary to store a file of 2G bytes? (I.e., 0, 1, 2 or 3) • What is the largest possible file size in i-node? • What is the size of the i-node itself for a file of 10GB with only 512 MB downloaded? ecs251, spring 2011

29. A File System partition partition partition b s i-list directory and data blocks d i-node i-node ……. i-node ecs251, spring 2011

30. dirp = opendir(const char *filename); struct dirent *direntp = readdir(dirp); struct dirent { ino_t d_ino; char d_name[NAME_MAX+1]; }; directory dirent inode file_name dirent inode file_name dirent inode file_name file file file ecs251, spring 2011

31. ecs251, spring 2011

32. root wheel . 2 directory / 2 drwxr-xr-x .. 2 Apr 1 2004 usr 4 3 vmunix 5 root wheel . 4 drwxr-xr-x directory /usr .. 2 4 Apr 1 2004 bin 7 root wheel foo 6 rwxr-xr-x 5 file /vmunix Apr 15 2004 text data kirk staff 6 rw-rw-r-- file /usr/foo Hello World! Jan 19 2004 root wheel . 7 7 drwxr-xr-x .. 4 directory /usr/bin Apr 1 2004 ex 9 8 groff 10 bin bin vi 9 file /usr/bin/vi 9 rwxr-xr-x text data Apr 15 2004 ecs251, spring 2011

33. struct dirent { ino_t d_ino; char d_name[NAME_MAX+1]; }; struct stat {… short nlinks; …}; directory dirent inode file_name dirent inode file_name dirent inode file_name file file file ecs251, spring 2011

34. A File System partition partition partition b s i-list directory and data blocks d i-node i-node ……. i-node ecs251, spring 2011

35. What is the difference? • ln –s /usr/src/sys/sys/proc.h ppp.h • ln /usr/src/sys/sys/proc.h ppp.h ecs251, spring 2011

36. /usr/src/sys/ufs/ffs/ffs_balloc.c Buffer pages to Disk/I-Node ecs251, spring 2011

37. Snapshot of the FS • backup and restore • dump reliably an active File System • what will we do today to dump our 40GB FS “consistent” snapshots? (in the midnight…) • “background FSCK checks” ecs251, spring 2011

38. What is a snapshot?(I mean “conceptually”.) • Freeze all activities related to the FS. • Copy everything to “some space”. • Resume the activities. How do we efficiently implement this concept such that the activities will only be blocked for about 0.25 seconds, and we don’t have to buy a really big hard drive? ecs251, spring 2011

39. ecs251, spring 2011

40. ecs251, spring 2011

41. Copy-on-Write ecs251, spring 2011

42. Snapshot: a file Logical size Versus physical size ecs251, spring 2011

43. Example # mkdir /backups/usr/noon # mount –u –o snapshot /usr/snap.noon /usr # mdconfig –a –t vnode –u 0 –f /usr/snap.noon # mount –r /dev/md0 /backups/usr/noon /* do whatever you want to test it */ # umount /backups/usr/noon # mdconfig –d –u 0 # rm –f /usr/snap.noon ecs251, spring 2011

44. ecs251, spring 2011

45. ecs251, spring 2011

46. #include <stdio.h> #include <stdlib.h> int main (void) { FILE *f1 = fopen("./sss.txt", "w"); int i; for (i = 0; i < 1000; i++) { fseek(f1, rand(), SEEK_SET); fprintf(f1, "%d%d%d%d", rand(), rand(), rand(), rand()); if (i % 100 == 0) sleep(1); } fflush(f1); } ecs251, spring 2011

47. Example # mkdir /backups/usr/noon # mount –u –o snapshot /usr/snap.noon /usr # mdconfig –a –t vnode –u 0 –f /usr/snap.noon # mount –r /dev/md0 /backups/usr/noon /* do whatever you want to test it */ # umount /backups/usr/noon # mdconfig –d –u 0 # rm –f /usr/snap.noon ecs251, spring 2011

48. ecs251, spring 2011

49. ecs251, spring 2011

50. ecs251, spring 2011