e cs251 Spring 2014 : Operating System #1: Syllabus + File Systems

1. ecs251 Spring 2014:Operating System#1: Syllabus + File Systems Dr. S. Felix Wu Computer Science Department University of California, Davis http://www.facebook.com/groups/ecs251/ http://cyrus.cs.ucdavis.edu/~wu/ecs251 ecs251, spring 2014

2. About the Instructor • S. Felix Wu • Facebook group: ecs251 • wu@cs.ucdavis.edu • Office: 2109 Watershed • Phone: 530-754-7070 • Office Hours: • 3-4 p.m. on Monday and Friday • by appointment ecs251, spring 2014

3. ecs251, spring 2014

4. Spring 2014 • File systems: UFS/FFS, NFS/AFS/Coda, HDFS • Distributed Systems: Map-Reduce, P2P, Mobility, Virtualization • Internet Computing: Search Engine, OSN ecs251, spring 2014

5. Textbook Seven papers have been assigned/announced on the FB group. BTW, this will NOT be a FreeBSD course, and if you want to learn all the details about OS kernel, you should take ecs150 in spring 2014 (or watch my lectures online) ecs251, spring 2014

6. Prerequisites • Programming Languages: C and assembly • Some undergraduate training in OSes • E.g., ecs150 • Please talk to me if you have any concern. ecs251, spring 2014

7. Course Requirements • 25%: Exam • midterm (format/time/place, TBA) • 75%: Final Research Project/Paper • Teamwork: 2~3 students • Topics: your choice with instructor’s approval • Final report, slides and demo ecs251, spring 2014

8. Project, Project, Project!!! • Leveraging your own research is highly encouraged! • Ambitiously, I like to push you to aim something “big/risky, initially clueless” • My first question to you in the final demo or presentation is “why is it interesting?” • Well, I will also ask you earlier than that. • My evaluation is based on the “dream-level”, not necessarily the “completeness”. ecs251, spring 2014

9. Operating Systems ecs251, spring 2014

10. ….where applications meet Hardware!!! Applications…….. OS Hardware: CPUs/Memory/HDs/DVD/Wireless/Internet… ecs251, spring 2014

11. VM/MVS, DOS, Win95/98/ME/2000/XP, Freebsd/Linux, MacOS-10, Mach, Minix, PalmOS, uCOS, TinyOS, … ecs251, spring 2014

12. ecs251, spring 2014

13. ecs251, spring 2014

14. OS OS ecs251, spring 2014

15. OS OS TCP/IP, VPN, SSL/TLS/SSH ecs251, spring 2014

16. Realtime Hadoop Cluster Scribe MidTier Web Servers Scribe Writers MySQL Oracle RAC Hadoop Hive Warehouse http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html ecs251, spring 2014 10/6/2014

17. OS OS OS TCP/IP, VPN, SSL/TLS/SSH ecs251, spring 2014

18. FB Friendship OS OS OS TCP/IP, VPN, SSL/TLS/SSH ecs251, spring 2014

19. 3rd Party Applications & Services (e.g. SINCERE.se) SIN API Social-aware OS Kernel Social Informatics CITRON crawler Application Plugins GENI API ecs251, spring 2014

20. ….where applications meet Hardware!!! Applications…….. OS Hardware: CPU/Memory/HD/DVD/Wireless… ecs251, spring 2014

21. ….where applications meet Hardware!!! Applications…….. OS Hardware: CPU/Memory/HD/DVD/Wireless… Multi-cores/Racks, Energy, Mobility, Social Informatics ecs251, spring 2014

22. Possibilities • Theory/Formalism • The system is so complicated, then how to formalize it (beyond, say, queuing theory)? • System, Experiment, Evaluation, or Visualization • A novel idea (how to realize), • maybe an idea from others (how to extend or how to evaluate), • large-scale experiments (on GENI), or • information visualization for OSs ecs251, spring 2014

23. Again, Discussion!!! • Please do so on the Facebook group so we can all help each other to figure out what we need to pursue for our final project. ecs251, spring 2014

24. ecs251, spring 2014

25. Homeworks for ecs150 • Adding new system calls (KLD) • Process ID, FB/Social ID, scheduling mode • Social-aware Lottery Scheduling • Intercepting File System calls • Also, IP address and FB identifiers • Enhancing Bit Torrent (Interactive Grading) ecs251, spring 2014

26. File System, the basic • Overlap with ecs150 • The key part of the master exam for OS • i.e., if you will take it from me. • FYI, I will also add a touch of social computing from time to time… ecs251, spring 2014

27. File  Disk • separate the disk into blocks • separate the file into blocks as well • “paging” from file to disk blocks: 4 - 7- 2- 10- 12 How to represent the file?? How to link these 5 pages together?? ecs251, spring 2014

28. Bit Torrent pieces • 1 big file (X Gigabytes) with a number of pieces (5%) already in (and sharing with others). • How much disk space do we need at this moment? ecs251, spring 2014

29. One Logical File  Physical Disk Blocks efficient representation & access File - Pieces/Blocks Free Pieces/Blocks ecs251, spring 2014

30. File  Disk blocks 0 file block 0 file block 1 file block 2 file block 3 file block 4 4 7 2 10 12 • What are the disadvantages? • disk access can be slow for “random access”. • How big is each block? 2^X bytes? 2^X+8 bytes? ecs251, spring 2014

31. One Logical File  Physical Disk Blocks efficient representation & access File - Pieces/Blocks Free Pieces/Blocks ecs251, spring 2014

32. ecs251, spring 2014

33. An i-node A file ??? entries in one disk block ecs251, spring 2014

34. Requirements • performance/efficiency • Robustness/recovery • Extensibility/Flexibility ecs251, spring 2014

35. 125 struct ufs2_dinode { 126 u_int16_t di_mode; /* 0: IFMT, permissions; see below. */ 127 int16_t di_nlink; /* 2: File link count. */ 128 u_int32_t di_uid; /* 4: File owner. */ 129 u_int32_t di_gid; /* 8: File group. */ 130 u_int32_t di_blksize; /* 12: Inode blocksize. */ 131 u_int64_t di_size; /* 16: File byte count. */ 132 u_int64_t di_blocks; /* 24: Bytes actually held. */ 133 ufs_time_t di_atime; /* 32: Last access time. */ 134 ufs_time_t di_mtime; /* 40: Last modified time. */ 135 ufs_time_t di_ctime; /* 48: Last inode change time. */ 136 ufs_time_t di_birthtime; /* 56: Inode creation time. */ 137 int32_t di_mtimensec; /* 64: Last modified time. */ 138 int32_t di_atimensec; /* 68: Last access time. */ 139 int32_t di_ctimensec; /* 72: Last inode change time. */ 140 int32_t di_birthnsec; /* 76: Inode creation time. */ 141 int32_t di_gen; /* 80: Generation number. */ 142 u_int32_t di_kernflags; /* 84: Kernel flags. */ 143 u_int32_t di_flags; /* 88: Status flags (chflags). */ 144 int32_t di_extsize; /* 92: External attributes block. */ 145 ufs2_daddr_t di_extb[NXADDR];/* 96: External attributes block. */ 146 ufs2_daddr_t di_db[NDADDR]; /* 112: Direct disk blocks. */ 147 ufs2_daddr_t di_ib[NIADDR]; /* 208: Indirect disk blocks. */ 148 int64_t di_spare[3]; /* 232: Reserved; currently unused */ 149 }; ecs251, spring 2014

36. 125 struct ufs2_dinode { 126 u_int16_t di_mode; /* 0: IFMT, permissions; see below. */ 127 int16_tdi_nlink; /* 2: File link count. */ 128 u_int32_t di_uid; /* 4: File owner. */ 129 u_int32_t di_gid; /* 8: File group. */ 130 u_int32_t di_blksize; /* 12: Inode blocksize. */ 131 u_int64_tdi_size; /* 16: File byte count. */ 132 u_int64_tdi_blocks; /* 24: Bytes actually held. */ 133 ufs_time_t di_atime; /* 32: Last access time. */ 134 ufs_time_t di_mtime; /* 40: Last modified time. */ 135 ufs_time_t di_ctime; /* 48: Last inode change time. */ 136 ufs_time_t di_birthtime; /* 56: Inode creation time. */ 137 int32_t di_mtimensec; /* 64: Last modified time. */ 138 int32_t di_atimensec; /* 68: Last access time. */ 139 int32_t di_ctimensec; /* 72: Last inode change time. */ 140 int32_t di_birthnsec; /* 76: Inode creation time. */ 141 int32_t di_gen; /* 80: Generation number. */ 142 u_int32_t di_kernflags; /* 84: Kernel flags. */ 143 u_int32_t di_flags; /* 88: Status flags (chflags). */ 144 int32_t di_extsize; /* 92: External attributes block. */ 145 ufs2_daddr_t di_extb[NXADDR];/* 96: External attributes block. */ 146 ufs2_daddr_tdi_db[NDADDR]; /* 112: Direct disk blocks. */ 147 ufs2_daddr_tdi_ib[NIADDR]; /* 208: Indirect disk blocks. */ 148 int64_t di_spare[3]; /* 232: Reserved; currently unused */ 149 }; ecs251, spring 2014

37. 166 struct ufs1_dinode { 167 u_int16_t di_mode; /* 0: IFMT, permissions; see below. */ 168 int16_t di_nlink; /* 2: File link count. */ 169 union { 170 u_int16_t oldids[2]; /* 4: Ffs: old user and group ids. */ 171 } di_u; 172 u_int64_t di_size; /* 8: File byte count. */ 173 int32_t di_atime; /* 16: Last access time. */ 174 int32_t di_atimensec; /* 20: Last access time. */ 175 int32_t di_mtime; /* 24: Last modified time. */ 176 int32_t di_mtimensec; /* 28: Last modified time. */ 177 int32_t di_ctime; /* 32: Last inode change time. */ 178 int32_t di_ctimensec; /* 36: Last inode change time. */ 179 ufs1_daddr_t di_db[NDADDR]; /* 40: Direct disk blocks. */ 180 ufs1_daddr_t di_ib[NIADDR]; /* 88: Indirect disk blocks. */ 181 u_int32_t di_flags; /* 100: Status flags (chflags). */ 182 int32_t di_blocks; /* 104: Blocks actually held. */ 183 int32_t di_gen; /* 108: Generation number. */ 184 u_int32_t di_uid; /* 112: File owner. */ 185 u_int32_t di_gid; /* 116: File group. */ 186 int32_t di_spare[2]; /* 120: Reserved; currently unused */ 187 }; ecs251, spring 2014

38. #include <stdio.h> #include <stdlib.h> int main (void) { FILE *f1 = fopen("./sss.txt", "w"); int i; for (i = 0; i < 1000; i++) { fseek(f1, rand(), SEEK_SET); fprintf(f1, "%d%d%d%d", rand(), rand(), rand(), rand()); if (i % 100 == 0) sleep(1); } fflush(f1); } # ./t # ls –l ./sss.txt ecs251, spring 2014

39. ecs251, spring 2014

40. ecs251, spring 2014

41. ecs251, spring 2014

42. di_size vs. di_blocks • Logical • Physical • fstat • du ecs251, spring 2014

43. ecs251, spring 2014

44. ecs251, spring 2014

45. ecs251, spring 2014

46. Try on different platforms ecs251, spring 2014

47. One Logical File  Physical Disk Blocks efficient representation & access ecs251, spring 2014

48. An i-node A file ??? entries in one disk block Typical: each block 1K ecs251, spring 2014

49. An i-node A file ??? entries in one disk block Typical: each block 1K ecs251, spring 2014

50. Dirent/iNode Directory entries – naming the files (under a hierarchy) iNode – pointers to the content (blocks) But, nevertheless, it (iNode) provides a “sequential” list of pointers to every single block. The order of “access” (to the whole file) is still under the control of the user. ecs251, spring 2014