Virtual Memory, File-System Interface

1. Virtual Memory, File-System Interface

2. Background • Virtual memory – separation of user logical memory from physical memory. • Only part of the program needs to be in memory for execution • Logical address space can therefore be much larger than physical address space • Allows address spaces to be shared by several processes • Allows for more efficient process creation • Virtual memory can be implemented via Demand paging

3. Virtual Memory Larger Than Physical Memory 

4. Demand Paging • Bring a page into memory only when it is needed • Less I/O needed • Less memory needed • Faster response • More users • Page is needed  reference to it • invalid reference  abort • not-in-memory  bring to memory • Lazy swapper – never swaps a page into memory unless page will be needed • Swapper that deals with pages is a pager

5. Page Table When Some Pages Are Not in Main Memory

6. Handling a Page Fault

7. Process Creation • Virtual memory allows other benefits • Copy-on-Write (COW): more efficient process creation • allows both parent and child processes to initially share same pages in memory • If either process modifies a shared page, only then is the page copied

8. What happens if there is no free frame? • Page replacement – find some page in memory, but not really in use, swap it out • Goal –minimize number of page faults • Only modified pages are written to disk to reduce overhead of page transfers

9. Basic Page Replacement • Find the location of the desired page on disk • Find a free frame: - If there is a free frame, use it - If there is no free frame, use a page replacement algorithm to select a victim frame • Bring desired page into the (newly) free frame; update the page and frame tables • Resume the process

10. Page Replacement Algorithms • Want lowest page-fault rate • Evaluate algorithm by running it on a particular string of memory references (reference string) and computing the number of page faults on that string • In all our examples, the reference string is 1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5

11. First-In-First-Out (FIFO) Algorithm • Reference string: 1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5 • 3 frames (3 pages can be in memory at a time per process) • 4 framesBelady’s Anomaly: more frames  more page faults 1 1 4 5 2 2 1 3 9 page faults 3 3 2 4 1 1 5 4 2 2 1 10 page faults 5 3 3 2 4 4 3

12. Optimal Page Replacement • Replace page that will not be used for longest period of time • Used for measuring how well your algorithm performs

13. Least Recently Used (LRU) Page Replacement • Every page entry has a counter; every time page is referenced through this entry, copy the clock into the counter • When a page needs to be changed, look at the counters to determine which to change

14. LRU Algorithm (Cont.) • Stack implementation – keep a stack of page numbers in a double link form: • Page referenced: • move it to the top • requires 6 pointers to be changed

15. Use Stack to Record The Most Recent Page References • keep a stack of page numbers in a double link form: • Page referenced move it to the top, requires 6 pointers to be changed

16. Counting Algorithms • Keep a counter of the number of references that have been made to each page • Least Frequently Used (LFU) Algorithm: replaces page with smallest count • Most Frequently Used (MFU) Algorithm: based on the argument that the page with the smallest count was probably just brought in and has yet to be used

17. Frame Allocation • Equal allocation – For example, if there are 100 frames and 5 processes, give each process 20 frames. • Proportional allocation – Allocate according to the size of process

18. Thrashing • If a process does not have “enough” pages, the page-fault rate is very high. • Thrashing a process is busy swapping pages in and out • Demand paging works because of locality model • Process migrates from one locality to another • Localities may overlap • Why does thrashing occur? size of locality > total memory size

19. Working-Set Model •   working-set window  a fixed number of page references Example: 10,000 instruction • WSSi (working set of Process Pi) =total number of pages referenced in the most recent  (varies in time) • if  too small will not encompass entire locality • if  too large will encompass several localities • if  =   will encompass entire program • D =  WSSi  total demand frames • if D > m  Thrashing • Policy if D > m, then suspend one of the processes

20. Move on to File System • To explain the function of file systems • To describe the interfaces to file systems • To explore file-system protection

21. File Concept • A named collection of related information that is stored on secondary storage • The smallest allotment of secondary storage • A sequence of bits, bytes, lines or records… • Types: • Data • numeric • character • binary • Program

22. File Structure • None - sequence of words, bytes • Simple record structure • Lines • Fixed length • Variable length • Complex Structures • Formatted document • Relocatable load file: executable files, library files • Indexed file: for fast access to data • Can simulate last two with first method by inserting appropriate control characters

23. Example of Index and Relative Files 23

24. File Attributes • Name – only information kept in human-readable form • Identifier – unique tag (number) identifies file within file system • Type – needed for systems that support different types • Location – pointer to file location on device • Size – current file size • Protection – controls who can do reading, writing, executing • Time, date, and user identification– for creation/last modification/access, used for protection, security, and usage monitoring • Information about files are kept in directory structure, which is maintained on the disk

25. File Operations • File is an abstract data type with operations such as: • Create • Write • Read • Reposition within file • Delete • Truncate • Open(Fi) – search the directory structure on disk for entry Fi, and move the content of entry to memory • Close (Fi) – move the content of entry Fi in memory to directory structure on disk

26. Open Files • Several pieces of data are needed to manage open files: • File pointer: pointer to last read/write location, per process that has the file open • File-open count: counter of number of times a file is open – to allow removal of data from open-file table when last processes closes it • Disk location of the file: cache of data access information • Access rights: per-process access mode information

27. Open File Locking • Provided by some operating systems and file systems • Mediates access to a file • Mandatory or advisory: • Mandatory – access is denied depending on locks held and requested • Advisory – processes can find status of locks and decide what to do

28. File Types – Name, Extension

29. Access Methods • Sequential Access read next write next reset no read after last write (rewrite)

30. Simulation of Sequential Access on Direct-access File • Direct Access, n = relative block number read n write n position to n read next write next rewrite n

31. Directory Structure • Directory: a collection of nodes containing information about all files Directory Files F 1 F 2 F 3 F 4 F n Both the directory structure and the files reside on disk

32. Disk Structure • Disk can be subdivided into partitions • also known as minidisks, slices • Disks or partitions can beprotected against failure using: RAID (Redundant Array of Independent Disks) • Disk or partition can be used raw – without a file system, or formatted with a file system • Entity containing file system known as a volume • Each volume containing file system also tracks that file system’s info in device directory or volume table of contents • general-purpose file systems vs special-purpose file systems

33. A Typical File-system Organization

34. Operations Performed on Directory • Search for a file • Create a file • Delete a file • List a directory • Rename a file • Traverse the file system

35. Organize the Directory (Logically) to Obtain • Efficiency – locating a file quickly • Naming – convenient to users • Two users can have same name for different files • The same file can have several different names • Grouping – logical grouping of files by properties, (e.g., all Java programs, all games, …)

36. Single-Level Directory • A single directory for all users Unique naming problem Grouping problem

37. Two-Level Directory • Separate directory for each user • Path name • Can have the same file name for different user • Efficient searching • No grouping capability

38. Tree-Structured Directories

39. Tree-Structured Directories (Cont) • Efficient searching • Grouping Capability • Current directory (working directory) • cd /spell/mail/prog

40. Tree-Structured Directories (Cont) • Absolute or relativepath name • Creating a new file is done in current directory • Delete a file rm <file-name> • Creating a new subdirectory is done in current directory mkdir <dir-name> Example: if in current directory /mail mkdir count mail prog copy prt exp count Deleting “mail”  deleting the entire subtree rooted by “mail”

41. Acyclic-Graph Directories • Have shared subdirectories and files

42. Acyclic-Graph Directories (Cont.) Issues: • A file can have more than one path (aliasing problem) • If dict deletes list dangling pointer Solutions: • Backpointers, so we can delete all pointers • Count number of references to a file • Implement shared files / directories: • New directory entry type: • Link – another name (pointer) to an existing file • Resolve the link – follow pointer to locate the file

43. General Graph Directory

44. General Graph Directory (Cont.) • How do we guarantee no cycles? (avoid infinite loops) • Allow only links to files, not subdirectories • Garbage collection: delete items that have no reference to it • Traverse file system and mark everything that can be accessed • Collected everything that is not marked onto a list of free space • Every time a new link is added, use a cycle detectionalgorithm to determine whether it is OK

45. File Sharing in Multiple User System • Sharing of files on multi-user systems is desirable • Sharing may be done through a protection scheme • Identify users • User IDs identify users, allowing permissions and protections to be per-user • Group IDs allow users to be in groups, permitting group access rights

46. Protection • File owner/creator should be able to control: • what can be done • by whom • Change owner user or group • chgrp: change group associated with file • chown: change owner of file • Types of access • Read • Write • Execute • Append • Delete • List

47. Access Lists and Groups • chmod 761 prog1.out • Mode of access: read, write, execute, setuid, setgid • Three classes of users RWX a) owner access 7  1 1 1 RWX b) group access 6  1 1 0 RWX c) public access 1  0 0 1 chmod: change access modes

49. setuid, setgid access right 49 • Mode of access: read, write, execute, setuid, setgid • setuid, setgid: Unix access rights flags that allow users to run an executable with permissions of the executable's owner or group. • Used to allow users to run programs with temporarily elevated privileges in order to perform a specific task. • When an executable file has been given setuid attribute, normal users who have permission to execute this file gain the privileges of the user who owns the file (commonly root) within the created process. When root privileges have been gained within the process, the application can then perform tasks on the system that regular users normally would be restricted from doing. • E.g. passwd, chsh commands for changing password or login shell • Need to modify system file /etc/passwd • Another example: program you used for submitting programs

50. File Sharing – Remote File Systems • Network allow file system access between systems • Manually via FTP • Automatically, seamlessly using distributed file systems • Semi automatically via world wide web • Client-servermodel allows clients to mount remote file systems from servers • Server can serve multiple clients • Client and user-on-client identification is insecure or complicated • NFS is standard UNIX client-server file sharing protocol • CIFS is standard Windows protocol • Standard operating system file calls are translated into remote calls • Distributed Information Systems (distributed naming services) such as LDAP, DNS, NIS, Active Directory implement unified access to information needed for remote computing