Nachos Instructional OS: Part 3

1. Nachos Instructional OS: Part 3 CS 170, Tao Yang, Fall 2011

2. Topics • File system implementation. • Assignment 3 • Virtual memory • File system extension

3. Nachos File System • Two versions (check openfile.h and filesys.h): • A ``stub'' version is simply a front-end to the Linux. • A Nachos version implemented on top of a raw disk. • Function layers (from top to down): • FileSystem object • Directory support. • I-node support (FileHeader) • Free block map • OpenFile objects (many) • Created by every call to open/access a file in a directory. • SynchDisk object • Created by synchDisk = new SynchDisk("DISK") in system.cc • The Disk object. • Created by disk = new Disk(name,..) in SynchDisk::SynchDisk()

4. File System Layers in Nachos FreeMap Bitmap for 32x32 blocks Nachos kernel thead Directory: Manage a directory of file names FileSystem: Directory management file create/open/delete FileHeader: i-node OpenFile: access individual file with read/write/seek. Operate on bytes. SynchDisk: synchronous access to a disk with concurrent threads. Operate on sectors. Disk: low-level interface that initiates asynchronous IO to physical disk sectors Base Operating System (Linux for our class)

5. Operations of File System Object • FileSystem(bool format). • called once at ``boot'' time. • For the stub version, just use Unix file system • Otherwise create a directory/control info. • Sector 0 is fileheader (i-node) for FreeMap. • Sector 1 is i-node for directory table. • Create(char *name, int initialSize). • Create a Nachos disk file. ``initialSize'' is ignored in the stub version; Unix doesn't require it. • OpenFile *Open(char *name). • Assume both read and write access. • in the stub version, Open simply opens the Unix file. • Remove(char *name) • List()

6. Directory Table stored in DirectoryFile • Sector 1 is the i-node (file header). • File content is 10 table entries. • DirectoryFile disk space is allocated in FileSystem() during initialization … Bool inUse --- is this entry used? Int sector --- location of file header (i-node) Filename -- 9 characters+ 1 for ‘\n’ …

7. directory.cc • Limit • FileNameMaxLen 9 // 9 characters at most for a file name • NumDirEntries 10 // at most 10 files • Directory Object attributes • tableSize -- # of directory entries which is 10 now. • table = in-memory directory entries, each of which is for one Nachos file. Content is copied from and copies back to a disk DirectoryFile. • Operations • Directory (int size) -- Create an in-memory directory of the desired size. • FetchFrom (OpenFile *file) -- Fetch disk DirectoryFile that contains directory entries, and copy to in-memory directory table. • WriteBack(OpenFile *file) -- Write back in-memory directory content to disk DirectoryFile • called every time when a file is added or removed to directory. • FindIndex (Char *name) --Search a file string name in a directory • Add (Char *name, int sector) – Add file name into a directory with disk sector no that contains the file header (i-node). • Remove(char *name) – remove a file.

8. File header (i-node) … int numBytes; // file size int numSectors; //sectors used • One filer header per file, fitting one sector (128 bytes) • Operations • Allocate(BitMap *bitMap, int fileSize); Initialize a file header • FetchFrom (int SectorNo) -- Fetch Fileheader sector from disk. • WriteBack(int SectorNo) -- Write back in-memory file header content to disk • ByteToSector(int offset) –Convert a file offset in bytes to a sector no Data sector int dataSectors[NumDirect] Data sector pointer 0 Data sector Data sector pointer 1 … Data sector Data sector pointer 29

9. OpenFile Object • OpenFile(int sector) • sector – the location on disk of the file header. • Create an OpenFile object which contains file control info, loaded from the disk file header. • Seek(int position) • Read/write in the current position • Read(char *buffer, int numBytes) • Write(char *buffer, int numBytes) • Read/write in a specific position • ReadAt(char *buffer, int numBytes, int FilePosition). • Identify specific sectors in the file and fetch them, copy to buffer. • WriteAt(char *buffer, int numBytes, int FilePosition) • May read partially written sectors first, and then write out updated sectors. • Length(). Return the file size.

10. SynchDisk Object: Synchronously access a disk • SynchDisk () • Called only once to create a physical disk object. Set the disk interrput hander as RequstDone(). • Create lock/semaphore for synchronization. • Notice that physical disk is an asynchronous device (disk requests return immediately, and an interrupt happens later on). • ReadSector(int sectorNumber, char* buffer) • Send a read request for a sector • Wait IO complete (wait for interrupt using a semaphone) • WriteSector(int sectorNumber, char* data) • Send a sector write request. Wait for interrupt. • RequestDone() • Disk interrupt handler. Wake up any thread waiting for the disk request to finish.

11. Disk Object • Simulates the behavior of a disk I/O device. • The disk has only a single platter, with multiple tracks (32). • Each track contains the same number of sectors (32). • Each sector is 128 bytes • Disk size = 32x32*128 =128K • Asynchronous read/write with an option of disk cache • Supported operations: • Disk(char *name, VoidFunctionPtr callWhenDone, int callArg) • Create a disk: just create a Unix file to simulate. • Setup a handling function (RequestDone in SynchDisk.cc) when a read/write request is done.

12. Operations in Disk device • ReadRequest(int sectorNumber, char *buffer) • Read from the specific location of the “physical disk”. • ticks = ComputeLatency(sectorNumber, FALSE); • interrupt->Schedule(DiskDone, (int) this, ticks, DiskInt); • WriteRequest(int sectorNumber, char *data) • Write data to specific location of the “physical disk”. • ticks = ComputeLatency(sectorNumber, TRUE); • interrupt->Schedule(DiskDone, (int) this, ticks, DiskInt); • ComputeLatency(int newSector, bool writing) • Latency= seek time + rotation time

13. Assignment 3 Workload • vm directory for part 1 is empty • Need to come up a design and code • Test C programs provided are sample code. • Amount of work • Part 1. ~400 lines of code in vm ( few changes in userprog) • Part 2. ~250 or less in filesys (few in userprog) • Part 1 and 2 can be done in parallel

14. Project 3 part 1: Virtual Memory • Work on vm subdirectory mainly • And addrspace.cc/.h and exception.cc in userprog • Create/manage a backing store (a file called SWAP using the OpenFile class). • Implement a page fault handler with dirty bit handling and a page replacement policy (LRU or second chance) • Test under various conditions: • One process with an address space larger than physical memory. • Concurrent processes with combined address space larger than physical memory.

15. Project 3 part 2: File system • Work on filesys subdirectoy mainly. • And change system call handling code in userprog if needed (to use this extended file system). • Extend the size of each file from 4K to 100K • A few single indirect pointers in i-node. • The size of a file can grow.

16. Report to be submitted •  HW3_WRITEUP • Summarize what is completed, what is not. • For VM, describe the design of VM. • For file system extension, • Describe design options and their trade-offs (for i-node and for handling file size growth. • List/explain main components and code modification in implementing your design • it will be graded: Do you write down the design, list advantages/disadvantages of options? Explain the reason of your choice? • Summarize the test effort (what is passed, the purpose of each test)

17. Emphasis of Project 3 •  Play tradeoffs • Understand the design options, especially for addressing required issue in file system extension. • Deadline-driven approach • Choose something simpler and easier to do first. • Still not enough time? • Complete with minimal efforts to pass the tests. • Hidden tests are very similar to public test cases. No complex test cases. • You are good as long as your code can run virtual memory>physical, can support growable file up to 100K. • Interpret the requirement towards your advantage if vague. If not specified, do something easier.

18. HW 3 Implementation Notes

19. Part 1: Getting Started • Read machine/translate.cc: • In Machine:Translate() for virtual address translation, PageFaultException is detected when the desired page is not in memory. • In Machine:ReadMem, Translate() is called for translating the desired virtual memory address and machine->RaiseException() is called with PageFaultException error.

20. What is next • Read mipssim.cc • Machine->ReadMem() is called in executing each instruction. • If PageFaultException is detected, the exception handler should load the desired page. • The hardware will try again. • Need to expand exception.cc to handle PageFaultException. • Once handled, return to user mode and restart the instruction caused the fault

21. User Instruction Execution Re-execute if Exception is raised Machine:Run () OneInstruction () ReadMem () WriteMem () Machine:Translate() Page writing? Set dirty bit Cannot find this page? Raise PageFaultException ExceptionHandler() Deal with PageFaultException

22. Files to be modified for Part 1 • New files in directory vm • Virtual memory manager • Swap space manager • Directory userprog (extension to HW 2) • exception.cc • Extension to handle PageFaultException • Addrspace.cc/.h • Prepare code/data for SWAP backstore. • Virtual address translation -> paging if needed

23. PageFaultException handling • For a virtual address which is not in memory, • Find free space to host the desired page in memory. • Load the desired page from SWAP to memory. • If no free space available, execute a replacement policy to find a victim page. • Swap out the victim page. • Swap in the desired page. • Adjust the page table.

24. Suggested Class in vm • Write a Swap Manager to facilitate free sector allocation in Swap File. • Initialize SWAPSIZE (512) as the total free sector available. • Allocate a sector. • Free a sector.

25. Suggested Class in vm • Virtual memory manager • load a page containing the virtual address (copy from the SWAP file to an allocated memory page). • Find a victim page • Swap out a selected page to SWAP

26. Modify AddSpace.cc • In translating a virtual user address for kernel, if it is not in memory, bring data from SWAP. • When allocating a new address space, • Allocate a proper number of sectors from SWAP for this process. • Copy binary data to the allocated SWAP sectors. • Initial page number for this process can be 0.

27. Synchronization issues • Two system calls may be processed concurrently and the synchronization is needed. • Any time a process sleeps in the kernel, another process can run • Two processes try to initiate I/O on the same page at the same time • May need a lock (or at least a ``busy'' flag) for each physical memory page, and possibly also for each virtual memory page.

28. Part 2: Steps for Nachos file system extension • Understand how the Nachos file system operates and how Nachos implements files. • Modify the file system calls from HW 2 to use Nachos file system • Modify Nachos file system so that you can create larger (100K) • Allow the Write system call to grow the size of a file.

29. Files to be modified in Part 2 • Directory filesys • filehdr.cc/.h Support single indirect pointers in i-node and can expand the file length with more sectors allocated. • openfile.cc Expand file length/allocate sectors when writing data at the end or beyond. • directory.cc Add more table entries when the current table is full and a new file is added. • Directory userprog • exception.cc • Make sure that file system calls use extended Nachos file system.

30. Comparison: Unix file i-node (4K bytes per block)

31. Step 3: Extend maximum file size • Modify the FileHeader i-node to support 100K size. • Include 5 data sector pointers, and 25 sectors for single indirect mapping. • At most map 5 + 25*32=805 blocks. Maximum file length = 805*128=103040bytes. • Another option: every entry points to an i-node. • Suggest to add a class called IndirectPointerBlock which takes 1 sector (32 entries) with following operations. • FetchFrom (int sector). WriteBack(int sector). • AddSector (int sector) – add a new sector. • Deallocate(BitMap *freeMap) – deallocate all sectors. • ByteToSector (int offset). – convert an offset in bytes to a sector.

32. Step 4: File size can grow • Modify the current implementation so that a write beyond the end of the file extends the size of the file. • Another option is: have a fixed size initially to allocate the i-node space initially. But donot allocate data blocks until they are needed. • Gracefully handle the case of the disk being full or maximum size is reached - the user process should not crash. Instead an error should be returned.

33. Step 4 • Modify WriteAt() method in openfile.cc : • Case 1: Write request is within the bound of the file. Handle like the existing code • Case 2: Write request overwrites some portion of existing file and goes beyond by N bytes. • Case 3: No overwrite to the current data, but goes beyond the end of the file. Here. • To extend a file dynamically, • Extend i-node/data dynamically: include a routine in FileHeader to extend a file by N bytes. • Or pre-allocate the i-nodes, allocate data sectors when needed.

34. Testing issues • Clean out all the .c files in the code/test directory. There might be name conflicts from previous projects • Get sample test programs in part1 and part2 of the ~cs170/nachos-projtest/HW3 directory. • Test part 1. Test as you would for project 2. You can run the ./nachos in the vm/ • Test part 2. It can be convenient to create another test directory (e.g. test under test2 directory or fielsys/test)  • Notice file names cannot exceed 9 characters.

35. Testing issues in Part 2 • May need to load user binaries/data into Nachos file system during execution • Example: csil$ pwd cs170/nachos/code/filesys csil$ ./nachos -f -cp ../test2/Prog1 Prog1 -x Prog1 • Another example with two binary programs csil$ ./nachos -f -cp ../test2/Prog2 Prog2 -cp ../test2/Prog3/Prog3 -x Prog2