More I/O Performance Considerations

1. More I/O Performance Considerations Chris Gill, Brian Kocoloski CSE 522S - Advanced Operating Systems Washington University in St. Louis St. Louis, MO 63130

2. Character vs. Block Devices Character Devices Block Devices SD card keyboard hard disk drive serial port USB drive Data streams from a device sequentially Data can be randomly accessed in anarbitrary order by software CSE 522S – Advanced Operating Systems

3. I/O scheduling for block devices • Performance comparison of most block devices with other hardware • Reading/writing a CPU register: • O(nanoseconds) • Access memory in CPU cache: • O(nanoseconds) • DRAM memory access: • O(microseconds) • Access to spinning hard drive: • O(milliseconds) CSE 522S – Advanced Operating Systems

4. Comparing orders of magnitude • Assume 1 nanosecond=1 minute • 1 microsecond: almost 17 hours • 1 millisecond: almost 2 years • Takeaway: block I/O operations are much slower than most other hardware the kernel accesses CSE 522S – Advanced Operating Systems

5. Sectors and Blocks • Hard disks typically have sectors, the size of which is specified by the device • File system code operates on units called blocks • Block size must be no smaller than sector size • Block size must be no larger than physical memory page size • Accessing disk sectors incurs several types of device-specific overhead • Seek time • Rotational latency LKD pp. 291 CSE 522S – Advanced Operating Systems

6. I/O Scheduling • Way in which I/O requests are handled matters • At least, requests for adjacent blocks should be merged (fewer accesses gives lower overhead) • Key ideas behind I/O scheduling: • Request merging • Request sorting • Read-vs-write priority CSE 522S – Advanced Operating Systems

7. Linus Elevator • Keeps sorted list of requests • i.e.; assume requests are issued for blocks 2, 5, 0, and 8 • Maintains sorted order 0,2,5,8 • Total seek time? • 0+2+3+3=8 • Total seek time if using FIFO? • 2+3+5+8=18 CSE 522S – Advanced Operating Systems

8. Other I/O scheduling algs • deadline • Prioritize read over writes • Maintains threedistinct request queues • Sorted: all requests, sorted by disk sector • Read FIFO: read requests, sorted by arrival time • Expiration value of 500 ms • Write FIFO: write requests, sorted by arrival time • Expiration value of 5 s • Basic algorithm: Select from Sorted queue unless head of Read/Write has expired, then take from there first LKD pp. 301 CSE 522S – Advanced Operating Systems

9. Other I/O scheduling algs • anticipatory(as) • Builds on deadline • Tries toanticipate future read requests • After scheduling a read operation, wait for a little bit (up to 6ms in current kernels) to see if another read comes • Assumes reads will come to nearby locations • Why would they? LKD pp. 301 CSE 522S – Advanced Operating Systems

10. Other I/O scheduling algs • Complete fair queuing (cfq) • Fundamentally different from other algs • Goal is per-process fairness • All processes have their own scheduling queues, sorted by disk sector • Noop • No read-write priority or sorting based on disk sectors • What is it used for? Block devices that can randomly access data just as easily as they can sequentially access data • Solid state drives • NVME flash drives CSE 522S – Advanced Operating Systems

11. Other Optimizations • Completely user-space optimizations • Custom elevator algorithms • User-space + kernel optimizations • Asynchronous I/O • Scatter/gather I/O • User-space providing advice to the kernel • Add information about your pending access patterns via fadvise • E.g., can help the kernel make intelligent use of the page cache CSE 522S – Advanced Operating Systems

12. Custom I/O Elevators • The Linux elevators are all implemented in the kernel – but you can (sort of) implement your own algorithms in userspace by ordering the way you issue I/O requests • Potential approaches? CSE 522S – Advanced Operating Systems

13. Custom I/O Elevators • The Linux elevators are all implemented in the kernel – but you can (sort of) implement your own algorithms in userspace by ordering the way you issue I/O requests • Potential approaches? • Sort your I/O requests based on the filename • Sort your I/O requests based on inode • Sort your I/O requests based on physical block location CSE 522S – Advanced Operating Systems

14. e.g., Sorting by Inode or Physical Block • Relevant system calls: • ioctl • fstat CSE 522S – Advanced Operating Systems

15. Asynchronous vs. Synchronous I/O Benefits of asynchronous I/O (aio) • Decouples I/O from other process activities aio_read() does not wait for data to arrive • Returns as soon as read request is queued • Request points to a buffer for the data • Can register function that’s called on completion aio_write() does not wait for data to be written • Returns as soon as write request is queued • Can register function that’s called on completion • Completion notification can be set either for when data are in kernel’s buffers, or are synched to disk CSE 522S – Advanced Operating Systems

16. Asynchronous I/O CSE 522S – Advanced Operating Systems

17. Asynchronous I/O CSE 522S – Advanced Operating Systems

18. Scatter/Gather I/O Atomic input and output with “io vectors” (iovec) • Arrays of pointers to (and sizes of) memory buffers • Copying pointers usually costs less than copying data “Scatter-read” using readv • Moves data from a file into a set of memory buffers “Gather-write” using writev • Moves data from a set of memory buffers into a file ioveciov[5] 4 7 6 8 4 the slithy Ptr head 1 (e.g., a linked list of sentences) Ptr head 2 and t’was brillig CSE 522S – Advanced Operating Systems

19. Advice to the Kernel for File I/O • advice: • POSIX_FADV_NORMAL: no advice – treat as normal • POSIX_FADV_RANDOM: application intends to access in random (non-sequential) order • POSIX_FADV_SEQUENTIAL: application intends to access in sequential order • POSIX_FADV_WILLNEED: application intends to access data in the near future • POSIX_FADV_NOREUSE: application intends to access data exactly once in the near future • POSIX_FADV_DONTNEED: application does not intend to access data at all CSE 522S – Advanced Operating Systems