Conquest : Better Performance Through A Disk/Persistent-RAM Hybrid File System

1. Conquest: Better Performance Through A Disk/Persistent-RAM Hybrid File System USENIX 2002 An-I Andy Wang • Peter Reiher • Gerald Popek University of California, Los Angeles Geoffrey Kuenning Harvey Mudd College

2. Conquest Overview • File systems are optimized for disks • Performance problem • Complexity • Now we have tons of inexpensive RAM • What can we do with that RAM?

3. Conquest Approach • Combine disk and persistent RAM (e.g., battery-backed RAM) in a novel way • Simplification • > 20% fewer semicolons than ext2, reiserfs, and SGI XFS • Performance (under popular benchmarks) • 24% to 1900% faster than LRU disk caching

4. Motivation • Most file systems are built for disks • Problems with the disk assumption: • Performance • Complexity Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion

5. 106 105 Hardware Evolution CPU (50% /yr) 1 GHz Memory (50% /yr) Accesses Per Second (Log Scale) 1 MHz 1 KHz Disk (15% /yr) 1990 1995 2000 (1 sec : 6 days) (1 sec : 3 months) Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion

6. Disk arm Disk platters Access time = seek time (disk arm) + rotational delay (disk platter) + transfer time Inside the Pandora’s Box Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion

7. Disk arm scheduling Group information on disk Disk readahead Buffered writes Disk caching Disk Optimization Methods • Data mirroring • Hardware parallelism Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion

8. predictive readahead synchronization cache replacement elevator algorithm data clustering data consistency asynchronous write Complexity Bytes Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion

9. Magnetic RAM? battery-backed DRAM (write once) flash memory disk tape persistent RAM Storage Media Alternatives $/MB (log) 10-3 106 100 103 accesses/sec (log) 10-3 Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion [Caceres et al., 1993; Hillyer et al., 1996; Qualstar 1998; Tanisys 1999; Micron Semiconductor Products 2000; Quantum 2000]

10. Booming of digital photography 4 to 10 GB of persistent RAM paper/film Persistent RAM 1” HDD 2.5” HDD 3.5” HDD Price Trend of Persistent RAM 102 101 $/MB (log) 100 10-1 10-2 1995 2000 2005 Year Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion [Grochowski 2000]

11. Old Order; New World • Disk staying around • Cost, capacity, power, heat • RAM as a viable storage alternative • PDAs, digital cameras, MP3 players • More architectural changes due to RAM • A big assumption change from disk • Rethink data structures, interface, applications Motivation– Conquest Design – Conquest Components – Performance Evaluation – Conclusion

12. Getting a Fresh Start What does it take to design and build a system that assumes amplepersistent RAM as the primary storage medium? Motivation –Conquest Design– Conquest Components – Performance Evaluation – Conclusion

13. Design and build a disk/persistent-RAM hybrid file system Deliver all file system services from memory, with the exception of high-capacity storage Benefits: Simplicity Performance Conquest Motivation –Conquest Design– Conquest Components – Performance Evaluation – Conclusion

14. Simplicity • Remove disk-related complexities for most files • Make things simpler for disk as well • Less complexity • Fewer bugs • Easier maintenance • Shorter data path Motivation –Conquest Design– Conquest Components – Performance Evaluation – Conclusion

15. Performance • Overall • All management performed in memory • Memory data path • No disk-related overhead • Disk data path • Faster speed due to simpler access models Motivation –Conquest Design– Conquest Components – Performance Evaluation – Conclusion

16. Conquest Components • Media management • Metadata management • Allocation service • Persistence support • Resiliency support Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

17. User Access Patterns • Small files • Take little space (10%) • Represent most accesses (90%) • Large files • Take most space • Mostly sequential accesses • Except database applications Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion [Iram 1993; Douceur et al., 1999; Roselli et al., 2000]

18. Files Stored in Persistent RAM • Small files (< 1MB) • No seek time or rotational delays • Fast byte-level accesses • Contiguous allocation • Metadata • Fast synchronous update • No dual representations • Executables and shared libraries • In-place execution Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

19. Conquest Memory Data Path Storage requests Persistence support Battery-backed RAM Small file and metadata storage Memory Data Path of Conquest Conventionalfile systems Storage requests IO buffer management IO buffer Persistence support Disk management Disk Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

20. Large-File-Only Disk Storage • Allocate in big chunks • Lower access overhead • Reduced management overhead • No fragmentation management • No tricks for small files • Storing data in metadata • No elaborate data structures • Wrapping a balanced tree onto disk cylinders Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion [Devlinux.com 2000]

21. Sequential-Access Large Files • Sequential disk accesses • Near-raw bandwidth • Well-defined readahead semantics • Read-mostly • Little synchronization overhead (between memory and disk) Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

22. Disk Data Path of Conquest Conventionalfile systems Conquest Disk Data Path Storage requests Storage requests IO buffer management IO buffer management IO buffer Battery-backed RAM IO buffer Small file and metadata storage Persistence support Disk management Disk management Disk Disk Large-file-only file system Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

23. Random-Access Large Files • Random access? • Common definition: nonsequential access • A typical movie has 150 scene changes • MP3 stores the title at the end of the files • Near Sequential access? • Simplify large-file metadata representation significantly Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

24. i-node • File attributes • Data Name(s) Logical File Representation File Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

25. i-node • File attributes • Data locations • Data blocks Physical File Representation Name(s) File Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

26. data block location data block location data block location data block location data block location data block location index block location index block location data block location index block location index block location index block location data block location index block location 10 Ext2 Data Representation i-node Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

27. Problems with Ext2 Design - Designed for disk storage - Optimization for small files makes things complex - Random-access data structure for large files that are accessed mostly sequentially - Data access time dependent on the byte position in a file - Maximum file size is limited Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

28. Conquest Representation • Persistent RAM • Hash(file name) = location of data • Offset(location of data) • Disk storage • Per-file, doubly linked list of disk block segments (stored in persistent RAM) Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

29. Conquest Design + Direct data access for in-core files + Worse case: sequential memory search for infrequent random accesses to on-disk files + Maximum file size limited by physical storage Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

30. Implementation Status • Kernel module under Linux 2.4.2 • Fully functional and POSIX compliant • Modified memory manager to support Conquest persistence • Preparing for office-wide deployment Motivation – Conquest Design –Conquest Components– Performance Evaluation – Conclusion

31. Conquest Evaluation • Architectural simplification • Feature count • Performance improvement • Memory-only workload • Memory and disk workload Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion

32. Conventional Data Path • Buffer allocation management • Buffer garbage collection • Data caching • Metadata caching • Predictive readahead • Write behind • Cache replacement • Metadata allocation • Metadata placement • Metadata translation • Disk layout • Fragmentation management Conventionalfile systems Storage requests IO buffer management IO buffer Persistence support Disk management Disk Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion

33. Memory Path of Conquest • Buffer allocation management • Buffer garbage collection • Data caching • Metadata caching • Predictive readahead • Write behind • Cache replacement • Metadata allocation • Metadata placement • Metadata translation • Disk layout • Fragmentation management Conquest Memory Data Path Storage requests Persistence support Battery-backed RAM Small file and metadata storage • Memory manager encapsulation Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion

34. Disk Path of Conquest • Buffer allocation management • Buffer garbage collection • Data caching • Metadata caching • Predictive readahead • Write behind • Cache replacement • Metadata allocation • Metadata placement • Metadata translation • Disk layout • Fragmentation management Conquest Disk Data Path Storage requests IO buffer management Battery-backed RAM IO buffer Small file and metadata storage Disk management Disk Large-file-only file system Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion

35. PostMark Benchmark • Conquest is comparable to ramfs • At least 24% faster than the LRU disk cache • ISP workload (emails, web-based transactions) 250 MB working set with 2 GB physical RAM Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion [Katcher 1997; Sweeney et al., 1996; Card et al., 1999; Namesys 2002]

36. <= RAM > RAM PostMark Benchmark • When both memory and disk components are exercised, Conquest can be several times faster than ext2fs, reiserfs, and SGI XFS 10,000 files, 3.5 GB working set with 2 GB physical RAM Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion

37. PostMark Benchmark • When working set > RAM, Conquest is 1.4 to 2 times faster than ext2fs, reiserfs, and SGI XFS 10,000 files, 3.5 GB working set with 2 GB physical RAM Motivation – Conquest Design – Conquest Components –Performance Evaluation – Conclusion

38. Lessons Learned • Faster than LRU caching, unexpected • Heavyweight disk handling • Severe penalty for accesses to content • Matching user access patterns to storage media offers considerable simplification and better performance • Not an automatic result • Need careful design Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

39. Conclusion • Conquest demonstrates how rethinking changes in underlying assumptions can lead to significant architectural and performance improvements • Radical changes in hardware, applications, and user expectations in the past decade should lead us to rethink other aspects of OS as well. Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

40. Questions . . . Conquest: http://lasr.cs.ucla.edu/conquest Andy Wang: awang@cs.ucla.edu