Experience with Recoverable Virtual Memory

1. Experience withRecoverable Virtual Memory M. Satyanarayanan School of Computer Science Carnegie Mellon University

2. Background • RVM = “Recoverable Virtual Memory” • software library to provide transactional properties for VM • in use since early 1990’s • primarily in Coda File System (large servers to tiny handhelds) • Also used by others outside our group (not by design!) • persistent garbage collection [O’Toole et al 1993] • transactional distributed shared memory [Feeley et al. 1994] • persistent Java [Jordan 1996] • Rio Vista transactional system [Lowell & Chen 1997] • Some questions • What lessons have we learned from RVM? • What is the role of the OS? • How can hardware support help RVM?

3. Application code Nesting Distribution Serializability RVM Atomicity Permanence: process failure Operating System Permanence: media failure Functionality • Focus on A & D of ACID (I provided by app) • Segment • persistent image of mapped data • resides in file or raw disk partition • Region • all or part of a segment in active use • area of an address space backed by RVM • multiple regions per address space • explicit mapping operations begin_transaction (&tid, restore_mode) set_range (tid, base_addr, nbytes) end_transaction (tid, commit_mode) abort_transaction (tid) map (region_desc, options_desc) unmap (region_desc) flush ( ) truncate ( )

4. RVM Lessons • Ability to treat VM transactionally is a BIG win • especially valuable in failure handling in distributed systems • encapsulates many messy details • Both meanings of “lightweight” important • small resource footprint • easy to use • Most of the value comes from very basic features (KISS) • additional fancy features, complexity not worth it • factoring out of isolation works well • Log optimizations valuable in reducing log traffic (20-60%) • inter-transaction and intra-transaction • no-flush transactions with explicit flush( ) calls useful

5. Role of OS • RVM is a highly portable library • absolutely no OS dependence (other than POSIX) • runs on Linux, *BSD, MacOS X, and even WinXP (using Cygwin) • Lack of integration has performance consequences • storage for segment distinct from swap space • longer startup latency (fixable with private mmap) • poorer paging performance • Predecessor of RVM was tightly integrated with OS • Camelot used many Mach mechanisms (e.g. external pager) • stressed little-used OS features, many subtle bugs • very complex to use, even more complex to debug • potential win of OS integration not realized in practice • Can’t share regions across processes without OS support • may be much more significant than performance issues • prevents use of multiple address spaces for robustness

6. How Can Hardware Help? • Persistent RAM-speed log! • would dramatically improve latency of log forces • battery-backed memory ok, but brittle abstraction • perhaps flash memory log device? • (not important to integrate into address space) • Avoid need for manual set_range( ) calls • forgetting this is major source of programming errors • page granularity not good enough; has to be byte granularity • perhaps get_ranges(tid) info from hardware? • Isolation is completely orthogonal • hardware concurrency control would fit easily into RVM • separate portable Isolation library would be ideal