General Adaptive Replacement Policies

1. General Adaptive Replacement Policies Yannis Smaragdakis Georgia Tech

2. My Agenda • Present a cool idea in replacement algorithms • Argue that replacement algorithms (and especially VM) are as much a part of ISMM as allocation/GC • same locality principles

3. Overview • Background: theory of replacement algorithms • How to make adaptive algorithms with good performance and theoretical guarantees • Experimental methodologies in replacement algorithms and evaluation of the idea

4. Overview • Background: theory of replacement algorithms • How to make adaptive algorithms with good performance and theoretical guarantees • Experimental methodologies in replacement algorithms and evaluation of the idea

5. registers CPU cache main memory (VM cache + file cache) disk (VM + files) Storage Hierarchies • Storage hierarchies are common in systems memory hierarchy

6. Management of Storage Hierarchies • One level of the hierarchy acts as a fast cache for elements of the next • A replacement algorithm determines how the cache is updated when it is full • the most recently used page must always be in the fast cache for easy access • hence, when the cache is full, references to pages not in the cache must cause a replacement: a page in the cache needs to be removed • called a “fault” or a “miss”

7. a b g e c k f d h i … j Replacement Schematically all blocks: b m e a r h k x f c g y p z i q buffer:

8. LRU Replacement • The Least Recently Used (LRU) algorithm has been predominant for decades • simple and effective • no general purpose algorithm has consistently outperformed LRU • supported by numerous results (both theoretical and experimental) • Under LRU • a cache of size M always holds the M most recently used elements • at replacement time, the least recently used element in the cache is removed (“evicted”)

9. Main Theoretical Results • Much major theory is based on competitive analysis • how many faults an algorithm incurs relative to the faults of another algorithm • beautiful results with potential functions

10. Example Theorems (slightly simplified) • LRU will not suffer more than M times as many faults as OPT • M: memory size in blocks. Large number • OPT: optimal (clairvoyant) algorithm • No other algorithm can do better • LRU with twice as much memory as OPT will suffer at most twice the faults of OPT

11. Example Proof Technique • Theorem: LRU with twice as much memory as OPT will suffer at most twice the faults of OPT (Sleator and Tarjan, ’84) • Proof idea: 2M pages need to be touched between successive LRU faults on the same page. OPT will suffer at least M faults.

12. Overview • Background: theory of replacement algorithms • How to make adaptive algorithms with good performance and theoretical guarantees • Experimental methodologies in replacement algorithms and evaluation of the idea

13. This Paper • Note that all previous theoretical results are negative results for practical applications • but we don’t care about OPT! We care about how close we can get to algorithms that work well in well-known cases • use the force (competitive analysis) to do good!

14. This Paper • Main result: take any two replacement algorithms A and B, produce adaptive replacement algorithm AB that will never incur more than twice (or three times) the faults of either A or B • for any input! We get the best of both worlds • result applies to any caching domain • thrashing avoidance is important, 2x guarantee is strong • we can’t avoid the negative theoretical results, but we can improve good practical algorithms indefinitely

15. Robustness • Definition: we say R1 is c-robust w.r.t. R2, iff R1 always incurs at most c times as many faults as R2

16. A Flavor of the Results • Given A, B, create AB such that it simulates what A and B do on the input. Then at fault time, AB does the following: • if A also faults but B doesn’t, imitate B • i.e., evict a block not in B’s memory (one must exist) • otherwise imitate A • i.e., evict a block not in A’s memory if one exists, evict the block that A evicts otherwise

17. Surprising Result • This very simple policy AB is 2-robust w.r.t. A and B! • Proof idea: to “fool” AB into bad behavior, say w.r.t. A, A needs to suffer a fault. For every “wrong” decision, AB takes two faults to correct it. • formalized with two potential functions that count the difference in cache contents between AB and A or B

18. More Sophisticated Adaptation • We can create a more sophisticated AB • remember the last k faults for either A or B • imitate the algorithm that incurs fewer • This is 3-robust relative to A and B(but will do better in practice) • the proof is quite complex, requires modeling the memory of past faults in the potential function • the result can probably be made tighter, but I’m not a theoretician 

19. Implementation • AB needs to maintain three times the data structures in the worst case • one to reflect current memory contents, two for the memory contents of A and B • but in practice these three structures have a high content overlap • AB only performs work at fault time of A, B, or AB • if A, B are realizable, AB is realizable

20. Overview • Background: theory of replacement algorithms • How to make adaptive algorithms with good performance and theoretical guarantees • Experimental methodologies in replacement algorithms and evaluation of the idea

21. Experimental Evaluation • We show results in virtual memory (VM) management • “boring” area: old and well-researched, with little fundamental progress in the past 20 years • strong programmatic regularities in behavior (e.g., regular loops) • unlike, e.g., web caching • we have the luxury to implement smart memory management algorithms

22. Trace-Driven Memory Simulation • Trace-driven simulation is a common technique for evaluating systems policies • How does it work? • the sequence of all memory references of a running program is captured to a file • this sequence is then used to simulate the system behavior under the proposed policy

23. Simulation Experiments in VM • Standard experimental evaluation practices: • one program at a time • otherwise scheduler adds too much noise. In practice VM dictates scheduling, not the other way • common cases include one large application that pages anyway • large range of memories • good algorithms are versatile, behave well in unpredictable situations

24. Simulation Experiments in VM • Standard experimental evaluation practices: • simulate idealized policies • the whole point is to see what policy captures locality • simulating policies with realizable algorithms is generally possible (although often non-trivial)

25. Results of Evaluating Adaptivity • Adaptive replacement is very successful • Almost always imitates the best algorithm it adapts over • apply the adaptivity scheme repeatedly • Never tricked by much • Occasionally better than all component algorithms

26. Example Results

27. Example Results

28. Example Results

29. Example Results

30. Conclusions • Adaptivity is cool • it is very simple • it works • it offers good theoretical guarantees