A Reusability-Aware Cache Memory Sharing Technique for High Performance CMPs with Private Caches

1. A Reusability-Aware Cache Memory Sharing Technique for High Performance CMPs with Private Caches Sungjune Youn, Hyunhee Kim and Jihong Kim School of Computer Science & Engineering Seoul National University Computer Architecture and Embedded Systems (CARES) Laboratory Workshop on Chip Multiprocessor Memory Systems and Interconnects 2007(CMP-MSI) 2007.2.11

2. Outline • Introduction • Motivation • Reusability-Aware Cache Sharing Technique (RACS) • Overview of the RACS technique • Two Major Steps • Step 1: Block Reusability Prediction • Step 2: Memory Demand Prediction • Evaluation • Conclusions

3. Introduction • Chip Multiprocessors (CMPs) emerge as a dominant architectural alternative • Most current CMPs support two levels of on-chip hierarchy • L1 cache organization is almost same • A small private L1 cache • L2 cache organization could be quite different • Private L2 cache vs. Shared L2 cache • Efficient L2 cache management is necessary • On-chip cache memory space is limited in CMPs • Off-chip memory accesses require a much longer latency than on-chip communication costs

4. L2 Cache Organization in CMPs • Private L2 cache vs. Shared L2 cache L1 Short access latency Utilizing capacity efficiently But inefficient in utilizing the L2 cache space But longer access latency, More on-chip network traffic How to Combine Strengths of Private & Shared Caches?

5. Cooperative Caching (CMP-CC) • “Cooperative Caching for Chip Multiprocessors”, ISCA 2006 • Based on the private cache organization • Writing back L2 victims from the local cache to peer cache P0 P1 P2 P3 Private L2 Private L2 Private L2 Private L2 Randomly with a given probability 0% ~ 100% L2 Victim

6. Problem of the Reusability-Oblivious Write Back • If the block is written back to other cache, but is not reused ->System performance could be degraded Should be reduced • <The number of unused and reused blocks in CMP-CC100%> Reusability-Aware Adaptive Write Backs are Necessary

7. Adaptive Write Back • Adaptive Write Back requires • Reusability of each block • Memory demand of each processor P0 P1 P2 P3 Private L2 Private L2 Private L2 Private L2 Which peer cache has a block with low reusability? Which peer cache has a memory demand smaller than P0? High Reusability? L2 Victim Low Reusability?

8. Reusability Prediction Technique • Goal: Do not write back blocks with low reusability • The reusability of a block is based on Access Time Interval and Frequency (ATIF) • Reusability after the eviction The block with long time interval has high reusability time Eviction The block with short time interval has low reusability time Eviction

9. Access Time Interval and Frequency Pattern • Classify blocks into 16 patterns • Two counters per each block • Number of accesses with long time interval • Number of accesses with short time interval • 2 bits of the short time interval counter + 2 bitsof the long time interval counter • Monitor how many blocks are reused per each pattern • If blocks in a certain pattern are highly reused -> Blocks in this pattern have high reusability

10. Fraction of Unused and Reused Blocks Long time interval Short time interval The larger number of long time interval accesses -> The larger number of reused blocks

11. Memory Demand Prediction Technique • Goal: Do not corrupt the L2 cache of processors with a high memory demand • Heuristic: The more replacements occur, the more processor requires memory • Replacement time interval history (Replinterval_history ) • Prediction value of the memory demand • Updated every time replacement occurs • Smaller Replinterval_history means the processor requires more memory • Wewrite back the block to the peer cache with smaller memory demand

12. Experiment Setting • Based on a CATS Shared-Memory Multiprocessor Simulator • Parameter • L1 I/D Cache • 16KB, 1-way • 1cycle • L2 Private Cache • 256 KB, 4-way • 6/40 cycles • L2 Shared Cache • 1MB, 16-way • 38 cycles • Off-chip memory latency • 500 cycles • Splash2 benchmark programs used: Cholesky, FMM, LU, Radix

13. The Number of Unused and Reused Blocks 61% 62% The number of reused blocks is same with CMP-CC100% The number of unused blocks is reduced by 62%

14. Normalized Memory Access Latency Even though memory access latency increases as the probability of the CMP-CC increases, RACS reduces it. Memory access latency increases as the probability of the CMP-CC increases, RACS also reduces it. The RACS scheme reduces the average memory access latency by 14% and 4% over the private L2 scheme and CMP-CC 100% on average, respectively

15. Normalized Average IPC Improves average IPC by 3% and 1% over the private L2 scheme and CMP-CC100% on average, respectively

16. Normalized Energy Consumption 10% less energy over the private cache 2% less energy over the CMP-CC100%

17. Conclusions • Proposed Reusability-Aware Cache memory Sharing technique (RACS) • Based on private L2 cache • Taking advantage of both private L2 cache and shared L2 cache • Adaptively writing back L2 victims to peer L2 cache • Using reusability of the block and memory demand of the processor • RACS reduces the number of unused blocks by 60% over CMP-CC. • RACS reduces the average memory access latency by 14% and 4% over the private L2 cache and CMP-CC, respectively.

18. Thank You

19. Overhead • Hardware overhead • Peer-to-peer communication lines between caches • For 4 CPU, 6 lines of 21 bits • Additional counters for two prediction technique • Reusability prediction • 4-bit counter: the number of accesses with long time interval per each block • 2-bit counters: the number of accesses with short time interval per each block • 2 bits: indicating which processor writes back this block • 1 bit: indicating that block is reused • 16 2-bit pattern counters per each private cache • Memory demand prediction • 8 bit counter: time from the last replacement • 8 bit : replacement interval history • Total: 9 bits per block and 48 bits per cache => Area overhead is less than 1% of the private cache

20. Time overhead • Write back decision is made after • A block is evicted from the cache and placed in the write back queue -> Write back decision is not on the critical path

21. How to distinguish access with short time interval and long time interval? • If there is any intervening access to the block that belongs to the same set -> long time interval • If not -> short time interval • Using 2-bit (for 4-way cache) to record the most recently accessed block

22. Yes Do not write back block a To peer L2 cache Block a is evicted from private L2 cache A Shared State? No Written back from other L2 cache and not used? Yes No Yes Low reusability? Exists any block with low reusability at the bottom of LRU in peer L2 caches? No Yes Write back blocka to the cache which has the low reusability block Exists any L2 cache whose memory demand is ω times smaller than cache A’s? No Yes Write back block a to the cache which has ω times smaller memory demand No

23. If there is no peer L2 cache with ω times smaller memory demand, we do not write back the block to a peer L2 cache even though it has high reusability • ω value? • Each private L2 cache has its own ω value • Decreased by 1 when block is reused • Increased by 1 when three blocks are written back to other cache • If many of blocks are reused, we can write back a block to a peer cache • Even though the difference of the memory demand is not large

24. 1 2 1 1 2 2 Processor Processor Set number, Replinterval_history L2 Cache L2 Cache L2 Cache L2 Cache Exist the block with low reusability ? And Received Replinterval_history > own Replinterval_history Processor Processor