iCFP: Tolerating All Level Cache Misses in In-Order Processors

1. iCFP: Tolerating All Level Cache Misses in In-Order Processors Andrew Hilton, Santosh Nagarakatte, Amir Roth University of Pennsylvania {adhilton,santoshn,amir}@cis.upenn.edu HPCA-15:: Feb 18, 2009

2. A Brief History … performance power Pentium (in-order) PentiumII (out-of-order) Core2Duo (out-of-order, 2 cores) POWER! Niagara2 (in-order, 16 cores, 64 threads) Nehalem (out-of-order, 4 cores, 8 threads)

3. In-order cores • Power efficiency • More cores Out-of-order cores • Single thread IPC (+63%) In-order vs. Out-of-Order Is there a compromise? Key idea • Main benefit of out-of-order: data cache miss tolerance • Can we add to in-order in a simple way?

4. Runahead Runahead execution[Dundas+, ICS’97] In-order + miss-level parallelism (MLP) Checkpoint and “advance” under miss Restore checkpoint when miss returns Poison RF0 I$ D$ • Regfile checkpoint-restore Forwarding$ • Per register “poison” bits • Forwarding cache Additional hardware? Can we do better?

5. Yes We Can! (Sorry) iCFP: in-order Continual Flow Pipeline Runahead, but … Save miss-independent work Re-execute only miss forward slice Slice Buffer Poison RF1 Poison RF0 I$ D$ • Slice buffer Forwarding$ • Replace forwarding cache with store buffer • Hijack additional regfile used for multi-threading Store Buffer Additional hardware? In-order adaptation of CFP [Srinivasan+, ASPLOS’04] • Unblock pipeline latches, not issue queue and regfile • Apply to misses at all cache levels, not just L2

6. iCFP Roadmap Motivation and overview (Not fully) working example Correctness features Register communication for miss-dependent instructions Store-load forwarding Multiprocessor safety Performance features Evaluation

7. Tail  last completed instruction  RF0 Instructions flowing through pipeline PC/instance Bold paths are active Example Tail Slice Buffer Poison RF1 Poison RF0 (Tail) C1 B1 A1 I$ D$ Store Buffer

8. r2 • Checkpoint regfile • Poison A1’s output register r2 Example Tail Slice Buffer Load A1 misses, transition to “advance” mode Poison RF1 Poison RF0 (Tail) C1 B1 A1 Miss I$ D$ Store Buffer

9. Example Pending miss (red) Slice Buffer Tail Load A1 misses, transition to “advance” mode Poison RF1 r2 Poison RF0 (Tail) D1 C1 B1 A1 I$ D$ Store Buffer • Checkpoint regfile • Poison A1’s output register r2 • Divert A1 to slice buffer

10. Example A1 Tail Slice Buffer • Propagate poison through data dependences Poison RF1 r2 Poison RF0 (Tail) D1 C1 B1 I$ D$ Store Buffer

11. Advance Miss-dependent instruction (this color) A1 Slice Buffer Tail • Propagate poison through data dependences • Divert miss-dependent instructions to slice buffer Poison RF1 r2 Poison r3 RF0 (Tail) E1 D1 C1 B1 I$ D$ Store Buffer

12. Advance A1 B1 C1 Slice Buffer • Propagate poison through data dependences • Divert miss-dependent instructions to slice buffer • Buffer stores in store buffer Poison Tail RF1 r2 Poison r3 RF0 (Tail) r5 F1 E1 I$ D$ D1 Store Buffer

13. Advance A1 B1 C1 Slice Buffer • Propagate poison through data dependences • Divert miss-dependent instructions to slice buffer • Buffer stores in store buffer • Miss-independent instructions execute as usual Poison RF1 Tail r2 Poison r3 RF0 (Tail) r5 A2 F1 E1 I$ D$ D1 Store Buffer

14. Advance Miss-independent instruction (green) A1 B1 C1 D1 Slice Buffer • Propagate poison through data dependences • Divert miss-dependent instructions to slice buffer • Buffer stores in store buffer • Miss-independent instructions execute as usual, update regfile Poison RF1 r2 E1 Tail Poison r3 RF0 (Tail) r5 B2 A2 F1 I$ D$ D1 Store Buffer

15. Advance A1 B1 C1 D1 Slice Buffer • Propagate poison through data dependences • Divert miss-dependent instructions to slice buffer • Buffer stores in store buffer • Miss-independent instructions execute as usual, update regfile Poison RF1 r2 E1 Poison r3 RF0 (Tail) Tail r5 B2 B2 A2 I$ D$ D1 Store Buffer

16. Advance A1 B1 C1 D1 Slice Buffer • Propagate poison through data dependences • Divert miss-dependent instructions to slice buffer • Buffer stores in store buffer • Miss-independent instructions execute as usual, update regfile • Can “un-poison” tail registers Poison RF1 E1 A2 Poison r3 RF0 (Tail) r5 Tail D2 C2 B2 I$ D$ D1 Store Buffer

17. Miss Returns Fill A1 B1 C1 D1 Slice Buffer When A1 miss returns, transition to “rally” • Stall fetch • Pipe in contents of slice buffer Poison RF1 E1 A2 B2 Poison RF0 (Tail) r5 E2 D2 C2 Tail I$ D$ D1 Store Buffer

18. Drain B1 C1 D1 Slice Buffer • Drain advance instructions already in pipeline (C2–D2) Poison RF1 E1 A2 B2 C2 Poison RF0 (Tail) E2 A1 I$ D$ Tail D2 D1 Store Buffer

19. Drain C1 D1 Slice Buffer • Drain advance instructions already in pipeline (C2–D2) Poison RF1 E1 A2 B2 C2 Poison RF0 (Tail) E2 B1 A1 I$ D$ D2 D1 Tail Store Buffer

20. Rally D1 Slice Buffer Rally • Complete deferred instructions from slice buffer Poison RF1 E1 A2 B2 C2 Poison RF0 (Tail) E2 C1 B1 I$ D$ D2 D1 Tail Store Buffer

21. Rally Slice Buffer Rally • Execute deferred instructions from slice buffer • When slice buffer is empty, un-block fetch Poison RF1 E1 A2 B2 C2 Poison RF0 (Tail) E2 D1 C1 I$ D$ D2 D1 Tail Store Buffer

22. Rally Slice Buffer Wait for deferred instructions to complete Poison Rally RF1 E1 A2 B2 C2 Poison RF0 (Tail) F2 E2 I$ D$ D2 D1 Tail Store Buffer

23. Back To Normal Slice Buffer When last deferred instruction completes Poison RF1 Rally E1 A2 B2 C2 Poison RF0 (Tail) F2 E2 I$ D$ D2 D1 Tail Store Buffer

24. Back To Normal Slice Buffer When last deferred instruction completes • Release register checkpoint Poison RF1 Rally E1 A2 B2 C2 Poison RF0 (Tail) F2 E2 I$ D$ D2 D1 Tail Store Buffer

25. Back To Normal Slice Buffer When last deferred instruction completes • Release register checkpoint • Resume normal execution at the tail Poison RF1 Poison RF0 (Tail) F2 E2 I$ D$ D2 D1 Tail Store Buffer

26. Back To Normal Slice Buffer When last deferred instruction completes • Release register checkpoint • Resume normal execution at the tail • Drain stores from store buffer to D$ Poison RF1 Poison RF0 (Tail) F2 E2 I$ D$ D2 D1 Tail Store Buffer

27. One Way Or The Other Tail Slice Buffer If rally hits mis-predicted branch, exception, etc. • Flush pipeline • Discard store buffer contents • Restore regfile from checkpoint Poison RF1 Poison RF0 (Tail) A1 I$ D$ Store Buffer

28. iCFP Roadmap Motivation and overview (Not fully) working example Correctness features Register communication for miss-dependent instructions Store-load forwarding Multiprocessor safety Performance features Evaluation

29. Rally Register Communication C1 D1 Rally Slice Buffer Where do A1–C1 write r2, r3, r5 during rally? • Not in Tail RF0 • Already written by logically younger A2–C2 Poison RF1 E1 A2 B2 C2 Poison RF0 (Tail) E2 B1 A1 I$ D$ D2 D1 Tail Store Buffer

30. Rally Register Communication D1 Slice Buffer Rally Use RF1 as rally scratch-pad • Update Tail RF0 if youngest writer (not in this example) A1 Poison RF1 (Rally) E1 A2 B2 C2 Poison RF0 (Tail) E2 C1 B1 I$ D$ D2 D1 Tail Store Buffer

31. Rally Register Communication Slice Buffer Use RF1 as rally scratch-pad • Update Tail RF0 if youngest writer (not in this example) Rally A1 B1 Poison RF1 (Rally) E1 A2 B2 C2 Poison RF0 (Tail) E2 D1 C1 I$ D$ D2 D1 Tail Store Buffer

32. Store-Load Forwarding iCFP is in-order but … Rally loads out-of-order wrt advance stores (possible WAR hazards) Store-load forwarding mechanism should Avoid WAR hazards Avoid redoing stores Forwarding cache? D$ with speculative writes? Not what we want What we really want is a large (64-entry+) store queue Like in an out-of-order processor Associative search doesn’t scale nicely

33. Root (SSN) 86 85 84 83 82 81 80 … … AC 85 B0 86 64-entries B4 83 B8 21 link 44 81 0 15 0 77 0 … … Chained Store Buffer Tail (younger) Head (older) address 7B0 2AC 388 1B4 384 1AC 380 value 90 78 ?? 56 ?? 34 12 poison 0 0 1 0 1 0 0 Replace associative search with iterative indexed search • Exploit fact that stores enter store buffer in order • Address must be known: otherwise stall • Overlay store buffer with address-based hash table

34. 85 81 AC 85 2AC 1AC 81 Match, forward Chained Store Buffer Tail (younger) Head (older) Root … … (SSN) 86 85 84 83 82 81 80 AC 85 address 7B0 2AC 388 1B4 384 1AC 380 B0 86 value 90 78 ?? 56 ?? 34 12 64-entries B4 83 poison 0 0 1 0 1 0 0 B8 21 link 44 81 0 15 0 77 0 … … Loads follow chain starting at appropriate root table entry • For example, load to address1AC

35. 83 1B4 B4 83 15 Younger store, ignore Go to D$ Chained Store Buffer Tail (younger) Head (older) Root … … (SSN) 86 85 84 83 82 81 80 AC 85 address 7B0 2AC 388 1B4 384 1AC 380 B0 86 value 90 78 ?? 56 ?? 34 12 64-entries B4 83 poison 0 0 1 0 1 0 0 B8 21 link 44 81 0 15 0 77 0 … … Loads follow chain starting at appropriate root table entry • For example, load to address1AC Rally loads ignore younger stores, avoid WAR hazards • For example, rally load to address1B4 … • … whose immediately older store 81 (note during advance)

36. Root (SSN) 86 85 84 83 82 81 80 … … AC 85 B0 86 64-entries B4 83 B8 21 link 44 81 0 15 0 77 0 … … Chained Store Buffer Tail (younger) Head (older) • Non-speculative (including no WAR hazards) • Scalable • Average number of excess hops < 0.05 with 64-entry root table • Must stall on (miss-dependent) stores with unknown addresses • These are rare address 7B0 2AC 388 1B4 384 1AC 380 value 90 78 ?? 56 ?? 34 12 poison 0 0 1 0 1 0 0

37. Multi-Processor Safety iCFP is in-order but … (yeah again) • Advance loads are vulnerable to stores from other threads • Just like in an out-of-order processor Must snoop/verify these • Associative load queue too expensive for in-order processor • Paper describes scheme based on local signatures

38. Methodology Cycle-level simulation • 2-way issue 9-stage in-order pipeline • 32KByte D$ • 20-cycle 1MByte, 8-way L2 (8 8-entry stream buffers) • 400 cycle main memory, 4Bytes/cycle, 32 outstanding misses • 128-entry chained store buffer, 128-entry slice buffer Spec2000 benchmarks • Alpha AXP ISA • DEC OSF compiler -04 optimization • 2% sampling with warm-up

39. SpecFP SpecINT Initial Evaluation iCFP vs. Runahead: advance on L2 misses • Roughly same performance: +10% • Dominated by MLP • iCFP’s ability to reuse work rarely significant (vortex)

40. SpecFP SpecINT Initial Evaluation Runahead advance on D$ misses too: performance drops • Chance for MLP is low and can’t reuse work • Overhead of restoring checkpoint is high • Especially because baseline stalls on use, not miss

41. SpecFP SpecINT Initial Evaluation iCFP advance under D$ misses too • Can reuse work without restoring checkpoint but … • iCFP* executes rallies until completion in blocking fashion • No efficient way to handle D$ misses under L2 misses

42. iCFP Performance Features Non-blocking rallies • Miss during rally (dependent or just pending)? Don’t stall, slice it out Fine-grain multi-threaded rallies • Proceed in parallel with advance execution at the tail • Rallies process dependence chains, can’t exploit superscalar These need: incremental updates of tail register state • Both values and poison bits • Note: store buffer is not a tail snapshot, so no additional support

43. Incremental Tail Updates C1 D1 Rally Slice Buffer A2 B2 Question: should current rally instruction update Tail RF? • A1? B1? C1? • No, no, yes Poison RF1 (Rally) r2 E1 Poison r3 RF0 (Tail) r5 C2 B1 A1 Tail I$ D$ D1 Store Buffer

44. Seq 7 Seq 8 3 Incremental Tail Updates C1 D1 Rally Slice Buffer A2 B2 Advance execution tags registers with sequence numbers • Distance of writing instruction from checkpoint Poison RF1 (Rally) r2 E1 Poison r3 RF0 (Tail) r5 C2 B1 A1 Tail I$ D$ D1 Store Buffer

45. Seq 7 Seq 8 3 A1’s is 1, so no Incremental Tail Updates D1 A2 Slice Buffer B2 Rally Rally updates Tail RF if seqnum matches A1 Poison RF1 (Rally) r2 E1 Poison r3 RF0 (Tail) r5 C2 C1 B1 Tail I$ D$ D1 Store Buffer

46. Seq 7 Seq 8 3 B1’s is 2, so no Incremental Tail Updates Slice Buffer A2 B2 Rally updates Tail RF if seqnum matches Rally A1 B1 Poison RF1 (Rally) r2 E1 Poison r3 RF0 (Tail) r5 C2 D1 C1 Tail I$ D$ D1 Store Buffer

47. Seq 7 Seq 8 3 C1’s is 3, so yes Incremental Tail Updates Slice Buffer A2 B2 Rally updates Tail RF if seqnum matches A1 B1 C1 Poison Rally RF1 (Rally) r2 E1 C1 Poison r3 RF0 (Tail) r5 D2 C2 Tail I$ D$ D1 Store Buffer

48. Seq 7 Seq 8 3 Incremental Tail Updates Slice Buffer A2 B2 Rally updates Tail RF if seqnum matches A1 B1 C1 Poison RF1 (Rally) Rally r2 E1 C1 Poison r3 RF0 (Tail) E2 D2 C2 Tail I$ D$ D1 Store Buffer

49. Seq 7 Seq 8 9 C2 sliced because r3 poison preserved Incremental Tail Updates Slice Buffer A2 B2 C2 Proper slicing can continue at tail A1 B1 C1 Poison RF1 (Rally) r2 E1 C1 Poison r3 RF0 (Tail) r5 F2 E2 D2 I$ D$ Tail D1 Store Buffer

50. Another iCFP Performance Feature Minimal rallies • Only traverse slice of returned miss, not entire slice buffer Implementation: borrow trick from TCI [AlZawawi+, ISCA’07] • Replace poison bits with bitvectors • Re-organize slice buffer to support sparse access • See paper for details