Dynamic Verification of Cache Coherence Protocols

1. Dynamic Verification of Cache Coherence Protocols Jason F. Cantin Mikko H. Lipasti James E. Smith

2. Introduction • Multiprocessors are used for a variety commercial and mission-critical tasks • Reliability is a growing concern • Coherence is a fundamental feature of shared-memory MPs • High design complexity • Relatively low interconnect reliability Workshop on Memory Performance Issues

3. Introduction:Cache Coherence Protocols • Notoriously difficult to design and verify • Often conceptually simple, but with complex implementations for efficiency and handling special cases • Multiple finite state machines operating concurrently Workshop on Memory Performance Issues

4. Introduction:A Simple Example MSI Protocol “Architected” State • Invalid / Not Present • Shared (readable) • Modified (read/write) Bus_RdX, Replace I S Read Bus_Rd Write Bus_RdX, Replace Write M Workshop on Memory Performance Issues

5. Introduction:Simple Example with a Bus MSI Protocol “Implementation” State • Transient states for pending operations • Arcs to satisfy requests while operations pending I S Bus_RdX, Replace Read Bus_Av Bus_Av Pend Rd I Write Pend WB M Bus_RdX Write Pend RdX I Bus_Av Bus_RdX Replace M Pend RdX S Bus_Av Bus_Rd Workshop on Memory Performance Issues

6. Problem • In practice, implementations can have dozens of states • Atomic memory operations • Split transaction buses • Protocol optimizations • Complexity grows exponentially with added states • Random testing: Low Coverage • Exhaustive testing: Too time consuming Workshop on Memory Performance Issues

7. Dynamic Verification Check the implementation at runtime • It is easier to check a computation than to do the actual computation, provided there is a delay between the computation and the check (Rotenberg, AR-SMT) • Simplified version of a processor implementation can be used for online verification (Austin, DIVA) Workshop on Memory Performance Issues

8. Dynamic Verification of Cache Coherence A distributed form of dynamic verification for multiprocessor memory systems • Simplified version of protocol added to each node • Maintains architected state • Check completed transitions and actions against simple protocol • Additional messages (assertions) sent between nodes to ensure coherence Workshop on Memory Performance Issues

9. Single, centralized check processor Receives instructions serially in program order from implementation Conceptual View for Superscalar Processors (DIVA) Physical registers Arch. registers Complex Execution Processor Prediction Tables R.O.B. Committed results Arch. registers Check Processor Workshop on Memory Performance Issues

10. Conceptual View for Coherence Shared Logical Bus • Distributed checking hardware • Transitions received in parallel, in completion order Implementation Protocol Completed Transitions Simple Protocol Shared Validation Bus Workshop on Memory Performance Issues

11. High Level Organization Memory DV-CC Checker P Cache Controller Validation bus (assertions to be checked) Shared logical bus (addresses, data, control) Workshop on Memory Performance Issues

12. Benefits • Detects hardware faults • Redundant computation • Including intermittent network failures • Detects design mistakes • Checker is simple and easy to verify Workshop on Memory Performance Issues

13. Drawbacks • Time is required for checking, but… • May be overlapped with other activities • Simple protocol requires fewer transitions • Assertions consume bandwidth • May need second bus / network • Additional hardware • But not much Workshop on Memory Performance Issues

14. DV for coherence in an SMP • Architected statestored in a second tag array • Transactions sent to the checker when architected state changes • Address • Initial State and Final States • Input (Request, Snoop Responses, etc) • Action (Send Data, Respond Shared, etc) Workshop on Memory Performance Issues

15. DV for coherence in an SMP (2) • Checker compares the initial state of a transition against the architected state • Final state and action recomputed and compared to implementation’s result • Assertions broadcast to other nodes to check coherence and confirm completion of transactions • Watchdog timer detects deadlock, livelock, and other omission failures Workshop on Memory Performance Issues

16. Checking a State Transition Transition From Implementation Protocol Address Final state Input Init. state Action Arch. Tag State Action Logic Next State Logic =? =? =? Update Tags Asst Send Buffer Error Detection / Diagnosis Watchdog timer Validation Bus Workshop on Memory Performance Issues

17. Checking an Assertion Address Arch. Tag State Remote State OK Error Detection / Diagnosis Watchdog timer Assert Recv Buffer Validation Bus Workshop on Memory Performance Issues

18. When to Broadcast Assertions For MSI: 1. I S (readable copy loaded) 2. I M (writeable copy loaded) 3. S  M (upgrade) 4. M  I (writeback) Note: The MS transition results from remote reads, and doesn’t require an extra assertion. Replacements (SI) are not considered here. Workshop on Memory Performance Issues

19. Preliminary Data (4-way SMP) Most memory references do not change cache state (checker need not have high bandwidth) Workshop on Memory Performance Issues

20. Preliminary Data (4-way SMP) Workshop on Memory Performance Issues

21. Future Work • Performance impact for a real SMP protocol implementation • In progress • Directory-based protocols • Dynamically verifying memory models • Recovery • Can stall to avoid error propagation • Can write checkpoints periodically Workshop on Memory Performance Issues

22. In Summary • Dynamic verification can be applied to multiprocessor systems (in a distributed manner) • Improves fault-tolerance, and design verification may be relaxed • More to come Workshop on Memory Performance Issues