1 / 20

Dynamic Runtime Testing for Cycle-Accurate Simulators

Dynamic Runtime Testing for Cycle-Accurate Simulators. Saša Tomić, Adri án Cristal, Osman Unsal , Mateo Valero. Barcelona Supercomputing Center (BSC) Universitat Polit ecnica de Catalunya (UPC). Can we trust the simulator-based evaluations?. Typical simulator evaluation:

ros
Download Presentation

Dynamic Runtime Testing for Cycle-Accurate Simulators

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Dynamic Runtime Testing for Cycle-Accurate Simulators Saša Tomić, Adrián Cristal,OsmanUnsal, Mateo Valero Barcelona Supercomputing Center (BSC) UniversitatPolitecnica de Catalunya (UPC)

  2. Can we trust the simulator-based evaluations? • Typical simulator evaluation: Make a simulator REPEAT { Debug Simulate } UNTIL: the results make sense (intuition!) • Discard and ignore the failed simulations • Are there any bugs left?

  3. Verifying the simulators • Verification is important! • Industry puts significant resources • Testing and Verification 50-70% of the costs • Mission critical application even 90% of the costs • Academia puts less resources • Why do we have bugs? • Simulators are complex • Proposed extensions are often complex • The extensions may uncover existing bugs

  4. Simulator bugs • Timing bugs • Incorrect estimation of the execution time • Simulation terminates without obvious errors • Needs other types of testing and verification • Functional bugs • Incorrect implementation of functional units • Simulations may or may not terminate correctly • Our target

  5. Outline • Examples of functional bugs • An overview of the Dynamic Simulator Testing methodology • Use Cases of Dynamic Simulator Testing • Performance evaluation • Conclusions

  6. Example: a bug in the cache coherence protocol Bug: X should be X = 10+20 = 30 X += 10 X += 20 X=0 time processor 1 processor 2 Proc 1 X+=10 Proc 2 X+=20 X=0 X=10 X=0 X=20 simulator of multi-level coherent caches X=30 X=0 X=20 shared memory

  7. Example: a bug in the HTM X = 0; Atomic { X += 10; } X += 10 processor X=0 X=10 HTM simulator Bug: not committed X = 10 X=0 nothing? shared memory

  8. Detecting functional bugs • The functionality of the simulators is often simple • Can be emulated with simple emulators • The emulators can be fast and stable • Can we take an advantage of the emulators?

  9. Dynamic Testing Methodology • Add a simple, no-timing emulator • Execute each operation in the simulator and then in the emulator • Compare the executions • We compared only the memory accesses • The execution must be identical during entire simulation

  10. An overview of dynamic testing input output output input timing simulator simpleno-timing emulator • Use the same input • Compare the outputs • Repeat for every operation! input output output input

  11. Dynamic testing a cache coherence protocol Check failed: should be X=10 X += 10 X += 20 processor 1 processor 2 X=20 X=0 X=10 X=0 X=10 X=0 X=10 output input output input output input timing simulator of multi-level coherent caches STL map X=0 X=10 input output output input X=0 X=0 X=20 shared memory

  12. Dynamic testing of an HTM processor TX_Begin; X += 10; TX_Commit; X=10 X=0 X=0 X=10 input output output input timing simulator of an HTM STL map per TX X=10 X=0 input output output input Check failed: should commit X=10 X=0 X=??? X=10 X=0 shared memory

  13. Other Use Cases • Out-Of-Order or pipelined processor • With a processor emulator, e.g., QEMU • Complex memory hierarchy • With an STL map • Incoherent multilevel memory hierarchy • W/ multiple STL maps, one per memory hierarchy • System-On-Chip, Routing Protocols, etc. • Simple emulators of the functionalities

  14. Performance Evaluation • Implemented on 4 HTMs with lazy and eager version management • Implemented for a directory-based cache-coherence protocol • Baseline: M5 full-system simulator

  15. Performance evaluationOS booting

  16. Performance evaluationapplications

  17. Our experience with Dynamic Testing • Reduced the time spent on writing tests • Faster debugging • Detects most bugs “in minutes” • Eliminating a bug takes tens of minutes instead of hours/days/weeks/… • Shortened the total simulator development from12-18 months to 3-4 months

  18. Conclusions • Presented the Dynamic Simulator Testing • Detects the functional bugs in Cycle-Accurate Simulators • Modest reduction of simulator performance

  19. Thanks! SashaTomić sasa.tomic@bsc.es Dynamic Runtime Testing for Cycle-AccurateSimulators

  20. Cache/HTM emulator implementation address line data STL map (dictionary) address line data address line data address line data . . . . . .

More Related