AVIO: Detecting Atomicity Violations

1. AVIO: Detecting Atomicity Violations via Access Interleaving Invariants Shan Lu, Joseph Tucek, Feng Qin, Yuanyuan Zhou Opera Group, UIUC http://opera.cs.uiuc.edu

2. Synchronization Bugs • Definition • Prevails in multi-thread/proc. programs • Programmers are used to sequential thinking • High severity • Hard to repeat/detect/diagnosis • Things may become worse • Multi-core architecture • More multi-threading applications

3. Previous work focuses on Data Race • Data Race Detection • Definition • Approaches • Lockset algorithms • Use lock to protect against conflict access • Happen-before algorithms • Maintain strict order among conflict accesses

4. Lock l Lock l Lock l Unlock l Unlock l Unlock l Data race is not enough • Data race free does not mean correct temp = x; x = temp + 1; temp2 = x; x = temp2 + 1;

5. Lock l Lock l Unlock l Unlock l Data race is not enough • Data race free does not mean correct • Data race does not mean incorrect • Other problems with existing tools • Synchronization semantic dependent While (x == 0) { } x = 1;

6. Atomicity Violation Detection • Definition • Why atomicity violation is important • Directly related to program correctness • Lock is one way to enforce atomicity correctness • Fit fore future transactional memory

7. Two key challenges --in atomicity violation detection • When should we check atomicity • Not all code region should be atomic • How to judge atomicity • What execution sequence is atomic? • State-of-the-art

8. What AVIO does • Automatic atomicity violation detection • No manual annotation • No (much less) heuristics • No synchronization semantic reliance • Practically usable

9. Outline • AVIO idea • Use invariance to infer atomicity intention • Use access-interleaving serializability to check atomicity • AVIO design • Implementation • Result • Future work

10. What code ‘should’ be atomic While (x == 0) { } • Two examples • Atomicity may be NOT welcomed • Atomicity may be not required • Some cases are ‘do-not-care’ • Previous work • Manual specification • Heuristics • Code analysis • AVIO • Follow programmers’ intention! We don’t mean annotation! x = 1;

11. Programmers’ intention and atomicity behavior • Code region used for synchronization, etc. • ALWAYS NOT atomic • Code region, whose atomicity is not important • SOMETIMES NOT atomic • Code region, intended to be atomic • ALWAYS atomic (properly protected) • RARELY NOT atomic (not protected, but make programmers make wrong assumption)

12. Use invariant to infer atomicity intention If a code region is executed atomically in all correct runs, it is probably intended to be so • Training • Learn which code region always holds atomicity • Detection • Monitor above region to detect invariance violation

13. Use invariant to infer atomicity intention • Two questions about this idea • Is ‘atomicity’ a proper invariance? • Is training feasible? If a code region is executed atomically in all correct runs, it is probably intended to be so

14. Idea Feasibility • Execution Atomicity can be a good invariance • It reflects execution correctness • It can be reduced to an easy-to-check property • Access-interleaving serializability • Training is feasible • Most runs are correct • The unique hard-to-repeat property of synchronization bugs • We can label correct/incorrect runs • Training is easy • non-determinism described later illustrated in experiments

15. Access-Interleaving serializability • Atomicity violation  Non-serializable • General atomicity is not easy to judge • AVIO focuses on a basic scenario • One shared memory location • Two threads • Relatively easy to analyze • Most prevalent • Building blocks for more complicated cases

16. Access serializability analysis Thread 1 Thread 2 Thread 1 Thread 2 Read x Read x Read x Read x Read x Write x Case 0 Case 4 Write x Read x Read x Write x Read x Write x Case 1 Case 5 Read x Write x Write x Read x Write x Read x Case 6 Case 2 Write x Write x Read x Write x Write x Write x Case 3 Case 7 Case 7

17. Access serializability analysis (i) Thread 1 Thread 2 Read x Read x Read x T1:Read1 x T2: Read x T1:Read2 x T1:Read1 x T1:Read2 x T2: Read x Case 0 Case 0 serializable T1:Write1 x T2: Read x T1:Read2 x T1:Write1 x T1:Read2 x T2: Read x Write x Read x Read x Case 1 Case 1 serializable T1:Read1 x T2: Write x T1:Read2 x T1:Read1 x T1:Read2 x T2: Write x Read x Write x Read x Case 2 Case 2 non-serializable T1: Read1 and T1:Read2 have different view Write x Write x Read x T1:Write1 x T2: Write x T1:Read2 x T1:Write1 x T1:Read2 x T2: Write x Case 3 Case 3 non-serializable T1: Read2 fails to get local write result

18. Access serializability analysis (ii) Thread 1 Thread 2 Read x Read x Write x T1:Read1 x T2: Read x T1:Write2 x T2: Read x T1:Read1 x T1:Write2 x Case 4 Case 4 serializable T1:Write1 x T2: Read x T1:Write2 x T1:Write1 x T1:Write2 x T2: Read x Write x Read x Write x Case 5 Case 5 non-serializable T2: Read reads intermediate results T1:Read1 x T2: Write x T1:Write2 x T1:Read1 x T1:Write2 x T2: Write x Read x Write x Write x Case 6 Case 6 non-serializable T1: write use stale value from T1:read1 Write x Write x Write x T1:Write1 x T2: Write x T1:Write2 x T2: Write x T1:Write1 x T1:Write2 x Case 7 Case 7 serializable

19. A-I Invariance Summary • Access-Interleaving Invariance held by • Two local instructions • Accesses to the same address • Never un-serializably interleaved in correct runs • We call them P-Instruction, I-Instruction • Get A-I Invariance during training • Monitor A-I Invariance to detect atomicity violation

20. Outline • AVIO idea • Use invariance to infer atomicity intention • Use access-interleaving serializability to check atomicity • AVIO design • Violation detection protocol • Training protocol • Implementation • Result • Future work

21. Violation Detection Algorithm • Violation detection is to differentiate non-serializable cases from serializable cases Read x Read x Read x Read x Read x Write x #0 #4 Write x Read x Read x Write x Read x Write x #1 #5 Read x Write x Read x Read x Write x Write x #2 #6 Write x Write x Read x Write x Write x Write x #3 #7

22. Type of P-Instruction Type of I-Instruction Detection Protocol I-Instruction #0 #4 P:Read x r: Read x I: Read x P: Read x r: Read x I: Write x write read #4,5,6,7 #0,1,2,3 #1 #5 P: Write x r: Read x I: Read x P: Write x r: Read x I: Write x read write #4,6 #5,7 #2 #6 Any interleaving Remote Read Gets P-write result? Any interleaving Remote Write? P: Read x r: Write x I: Read x P: Read x r: Write x I: Write x Yes Yes No No #3 #7 P: Write x r: Write x I: Read x P: Write x r: Write x I: Write x BUG! Pass! BUG! Pass! #2,3,6 #0,1,4 #5 #7 AVIO Detection Diagram

23. Training Protocol • Training • Collect instructions which are never un-serializably interleaved from predecessors AVIO-IE (Programbinary P) { AISet = all global memory accesses in P; while (AISet is changing in the last m iterations){ ViolationSet = RunOnceWithAvioDetection (P,AISet); AISet = AISet– ViolationSet; } AISet = AISet– NonTouched Instructions; }

24. Outline • AVIO idea • AVIO design • Violation detection protocol • Training protocol • Implementation • AVIO-H • AVIO-S • Result • Future work

25. AVIO-H • Essence • turn the AVIO detection protocol to hardware Type of I-Instruction L1 Cache Line INV DG PI write read An I-Instruction Type of P-Instruction Type of This instr. read write write read Any interleaving Remote Read Gets P-write result? Any interleaving Remote Write? PI Yes Yes No No 0 1 INV DG BUG! Pass! BUG! Pass! 0 1 1 0 #2,3,6 #0,1,4 #5 #7 BUG! BUG! AVIO Detection Diagram

26. AVIO-S • Similarly follow the general protocol • Use instrumentation to monitor all shared memory access • PIN tool • Maintain protocol required information in hash tables

27. Outline • AVIO idea • AVIO design • Implementation • Result • Methodology • Functionality Results • Performance Results • Sensitivity Study • Future work

28. Methodology (i) • 5 real bugs from 2 server applications • 4 SPLASH2 benchmarks • fft, lu, radix, fmm

29. Methodology (ii) • Simics/Flexus simulation • Software implementation based on PIN • Compare with • Lockset (Valgrind-lockset) • Happen-before* • SVD*

30. Results – Functionality (i) • Able to detect most tested real bugs

31. Results – Functionality (ii) • Few false positives

32. Results -- Performance • Detection overhead comparison • Training overhead • Similar single-run overhead as detection

33. Results -- Sensitivity • Splash2 • MySQL

34. Conclusion and Future Work • Contribution • AI Invariance • AVIO tool • Future work • More real bugs!! • Extend to multi-variable atomicity

35. Thanks!

36. Backup

37. Real bug examples (Case #2)

38. Real bug examples (Case #3)

39. Real bug examples (Case #5)

40. Synchronization Bugs simple example • Definition • Prevails in multi-thread/proc. programs • Programmers are used to sequential thinking Thread 1 x++ Thread 2 x++ x ++ temp1 = x x = temp1 + 1 temp2 = x x = temp2 + 1 temp = x x = temp + 1 Wrong x results!