Specification-Based Error Localization

1. Specification-Based Error Localization Brian Demsky Cristian Cadar Daniel Roy Martin Rinard Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

2. Have to trace symptom back to cause Error may be present but not visible in test suite Problem Crash or Unexpected Result Execution with Broken Data Structure Error Introduced

3. Goal is to discover bugs when they corrupt data not when effect becomes visible Perform frequent consistency checks Bug localized between first unsuccessful check and last successful check Problem Crash or Unexpected Result Execution with Broken Data Structure Error Introduced

4. Our Approach Specification of Data Structure Consistency Properties Archie Compiler Efficient Consistency Checker Program + Instrumented Program with Early Data Structure Corruption Detection

5. Architecture Abstract Model Concrete Data Structure Model DefinitionRules Model Consistency Constraints

6. Architecture Rationale Why use the abstract model? • Model construction separates objects into sets • Reachability properties • Field values • Different constraints for objects in different sets • Appropriate division of complexity • Data structure representation complexity encapsulated in model definition rules • Consistency property complexity encapsulated in (clean, uniform) model constraint language

7. structure node { node *next; value *data; } structure value { int data; } node * head; List Example

8. Sets and Relations in Model • Sets of objects set NODE of node set VALUE of value • Relations between objects – values of object fields, referencing relationships between objects relation NEXT : NODE -> NODE relation DATA : NODE -> VALUE

9. Model Translation Bits translated to sets and relations in abstract model using statements of the form: Quantifiers, Condition  Inclusion Constraint true  head in NODE for n in NODE, !n.next = NULL  n.next in NODE for n in NODE, !n.next = NULL  n,n.next inNEXT for n in NODE, !n.data = NULL  n.data in VALUE for n in NODE, !n.data = NULL  n,n.data inDATA

10. Generated Model DATA NEXT VALUE NODE

11. Consistency Properties Quantifiers, Body • Body is first-order property of basic propositions • Inequality constraints on numeric fields • Cardinality constraints on sizes of sets • Referencing relationships for each object • Set and relation inclusion constraints • Example: for n in NODE, size(NEXT.n)<=1 for v in VALUE, size(DATA.v)=1

12. Consistency Violations Evaluate consistency properties for v in VALUE, size(DATA.v)=1 DATA NEXT VALUE NODE

13. Consistency Violations Evaluate consistency properties for v in VALUE, size(DATA.v)=1 DATA Inconsistency Found!!! NEXT VALUE NODE

14. Default Instrumentation void copynode(struct node *n) { struct node * newnode= malloc(sizeof(struct node)); newnode.data=n.data; newnode.next=n.next; n.next=newnode; } Insert check here Insert check here

15. Instrumentation void copynode(struct node *n) { struct node * newnode= malloc(sizeof(struct node)); newnode.data=n.data; newnode.next=n.next; n.next=newnode; } Pass Insert check here Failed Insert check here

16. Instrumentation void copynode(struct node *n) { struct node * newnode= malloc(sizeof(struct node)); newnode.data=n.data; newnode.next=n.next; n.next=newnode; } Pass Insert check here Failed Insert check here

17. Performance is a Key Issue • Would like to perform checks as often as possible • Performance of consistency checking limits how frequently program can check • Have developed compiler optimizations • Fixed point elimination • Relation elimination • Set elimination • Key idea: Perform checks directly on data structures (eliminating model when possible)

18. Fixed Point Elimination • Evaluation of model definition rules requires fixed point computation • Replace fixed point computation with more efficient traversal when possible • Compute dependence graph for model definition rules • Compute strongly connected components (SCCs) • Topologically sort SCCs • Eliminate fixed point computation for SCCs with no cyclic dependences

19. Relation Elimination Model Definition Rules: for i in 0..C, true  f[i] in S for s in S, true  s,s.r in R for s in S, !s.q=NULL  s,s.q in Q for s in S, !s.q=NULL  s.q in T Model Constraints: for s in S, MIN<=s.R and s.R<=MAX for t in T, (Q.t).R!=K

20. Relation Elimination Model Definition Rules: for i in 0..C, true  f[i] in S for s in S, true  s,s.r in R for s in S, !s.q=NULL  s,s.q in Q for s in S, !s.q=NULL  s.q in T Model Constraints: for s in S, MIN<=s.r and s.r<=MAX for t in T, (Q.t).r!=K _

21. Model Definition Rule Inlining Model Definition Rules: for i in 0..C, true  f[i] in S for s in S, !s.q=NULL  s,s.q inQ for s in S, !s.q=NULL s.q in T

22. Model Definition Rule Inlining Model Definition Rules: for i in 0..C, true  f[i] in S !f[i].q=NULL  f[i],f[i].q inQ !f[i].q=NULL f[i].q in T

23. Constraint Inlining Model Definition Rules: for i in 0..C, true  f[i] in S !f[i].q=NULL  f[i],f[i].q inQ !f[i].q=NULL f[i].q in T Model Constraints: for s in S, MIN<=s.r and s.r<=MAX for t in T, (Q.t).r!=K

24. Constraint Inlining Model Definition Rules: for i in 0..C, true  f[i] in S !f[i].q=NULL  f[i],f[i].q inQ !f[i].q=NULL f[i].q in T MIN<=f[i].r and f[i].r<=MAX Model Constraints: for t in T, (Q.t).r!=K

25. Set Elimination Model Definition Rules: for i in 0..C, true  f[i] in S !f[i].q=NULL  f[i],f[i].q inQ !f[i].q=NULL f[i].q in T MIN<=f[i].r and f[i].r<=MAX Model Constraints: for t in T, (Q.t).r!=K

26. Set Elimination Model Definition Rules: for i in 0..C, true  f[i] in S !f[i].q=NULL  f[i],f[i].q inQ !f[i].q=NULL f[i].q in T MIN<=f[i].r and f[i].r<=MAX Model Constraints: for t in T, (Q.t).r!=K

27. Freeciv Benchmark • Multiplayer Client/Server based online game • Available at www.freeciv.org • Looked at the server • Server contains 73,000 lines of code • Added 750 instrumentation sites • 20,000 consistency checks performed in our sample execution

28. Performance Evaluation • Fixed point elimination (47x speedup) • Relation construction elimination (110x speedup) • Set construction elimination (820x speedup) • Bottom line • Baseline compiled version 5,100 times slower than uninstrumented • Optimized version 6 times slower than uninstrumented • Optimized version can be used interactively

29. User Study • Designed to answer following question: Does inconsistency detection help developers to more quickly localize and correct detected data structure corruption errors?

30. User Study • Created three buggy version of Freeciv • Two groups of three developers • One used conventional tools • One used specification-based consistency checking • Each participant was asked to spend at least one hour on each version • Both populations given an instrumented version of Freeciv

31. Results

32. Extension: Data Structure Repair • Do not stop program with inconsistent data • Instead, use consistency specification to repair data structure and keep executing! • Input: inconsistent data structure • Output: consistent data structure • “Automatic detection and repair of errors in data structures” (Demsky, Rinard OOPSLA 2003) • Repair enables continued execution • All programs execute successfully after repair

33. Related Work • Specification languages such as UML or Alloy • Specification-based testing • Korat (Boyapati et. al. ISSTA 2002) • Testera (Marinov and Khurshid ASE 2001) • Eiffel (Meyer 1992) • Invariant inference and checking • Daikon (Ernst et. al. ICSE 1999) • DIDUCE (Hangal and Lam ICSE 2002) • Carrot (Pytlik et. al. 2003) • Debugging tools • AskIgor (Zeller FSE 2002) • Debugging Backwards in Time (Lewis AADEBUG 2003)

34. Conclusion • Consistency checking to localize data structure corruption bugs • Optimizations for good performance • Experimental confirmation that consistency checking may be useful • Data structure repair