Applications of TVLA

1. Applications of TVLA Mooly Sagiv Tel Aviv University Shape Analysis with Applications http://www.cs.tau.ac.il/~rumster/TVLA/

2. Outline • Issues • Complexity of TVLA • Weak vs. Strong Updates • Cleanness • Null derefernces • Memory leaks • Freed storage (homework) • The concurrent modification problem • Partial correctness • Sorting • GC • Total Correctenss • Flow dependences • Multithreading • Other

3. Complexity of Shape Analysis x = malloc() if (…) y1 = x if (…) y2 = x if (…) y3 = x  if (…) yn =x

4. Complexity of TVLA analysis • Maximal number of Nodes in Blurred Structures • 3|A| • Size of 3-valued structure representation • Action cost • Focus • Precondition • Coerce • New • Update • Coerce • Blur

5. Weak vs. Strong Updates if (…) x = y else x = z xn = NULL

6. Detecting Incorrect Library Usages(J. Field, D. Goyal. G. Ramalingam, A. Warshavski) • Java provides libraries for manipulating data structures • Collections • Lists • Hashset • … • Iterators over collections allows sequential accesses • Statically detect incorrect library usages Set s = worklist.unprocessedItems(); for (Iterator i = s.iterator(); i.hasNext()){ Object item = i.next(); if (...) processItem(item);

7. The Concurrent Modification Problem • Static analysis of Java programs manipulating Java 2 collections • Inconsistent usages of iterators • An Iterator object i defined on a collection object c • No use of i may be preceded by update to the contents of c,unless the update was also made via I • Guarantees order independence

8. Artificial Example Set v = new Set(); Iterator i1 = v.iterator(); Iterator i2 = v.iterator(); Iterator i3 = i1; i1.next(); i1.remove(); if (...) { i2.next(); } if (...) { i3.next(); } v.add("..."); if (...) { i1.next();}

9. class Make { private Worklist worklist; public static void main (String[] args) { Make m = new Make(); m.initializeWorklist(args); m.processWorklist(); } void initializeWorklist(String[] args) { ...; worklist = new Worklist(); ... // add some items to worklist} void processWorklist() { Set s = worklist.unprocessedItems(); for (Iterator i = s.iterator(); i.hasNext()){ Object item = i.next(); if (...) processItem(item); } } void processItem(Object i){ ...; doSubproblem(...);} void doSubproblem(...) { ... worklist.addItem(newitem); ... } } public class Worklist { Set s; public Worklist() {. ..; s = new HashSet(); ... } public void addItem(Object item) { s.add(item); } public Set unprocessedItems() { return s; } } return rev; }

10. Static Detection of Concurrent Modifications • Statically Check for CME exceptions • Warn against potential CME • Sound (conservative) solution • Not too many false alarms • Coding in TVLA • Operational Semantics • Vanilla solution is Imprecise (and inefficient) • Derive instrumentation predicates • Java to TVP front-end • Extract potentially relevant client code

11. CME specification in Java’ class Version { /* represents distinct versions of a Set */ } class Collection { Version version; Collection() { version = new Version(); } boolean add(Object o) { version = new Version(); } Iterator iterator() { return new Iterator(this); } } class Iterator { Collection set; Version definingVersion; Iterator (Collection s) { definingVersion = s.version; set = s; } void remove() { requires (definingVersion == set.version); set.ver = new Version(); definingVersion = set.version; } Object next() { requires (definingVersion == set.version); } }

12. Vanilla TVLA Encoding • Local iterators are pointers • Unary predicates • Relevant fields are pointer selectors • Binary predicates

13. Artificial Example Set v = new Set(); Iterator i1 = v.iterator(); Iterator i2 = v.iterator(); Iterator i3 = i1; i1.next(); i1.remove(); if (...) { i2.next(); } if (...) { i3.next(); } v.add("..."); if (...) { i1.next();}

14. Improved TVLA Encodings • Use reachability • Explicitly maintain relevant information • valid[i] = i.defVersion == i.set.Version • iterOf[i, v] = i.set == v • mutex[i, j] = i.set ==j.set && i != j • same[v, w] == v == w • Can be automatically derived from the specification • Polynomial complexity in programs where iterators are not stored in the client heap Meet over all path solution • Adaptive to programs with client heap

15. Empirical Results

16. Partial Correctness • {P} S {Q} • How to derive loop invariants • Abstract interpretation provides a sound solution • The abstract domain represents a class of program invariants

17. Example Sorting of linked lists typedef struct node { struct node *n; int data; } *Elements; • dle(v1, v2) = v1.data  v2.data • inOrder[n, dle](v) = v1: n(v, v1) dle(v, v1) • inROrder[n, dle](v) = v1: n(v, v1) dle(v1, v) • Captures intermediate invariants as well

18. L insert_sort(L x) { L r, pr, rn, l, pl; r = x; pr = NULL; while (r != NULL) { l = x; rn = r ->n; pl = NULL; while (l != r) { if (l->data > r->data) { pr->n = rn; r->n = l; if (pl == NULL) x = r; else pl->n = r; r = pr; break; } pl = l; l = l->n; } pr = r; r = rn; } return x; } typedef struct node { struct node *n; int data; } *Elements;

19. n n x inOrder[dle,n]=½ r[n,x] inOrder[dle,n]=1/2 r[n,x] dle dle L insert_sort(L x) { L r, pr, rn, l, pl; … return x; } n x inOrder[dle,n] r[n,x] inOrder[dle,n] r[n,x] dle dle

20. /*pred.tvp */ foreach (z in PVar) { %p z(v_1) unique box } %p n(v_1, v_2) function %i is[n](v) = E(v_1, v_2) ( v_1 != v_2 & n(v_1, v) & n(v_2, v)) foreach (z in PVar) { %i r[n,z](v) = E(v_1) (z(v_1) & n*(v_1, v)) } %i c[n](v) = n+(v, v) %p dle(v1, v2) reflexive transitive %i inOrder[dle,n](v) = A(v_1) n(v, v_1) -> dle(v, v_1) nonabs %i inROrder[dle,n](v) = A(v_1) n(v, v_1) -> dle(v_1, v) nonabs %r !dle(v_1, v_2) ==> dle(v_2, v_1)

21. /* cond.tvp */ %action uninterpreted() { %t "uninterpreted" } %action Is_Not_Null_Var(x1) { %t x1 + " != NULL" %f { x1(v) } %p E(v) x1(v) } %action Is_Null_Var(x1) { %t x1 + " == NULL" %f { x1(v) } %p !(E(v) x1(v)) } %action Is_Eq_Var(x1, x2) { %t x1 + " == " + x2 %f { x1(v), x2(v) } %p A(v) x1(v) <-> x2(v) } %action Is_Not_Eq_Var(x1, x2) { %t x1 + " != " + x2 %f { x1(v), x2(v) } %p !A(v) x1(v) <-> x2(v) }

22. %action Greater_Data_L(x1, x2) { %t x1 + "->data > " + x2 + "->data" %f { x1(v_1) & x2(v_2) & dle(v_1, v_2) } %p !E(v_1, v_2) x1(v_1) & x2(v_2) & dle(v_1, v_2) } %action Less_Equal_Data_L(x1, x2) { %t x1 + "->data <= " + x2 + "->data" %f { x1(v_1) & x2(v_2) & dle(v_1, v_2) } %p E(v_1, v_2) x1(v_1) & x2(v_2) & dle(v_1, v_2) }

23. stat.tvp %action Set_Next_Null_L(x1) { %t x1 + "->" + n + " = null" %f { x1(v) } %message !(E(v) x1(v)) -> { n(v_1, v_2) = ... is[n](v) = ... r[n,x1](v) = ... foreach (z in PVar –{x}) { r[n, x](v) = ... } c[n](v) = inOrder[dle,n](v) = inOrder[dle,n](v) | x1(v) inROrder[dle,n](v) = inROrder[dle,n](v) | x1(v) } }

24. stat.tvp(more) %action Malloc_L(x1) { %t x1 + " = (L) malloc(sizeof(struct node)) " %new { x1(v) = isNew(v) inOrder[dle, n](v1, v2)=… inROrder[dle, n](v1, v2)=… } }

25. Abstract interpretation of if x->data <= y.data

26. From Local Outlook to Global Outlook • (Safety) Every time control reaches a given point: • there are no garbage memory cells • the list is acyclic • each cell is locally ordered • (History) The list is a permutation of the original list

27. Bugs Found • Pointer manipulations • null dereferences • memory leaks • Forget to sort the first element • Swap equal elements in bubble sort(non-termination)

28. L insert_sort_b2(L x) { L r, pr, rn, l, pl; if (x == NULL) return NULL; pr = x; r = x->n; while (r != NULL) { pl = x; rn = r->n; l = x->n; while (l != r) { if (l->d > r->d) { pr->n = rn; r->n = l ; pl->n = r; r = pr; break } pl = l; l = l->n; } pr = r; r = rn; } return x; } x n n inOrder[dle,n]=½ inOrder[dle,n]=1 dle dle dle

29. Running Times

30. Properties Not Proved • (Liveness) Termination • Stability

31. Example: Mark and Sweep void Mark(Node root) { if (root != NULL) { pending =  pending = pending  {root} marked =  while (pending ) { x = SelectAndRemove(pending) marked = marked  {x} t = x  left if (t  NULL) if (t  marked) pending = pending  {t} t = x  right if (t  NULL) if (t  marked) pending = pending  {t} } } assert(marked == Reachset(root)) } void Sweep() { unexplored = Universe collected =  while (unexplored ) { x = SelectAndRemove(unexplored) if (x  marked) collected = collected  {x} } assert(collected == Universe –Reachset(root) ) } Run Demo

32. Total Correctness • Usually more complicated • Need to show that something good eventually happens • Difficult for programs with unbounded concrete states • Example linked lists • Show decreased set of reachable locations

33. Program Dependences • A statement s1 depends on s2 if • s2 writes into a location l • s1 reads from location l • There is no intervening write in between • Useful for • Parallelization • Scheduling • Program Slicing • How to compute • Scalars • Stack pointers • Heap allocated pointers

34. Flow Dependences vs. May-Aliases int y; List p, q; q = (List) malloc(); p = q; t=p; p->d = 5; t->d = 7; y = q->d; int y; List p, q; q = (List) malloc(); p = q; p->d = 5; … y = q->d; int y; List p, q; q = (List) malloc(); p = q; t=p; p->d = 5; p=(List) malloc(); y = q->d;

35. void append() { List head, tail, temp; l1: head = (List) malloc(); l2: scanf("%c", &head->d); l3: head->n = NULL; l4: tail = head; l5: if (tail->d == `x') goto l12; l6: temp = (List) malloc(); l7: scanf("%c", &temp->d); l8: temp->n = NULL; l9: tail->n = temp; l10: tail = tail->n l11: goto l_5; l12: printf("%c", head->d); l13: printf("%c", tail->d); exit

36. /*pred.tvp */ foreach (z in PVar) { %p z(v_1) unique box } %p n(v_1, v_2) function %i is[n](v) = E(v_1, v_2) ( v_1 != v_2 & n(v_1, v) & n(v_2, v)) foreach (z in PVar) { %i r[n,z](v) = E(v_1) (z(v_1) & n*(v_1, v)) foreach (l in Label) { %p lst_w_v[l,z]() // l is the last write to into the variable z } foreach (l in Label) { %p lst_w_f[l,n](v_1) box // l is the last write to into the v_1.n %p lst_w_f[l,d](v_1) box// l is the last write to into v_1.data } } %i c[n](v) = n+(v, v)

37. Operational Semantics for Statements

38. “Read” Formulae for Statememts

39. void append() { List head, tail, temp; l1: head = (List) malloc(); l2: scanf("%c", &head->d); l3: head->n = NULL; l4: tail = head; l5: if (tail->d == `x') goto l12; l6: temp = (List) malloc(); l7: scanf("%c", &temp->d); l8: temp->n = NULL; l9: tail->n = temp; l10: tail = tail->n l11: goto l5; l12: printf("%c", head->d); l13: printf("%c", tail->d); exit lst_w_v[l1, head] lst_w_v[l10, tail] lst_w_v[l6, temp] temp head tail lst_w_f[l9,n] lst_w_f[l8,n] lst_w_f[l2, d] lst_w_f[l7,d]

40. Java Concurrency • Threads and locks are just dynamically allocated objects • synchronized implements mutual exclusion • wait, notify and notifyAll coordinate activities across threads

41. Example - Mutual Exclusion l_0: while (true) { l_1: synchronized(sharedLock) { l_C: // critical actions l_2: } l_3: } • Allocate new lock ? • Allocate new thread ? Two threads: (pc1,pc2,lockAcquired1,lockAcquired2)

42. Program Model • Interleaving model of concurrency • Program is modeled as a transition system

43. Configurations • A program configuration encodes: • global store • program-location of every thread • status of locks and threads • First-order logical structures used to represent program configurations

44. Configurations • Predicates model properties of interest • is_thread(t) • { at[lab](t) : lab  Labels } • { rval[fld](o1,o2) : fld  Fields } • held_by(l,t) • blocked(t,l) • waiting(t,l) • Can use the framework with different predicates

45. Configurations blocked is_thread at[l_1] held_by is_thread at[l_C] rval[this] blocked rval[this] is_thread at[l_1] is_thread at[l_0] is_thread at[l_0] rval[this]

46. Configurations • Program control-flow is not separately represented • Program location for each thread is encoded inside the configuration • { at[lab](t) : lab  Labels }

47. Structural Operational Semantics - actions • An action consists of: • precondition(when) formula • update formulae • Precondition formula may use a free variable ts for “currently scheduled” thread • Semantics is non-deterministic

48. lock(v) precondition tts: rval[v](ts,l)  held_by(l,t) predicate update held_by’(l1,t1) = held_by(l1,t1)  (l1 = l  t1 = ts) blocked’ (t1,l1) = blocked(t1,l1)  ((l1  l) (t1 ts)) Structural Operational Semantics - actions

49. Example: mutual exclusion t1,t2: (t1 t2)  (at[lcrit](t1)  at[lcrit](t2)) • Example: no total deadlock t,lb : is_thread(t) blocked(t, lb) Safety Properties • Configuration-local property as logical formula

50. Concrete Configuration blocked is_thread at[l_1] held_by is_thread at[l_C] rval[this] blocked rval[this] is_thread at[l_1] is_thread at[l_0] is_thread at[l_0] rval[this]