Hob: A Tool for Verifying Data Structure Consistency

1. Patrick Lam, Viktor Kuncak, Martin Rinard MIT CSAIL Massachusetts Institute of Technology Cambridge, MA 02139 Hob: A Tool for Verifying Data Structure Consistency

2. Goal Analyze Program to Verify Data Structure Consistency Properties • Local Properties (within a single data structure) for example, e in List. e.next.prev = e • Global Properties (multiple data structures) for example, no object in list & array

3. Challenges Scalability • Detailed static analyses are expensive! Diversity • No single analysis could possibly handle all properties of interest!

4. Multiple Modular Analyses Solution:

5. Inside Outside Applying Multiple Modular Analyses Two views of data structures:

6. Inside: full-complexity annotations analyzed using arbitrary analysis plugin Outside: only sets in public module interfaces all analysis plugins communicate via set specifications Applying Multiple Modular Analyses Two views of data structures:

7. Outside Module Interfaces Public module interfaces are set-based. For instance, the add procedure’s interface is: proc add(p:Item) requires p not in List modifies List ensures List’ = List  {p} add() ensures that p belongs to the list after execution.

8. Benefits of a Set Specification Language Captures important data structure aspects Captures interface requirements Provides productive perspective on program

9. Verifying“e in List. e.next.prev == e” impl module List { format Item { next : Item; prev : Item; } proc add(p:Item) { if (root==null) { root = p; p.next = null; p.prev = null; } else { p.next = root; p.prev = null; root.prev = p; root = p; } } } assumes “e in List. next.prev == e” guarantees “e in List. next.prev == e”

10. Another property of add() • Adds object p to the list. • impl module List { • format Item { next : Item; prev : Item; } • proc add(p:Item) { • if (root==null) { • root = p; • p.next = null; p.prev = null; • } else { • p.next = root; p.prev = null; • root.prev = p; root = p; • } • } • }

11. Verifying add() proc add(p) implementation concrete state • Preserves invariant • e in List. e.next.prev == e. • Adds p to the linked list.

12. Understanding add()’s effects abstract set interface proc add(p) requires p not in List modifies List ensures List’ = List {p} e in List. e.next.prev == e e in List. e.next.prev == e

13. Understanding add()’s effects abstract set interface proc add(p) requires p not in List modifies List ensures List’ = List {p} e in List. e.next.prev == e e in List. e.next.prev == e But what do abstract sets mean?

14. Assigning Meanings to Sets abstract set e in List. e.next.prev == e abstraction function concrete state

15. Verifying Data Structure Consistency abstract set interface proc add(p) requires p not in List modifies List ensures List’ = List {p} e in List. e.next.prev == e e in List. e.next.prev == e abstraction function proc add(p) implementation concrete state

16. What Happens Next? invariant translated interface proc add(p) e in List. e.next.prev == e requires p not in root<next*> ensures root<next*>’ = root<next*> {p}  frame  analysis plugin  proc add(p) implementation

17. Hob Demonstrations I

18. Client code: Loops Removing all items from a linked list: proc clear() requires init & card(Content’) >= 1 & (Iter in Content) modifies Content, Iter ensures card(Content’) = 0 & card(Iter’) = 0;

19. Loop Implementation Implementation: proc clear() { bool e = isEmpty(); while “e’  card(Content’) = 0” { removeFirst(); e = isEmpty(); } }

20. Demo: Verifying Loops

21. Analysis Plugins Hob currently contains three analysis plugins: • Flags plugin • verifies generalized typestate properties • PALE plugin • verifies linked list data structure properties • Theorem proving plugin • verifies arbitrary program properties

22. Verifying More Detailed Properties proc add(p) implementation concrete state • Preserves invariant o.next.prev == o. • Adds p to the linked list.

23. Defining the Linked List abst module List { analysis PALE; S = { p : Entry | root<next*>p}; • PALE analysis works with data structures that have a backbone and routing pointers next next

24. abst module List { analysis PALE; S = { p : Entry | root<next*>p}; invariant type L = { data next : L; data next : L says that the backbone consists of the next references of the objects Defining the Linked List next next

25. abst module List { analysis PALE; S = { p : Entry | root<next*>p}; invariant type L = { data next : L; pointer prev : L [this^L.next = {prev}]; prev is a routing pointer in the data structure prev is the inverse of next So p.next.prev = p.prev.next = p Defining the Linked List prev prev next next

26. Defining the Linked List abst module List { analysis PALE; S = { p : Entry | root<next*>p}; invariant type L = { data next : L; pointer prev : L [this^L.next = {prev}]; }; invariant data root : L; } root is the root of a data structure of L’s prev prev root next next

27. impl module List { proc add(p:Item) { if (root==null) { root = e; e.next = null; e.prev = null; } else { e.next = root; e.prev = null; root.prev = e; root = e; } } } spec module List { proc add(p:Item) requires p not in S modifies S ensures S’ = S  {p}; } Procedure Specifications and Implementations

28. Demo: Verifying a Linked List

29. Flags plugin For this plugin, sets are defined by field values: “All Entry objects whose field s is set to 2” is S = {x : Entry | “x.s = 2” } We use dataflow analysis, propagating boolean formulas and verifying them with MONA.

30. Flags plugin abst module Simple_flag { use plugin “flags”; S = {x : Entry | “x.s = 2” } }

31. Flags plugin impl module Simple_flag { format Entry { s:int; } proc makeS(e:Entry) { e.s = 2; } } abst module Simple_flag { use plugin “flags”; S = {x : Entry | “x.s = 2” } } spec module Simple_flag { format Entry; specvar S : Entry set; proc makeS(e:Entry) modifies S ensures S’ = S + e; }

32. Demo: Using the Flag Plugin

33. Verifying Even More Detailed Properties Sometimes shape analysis just isn’t enough. Ought to be able to do something! s c

34. Solution: Theorem Proving Plugin

35. A Theorem Proving Example S = { x : Node | “EX i. 0 <= i & i < s & x=d[i] & x ~= null”} s c

36. Demo: Theorem Proving

37. Global Data Structure Consistency Properties

38. Global Data Structure Consistency Properties Consider a process scheduler with a set of Running processes and a set of Idle processes. Running: all processes in running list Idle: all processes in idle array Necessary invariant P: RunningIdle =  Running Idle

39. Where must Hob check that P holds? requires P ensures P Process Scheduler

40. Where else must Property P hold? P P

41. What else must hold there? PQ P Q Q

42. Problem: Specification Aggregation! P Q R Q R Q P  Q P Q R Q Q R • Specifications become unwieldy • Explicit callee preconditions break modularity

43. Beating Specification Aggregation These invariants crosscut the specifications for the entire program, and would need to be included in all potential callers. Scopes enable developers to factor out these invariants and state the invariant once and for all.

44. Our Solution to Specification Aggregation The developer: • Specifies a consistency property • Identifies a region where property may be temporarily violated during updates The Hob analysis system: • Verifies the property in the program’s initial state and upon on exit from the region; and • May soundly assume the property holds upon entry to the region and outside the region

45. Scopes in Example Scheduler Module suspend(p) • RunningIdle =  may be violated anywhere within Scheduler, Idle Process, or Running Process modules • Scheduler must coordinate operations on Idle Process and Running Process Modules • Otherwise property may become permanently violated resume(p) add(p) ins(p) empty() rem(p) del() Idle Process Module Running Process Module

46. Example Scope scope C { invariant RunningIdle = ; modules scheduler, idle, running; export scheduler; } • Property holds except within modules in scope • Sets of property included in modules in scope • Outside scope, may only invoke procedures in exported modules

47. Scopes and Analysis System conjoins property to preconditions and postconditions of exported modules Analysis verifies that 1) property holds initially and 2) procedures preserve property Scheduler Module suspend(p) RunningIdle =  resume(p) add(p) ins(p) empty() rem(p) del()

48. Benefits of Scopes Specifications may omit many properties that are locally irrelevant (aiding concision). In particular, can omit crosscutting callee preconditions at caller sites (aiding modularity).

49. Benefits of Scopes Specifications may omit many properties that are locally irrelevant (aiding concision). In particular, can omit crosscutting callee preconditions at caller sites (aiding modularity).

50. Demo: Scopes in Action