Iterators Revisited: Proof Rules and Implementation

1. Iterators Revisited: Proof Rules and Implementation Bart Jacobs, Erik Meijer,Frank Piessens, Wolfram Schulte

2. Outline • Iterators in C# • How to specify and verify iterators and foreach loops? • How to prevent interference between iterators and foreach loops? • What are nested iterators? • How to implement nested iterators efficiently?

3. The Iterator pattern in C# 2.0 public interface IEnumerator<T> { T Current { get; } bool MoveNext(); } public interface IEnumerable<T> { IEnumerator<T> GetEnumerator(); }

4. Foreach Loops foreach (T x in C) S is implemented as IEnumerable<T> c = C; IEnumerator<T> e = c.GetEnumerator(); while (e.MoveNext()) { T x = e.Current; S }

5. C# 2.0 Iterator Methods IEnumerable<int> FromTo(int a, int b) { for (int x = a; x < b; x++) yield return x; } is implemented as IEnumerable<int> FromTo(int a, int b) { return new FromTo_Enumerable(a, b); } Compiler-generated class

6. C# 2.0 Iterator Methods class FromTo_Enumerator : IEnumerator<int> { int a; int b; int pc; int x; int current; public FromTo_Enumerator(int _a, int _b) { a = _a; b = _b; } public int Current { get { return current; } } public bool MoveNext() { switch (pc) { case 0: x = a; goto case 1; case 1: if (!(x < b)) goto case 4; case 2: current = x; pc = 3; return true; case 3: x++; goto case 1; case 4: pc = 4; return false; }}}

7. How to specify and verify iterators? static IEnumerable<int> FromTo(int a, int b) requires a <= b; invariantforall{int i in (0; b – a); values[i] == a + i}; invariantvalues.Count <= b – a; ensuresvalues.Count == b – a; { for (int x = a; x < b; x++) invariantvalues.Count == x – a; { yield return x; } } Enumeration invariant must be proved at start of iterator method… … and after each yield return statement. Ensures clause must be proved at end of method (and at yield break statements)

8. How to specify and verify foreach loops? int sum = 0; Seq<int> values = new Seq<int>(); foreach (int x in FromTo(1, 3)) invariant sum == Math.Sum(values); free invariantforall{int i in (0:values.Count); values[i]==1+i}; free invariant values.Count <= 3 – 1; { int x; havoc x; values.Add(x); assumeforall{int i in (0:values.Count); values[i]==1+i}; assume values.Count <= 3 – 1; sum += x; } assume values.Count == 3 – 1; assert sum == 6; while (*)

9. List<int> xs = new List<int>(); xs.Add(1); xs.Add(2); xs.Add(3); int sum = 0; foreach (int x in xs) { sum += x; xs.Remove(0); } //assert sum == 6; class List<T> : IEnumerable<T> { … IEnumerator<T> GetEnumerator() { int n = Count; for (int i = 0; i < n; i++) yield returnthis[i]; }} Interference ArgumentOutOfRangeException ! Parties execute in an interleaved fashion But we wish to verify them as if they executed in isolation Proposed solution: Prevent either party from seeing the other party’s effects

10. List<int> xs = new List<int>(); xs.Add(1); xs.Add(2); xs.Add(3); int sum = 0; foreach (int x in xs) { sum += x; xs.Remove(0); } //assert sum == 6; class List<T> : IEnumerable<T> { … IEnumerator<T> GetEnumerator() reads this; { int n = Count; for (int i = 0; i < n; i++) yield returnthis[i]; }} Proposed Solution Enforced using an extension of the Boogie methodology Error: unsatisfiedrequiresthis.readCount == 0; reads clause declares the set ofpre-existing objects the iterator method wishes to read And the foreach loop body may not write the objects in the reads clause The iterator method may not read or write any other pre-existing objects

11. The Boogie methodology • Enforces object invariants • Uses a dynamic ownership system • Each object gets two extra fields: • bool inv; • bool writable; • o.f := x; requires o.writable && !o.inv • unpack o; requires o.writable && o.inv • Sets o.inv := false; • Makes owned objects writable • pack o; reverses the effect of unpack o;

12. Adding read-only objects to the Boogie methodology • Each object gets three special fields: • bool inv; • bool writable; • int readCount; // never negative • o.f = x; requires o.writable && o.readCount == 0 && !o.inv • x = o.f; requires o.writable || 0 < o.readCount

13. partial class List<T> : IEnumerable<T> { IEnumerator<T> GetEnumerator() reads this; { int n = Count; for (int i = 0; i < n; i++) yield returnthis[i]; }} partial class List<T> { [Owned] T[] elems; T this[int index] { get requires inv && (writable || 0 < readCount); { read (this) { return elems[index]; } } }} Read-only Methods read (o) S means assert o.writable || 0 < o.readCount; assert o.inv; o.readCount++; foreach ([Owned] field f of o) o.f.readCount++; S foreach ([Owned] field f of o) o.f.readCount--; o.readCount--; How is this call verified?

14. General Approach to “Dependent Objects” • (Preliminary thoughts) • Each object has a set of dependee objects • Object may declare “dependent invariants” that dereference dependee objects • Dependent invariants become requires clauses, unless the dependent object is in Reader mode • Reader mode is statically nested within read blocks on dependee objects • Generic user of Iterator interface will require that Iterator object is in Reader mode

15. What areNested Iterators? yield foreach E; means foreach (T x in E) yield return x; but is implemented with better time complexity if E evaluates to a nested iterator and with less garbage generation if E is a recursive call of the same iterator

16. Nested Enumerations class Tree { int value; List<Tree>! children; IEnumerable<Tree> Nodes { get { yield return this; for (int i = 0; i < children.Count; i++) foreach (Tree t in children[i].Nodes) yield return t; } } } Number of IEnumerable<Tree> and IEnumerator<Tree> objects created is O(n) Number of recursive MoveNext calls is O(n*log(n)) Assume a balanced tree of n nodes

17. Nested Iterators class Tree { int value; List<Tree>! children; IEnumerable<Tree> Nodes { get { yield return this; for (int i = 0; i < children.Count; i++) yield foreach children[i].Nodes; } } }

18. Nested Iterators Space usage: O(log(n)) Nb. of reallocations: O(log(log(n)) struct TreeStackFrame { Tree self; int pc; int i; } class TreeEnumerator : IEnumerator<Tree> { Tree current; TreeStackFrame[]! stack = new TreeStackFrame[8]; int top; public TreeEnumerator(Tree self) { Push(self, 0, 0); } public bool MoveNext() { while (0 <= top) { switch (stack[top].pc) { case 0: current = this; stack[top].pc = 1; return true; case 1: stack[top].i = 0; goto case 2; case 2: if (!(stack[top].i < stack[top].self.children.Count)) goto case 4; stack[top].pc = 3; Push(stack[top].self.children[stack[top].i], 0, 0); break; case 3: stack[top].i++; goto case 2; case 4: Pop(); break; } } return false; } public Tree Current { get { return current; } } void Push(Tree self, int pc, int i) { … } void Pop() { top--; } } Total nb. of loop iterations across all MoveNext calls: O(n) Nb. of TreeStackFrame copy operations: O(log(n))

19. Nested Iterators But if we can statically detect tail recursion, the nb. of allocations becomes O(1)