Rigorous Software Development CSCI-GA 3033-011

1. Rigorous Software DevelopmentCSCI-GA 3033-011 Instructor: Thomas Wies Spring 2012 Lecture 7

2. Programming Project: Minesweeper

3. Project Goal • Implement a minesweeper game together with a perfect minesweeper solver • perfect solver: never guess the next move unless the current board configuration does not allow to deduce whether some unrevealed cell contains a mine or not. • in particular: a perfect solver can decide whether a given board configuration can be solved without guessing.

4. Project Outline • Phase 1: Alloy Model • Phase 2: Java Implementation

5. Project Phase 1: Alloy model • Objectives: • develop abstract minesweeper model • simulate board configurations and plays of the game • specify key invariants of board configurations • identify pre and post-conditions of operations on boards • implement simple minesweeper solver • Deadline: Tuesday, April 10, 2012

6. Project Phase 2: Java Implementation • Objectives: • Transfer Alloy model to JML specifications • Implement • simple parser for board configurations • game • solver • Check JML specifications using JML tools • Deadline: Tuesday, May 1, 2012 • Competition: • submitted solvers will compete against each other

7. Alloy Model • Abstract Minesweeper: game is played on an undirected graph rather than a grid-like board • nodes represent cells of the board • edges indicate which cells are neighbors 2

8. Invariants of Board Configurations • Marked cells are not revealed. • The number of marked cells is not larger than the number of mines. • If x is a revealed cell that has no neighboring mines, then all its neighbors are also revealed. • ...

9. Solving Minesweeper Games • Solving minesweeper games is closely related to the Minesweeper Consistency Problem: • given a board configuration • does there exist an assignment of mines to unrevealed cells that is consistent with the conf.?

10. Minesweeper Consistency: Example Is this configuration consistent? No!

11. Minesweeper Consistency: Example and this one? Yes!

12. Minesweeper Consistency • The Minesweeper Consistency Problem is NP-complete. • There is a simple brute-force algorithm to solve minesweeper games. • If you want a solver that works fast in practice, you will have to do something more clever.

13. Programming Project • Phase 1: Alloy Model • Deadline: Tuesday, April 10, 2012 • Phase 2: Java Implementation • Deadline: Tuesday, May 1, 2012 More detailed information will be available on the course webpage.

14. Today’s Topic:Automated Test Case Generation

15. How to Test Effectively? public class Factorial { /*@ requires n >= 0; @ ensures \result > 0; @*/ public static int factorial (int n) { int result = n; while (--n > 0) result *= n; return result; } public static void main (String[] param) { int n = Integer.parseInt(param[0]); intfact_n = factorial(n); System.out.println("n: " + n + ", n!: " + fact_n); } } Writing a main method for each test case does not scale.

16. How to Test Effectively? Faulty implementation of enqueue on binary heap: public void enqueue(Comparable o) { if(numElems >= elems.length) grow(); intpos = numElems++; intparent = pos/ 2; while(pos > 0 && elems[parent].compareTo(o) > 0) { elems[pos] = elems[parent]; pos= parent; parent = pos/ 2; } elems[pos] = o; } Writing all test cases manually does not scale.

17. Automated Testing • Unit Testing: write code to automatically test your code. • A unit test is a test suite for a unit (class/module) of a program and consists of • setup code to initialize the tested class; • tear down code to clean up after testing; • test cases that call methods of the tested class with appropriate inputs and then check the results. • Once test suites are written, they are easy to run repeatedly (regression testing).

18. Unit Testing in Java: JUnit • A popular framework for unit testing in Java • Frameworks are libraries with gaps • Programmer writes classes following particular conventions to fill in the gaps • Result is the complete product • JUnit automates • the execution and analysis of unit tests; • generation of tests cases from parameterized test oracles and user-provided test data.

19. JUnit Example import static org.junit.Assert.*; importorg.junit.*; ... public classPriorityQueueTest { privatePriorityQueuepq; @Beforepublic void setUp () { pq = new Heap(); } @Afterpublic void tearDown () { pa = null; } @Test public void enqueueTest () { Integer value = new Integer(5); pq.enqueue(value); assertEquals(pq.removeFirst, value); } ... }

20. Drawbacks of JUnit • Low degree of automation • Programmer still needs to write all the test cases • Redundant specification • Duplication between checks in test oracles and formal specification (e.g. provided as JML annotations)

21. JMLUnit: Unit Testing for JML JMLUnit is a unit testing framework for JML built on top of JUnit User: • writes specifications • supplies test data of each type JMLUnitautomatically: • constructs test cases from test data • assembles test cases into test suites • executes test suites • decides success or failure • reports results

22. Test Cases and Suites • A test case (o,x) consists of: • anon-null receiver object o • a sequence xof argument objects • A test suite for method mis a set of test cases with: • receiver of m’s receiver type • arguments of m’s argument types

23. Test Suites are Cross Products • For method enqueue:{ (pq, v) | pq∈ PriorityQueueTestData, v∈ IntegerTestData} • Default is to use all data for all methods • Filtered automatically by preconditions • Users can filter manually if desired • Factory method allows user control of adding test cases to test suite.

24. Errors and Meaningless Test Cases When testing method m: entry precondition violation receiver.m(arg1, ...) internal precondition violation check m’s precondition check f’s precondition { ... x.f(...); } { ... } check f’s postcondition check m’s postcondition other violation Entry precondition violation )test case rejected Internal or other violation) error reported

25. Supplying Test Data • Programmer supplies data in form of strategies • A strategyfor type T: • has method that returns iterator yielding T • Strategies allow reuse of test data • JMLUnit provides framework of built-in strategies • Strategies for built-in types • Allow for easy extension, composition, filtering, etc.

26. Strategies for Test Data • Standard strategies: • Immutable: iterate over array of values; • Cloneable: iterate over array, clone each; • Other: create objects each time. • Cloning and creating from scratch can prevent unwanted interference between tests. • JMLUnittries to guess appropriate strategy.

27. Example Strategies importorg.jmlspecs.jmlunit.strategies.*; importjunit.framework.*; public abstract classHeap_JML_TestDataextendsTestCase { publicIntIteratorvCompIter(String methodName, intargNum) { returnvComparableStrategy.ComparableIterator(); } privateStrategyTypevComparableStrategy = newImmutableObjectAbstractStrategy() { protected Object[] addData() { return newInteger[] {10, -22, 55, 3000}; } }; ...

28. Example Strategies ... publicIndefiniteIteratorvHeapIter (String methodName, intargNum) { returnvPointStrategy.iterator(); } privateStrategyTypevHeapStrategy = newNewObjectAbstractStrategy() { protectedObject make(int n) { switch(n) { case0: return new Heap(); case1: return new Heap(new Integer {1, 2, 3}); default: break; } throw new NoSuchElementException(); } }; }

29. Using JMLUnit • JML-compile the class to be tested jmlc Factorial.java • generate the test suite and test data templates jmlunit Factorial.java • supply the test data $EDITOR Factorial_JML_TestData.java • compile the test suite export CLASSPATH=.:$JMLDIR/bin/jmlruntime.jar:$JMLDIR/specs javacFactorial_JML_Test*.java • execute the test suite jmlracFactorial_JML_Test

30. Drawbacks of JMLUnit • High memory consumption: • JUnit test suite with all test cases is constructed in memory before it is executed. • Limited degree of automation: • only test data for primitive types is generated automatically • Limited degree of granularity: • fine-grained filtering of test data for individual methods is difficult

31. Alternatives to JMLUnit • JMLUnitNG • similar feature set as JMLUnit • lazy generation of test suites and lazy result aggregation (better memory footprint) • improved filtering of test data • not based on JUnit • Korat, TestEra, UDITA • automated generation of test data for complex data types

32. Automated Test Case Generation with Korat • Provides test case generation for complex data types. • Supports checking of JML specifications. • User provides for each complex data type • a Java predicate capturing the representation invariant of the data type; • a finitization of the data type. • Korat generates test cases for all instances that satisfy both the finitization constraints and the representation predicate.

33. Example: Binary Trees importjava.util.*; classBinaryTree { private Node root; privateint size; static class Node { private Node left; private Node right; } ...}

34. Representation Predicate for BinaryTree publicbooleanrepOK() { if (root == null) return size == 0; Set visited = newHashSet(); visited.add(root); LinkedListworkList = newLinkedList(); workList.add(root); while (!workList.isEmpty()) { Node current = (Node) workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; worklist.add(current.left); } if (current.right!= null) { ... } } returnvisited.size () == size; }

35. Finitization for BinaryTree public static FinitizationfinBinaryTree (intNUM_Node) { IFinitization f = newFinitization(BinaryTree.class); IObjSet nodes = f.createObjSet(Node.class, NUM_Node, true); // #Node = NUM_Node f.set(“root”, nodes); // root in null + Node IIntSet sizes = f.createIntSet(Num_Node); f.set(“size”, sizes); // size = NUM_Node f.set(“Node.left”, nodes); // Node.left in null + Node f.set(“Node.right”, nodes); // Node.right in null + Node return f; }

36. Finitization for BinaryTree Instances generated for finBinaryTree(3) left right right left right left right right left left