Programming Language Support for Automated Testing

1. Programming Language Support for Automated Testing Roy Patrick Tan Virginia Tech

2. Software tools help us write bug-free programs • Compiler • Syntax errors • Type checking • Formal Verification • Need to write formal specifications • Proofs usually need human intervention • Unit testing • Need to write unit tests • However, running unit tests can be automatic

3. We can use formal specs to generate unit tests • Design-by-contract style specs (i.e. “executable” specifications) • Generate test-case candidate using sequences of method calls • Use postconditions as test-oracle • Use preconditions to filter invalid tests

4. Sample Spec: Stack conceptStack( Item ) { modelmethodsequence(): concept Sequence( Item ); methodpush( element: Item ) ensures sequence().equals( \old(sequence().insertLast( element.mathObject() )); methodpop( element: Item ) requires sequence().notEmpty() ensures \old(sequence()).equals( sequence().insertLast( element.mathObject() )); methodlength(): Int ensures { \old(sequence()).equals(sequence()); length.equals(sequence().length()); }; }

5. Model object lifetime as walks through a graph • Every walk from init to finalize is an object lifetime -- can use as a test-case • Some walks may be infeasible • Generating all possible walks may be impossible • Reseach problem: what set of walks will be good test-cases?

6. Automated-testing: what PL features do we need? • Design-by-contract formal specs • Preconditions, postconditions, invariants should be executable • Enough features to be realistic, but not too much to be overwhelming • Encourage language features that make it easier to write specs (separate specs and implementation) • Eliminate language features that make it difficult to write specs (aliasing, null values)

7. Sulu • Main influences: RESOLVE, JML • Main differences from RESOLVE: • object-oriented notation • spec language geared toward DBC • Separation between specification (concepts) and implementation (realizations) • Every object has a default initial state (no null values) • Swapping ‘:=:’ as the main data movement operator

8. Digression: What’s wrong with assignment? • Consider this java snippet: int a, b; a = 1; b = a; b = b + 1; • After these operations what is the value of a? • What is the value of b?

9. Object assignment is different from scalar assignment • Now consider this java snippet java.util.Stack a, b; a = new java.util.Stack(); b = new java.util.Stack(); b = a; b.push(“Hello”); • What does b contain after these statements? • What about a?

10. Aliasing is evil • Scalar assignment in Java creates a copy of the value • Object assignment creates an alias to the same object, not a copy. • Aliasing is a big source of bugs! • Simple example, squaring a matrix: matrix_multiply(a,a); • Aliasing breaks modularity of reasoning

11. Is there a different way? • Copying preserves modularity of reasoning but can be inefficient • Aliasing is efficient, but breaks modularity of reasoning • Swapping! • a :=: b • The pre-value of a becomes the post-value of b, and vice versa

12. Swapping as an alternative to assignment? • Swapping is efficient • Internally, compiler can make every swap constant time. • Swapping does not break modularity of reasoning • No aliases

13. class StringStack extends concept Stack(String) realization LinkedList(); x.push("Hello"); x.push("World"); x.push("!"); var c: Console; x.pop(z); c.println(z); x.pop(z); c.println(z); x.pop(z); c.println(z); Output: ! World Hello Sulu Example:Using the Stack component

14. realization LinkedList() implements Stack( Item ) { /* A linked-list node is a pair containing the item to store and a pointer to another node*/ classNode extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious(); vartop: concept ChainPointer(Node) realization Builtin(); varcount: Int; methodpop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode ); //get the value at the top of //the stack and put in element topNode.swapFirst( element ); //set the value of top to the //next-pointer topNode.swapSecond( top ); count := count.minus( 1 ); } Stack implementation

15. realization LinkedList() implements Stack( Item ) { /* A linked-list node is a pair containing the item to store and a pointer to another node*/ classNode extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious(); vartop: concept ChainPointer(Node) realization Builtin(); varcount: Int; methodpop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode ); //get the value at the top of //the stack and put in element topNode.swapFirst( element ); //set the value of top to the //next-pointer topNode.swapSecond( top ); count := count.minus( 1 ); } Stack implementation Pointers are just another component, No special language constructs for pointers

16. realization LinkedList() implements Stack( Item ) { /* A linked-list node is a pair containing the item to store and a pointer to another node*/ classNode extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious(); vartop: concept ChainPointer(Node) realization Builtin(); varcount: Int; methodpop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode ); //get the value at the top of //the stack and put in element topNode.swapFirst( element ); //set the value of top to the //next-pointer topNode.swapSecond( top ); count := count.minus( 1 ); } Stack implementation Swapping is the main data-movement operator

17. realization LinkedList() implements Stack( Item ) { /* A linked-list node is a pair containing the item to store and a pointer to another node*/ classNode extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious(); vartop: concept ChainPointer(Node) realization Builtin(); varcount: Int; methodpop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode ); //get the value at the top of //the stack and put in element topNode.swapFirst( element ); //set the value of top to the //next-pointer topNode.swapSecond( top ); count := count.minus( 1 ); } Stack implementation Assignment is only allowed with method return values.

18. What’s next? • Fully implement specification language • Implement test-case generation algorithms (all-pairs, all-triples, others?) • Run experiments • Graduate