Theory and Practice of Co-verification Process: UniTesK Story

1. Theory and Practice ofCo-verification Process: UniTesK Story RedVerst group of ISP RAS Alexander K. Petrenko Victor V. Kuliamin http://www.ispras.ru

2. Overview Introduction : Why co-verification? Main part : What is UniTesK?Solving Engineering Problems Case studies

3. What these numbers mean? • The Economic Impacts of Inadequate Infrastructure for Software Testing, • NIST Report, May 2002

4. Waterflow Process Model Requirements Design Implementation Testing Deployment

5. Iterative Process Model Inception Elaboration Construction Transition Requirements Design Implementation Testing Deployment

6. Agile Development Methods Expansion Percent of organizations using modern development methods According to “The Decision is in: Agile versus Heavy Methodologies” by Robert Charette, Cutter IT Journal, vol. 2, No. 19, 2002 http://www.cutter.com/freestuff/apmupdate.html

7. Agile Development Methods Expansion Percent of organizations recognizing more than 50% of projects as agile According to “The Decision is in: Agile versus Heavy Methodologies” by Robert Charette, Cutter IT Journal, vol. 2, No. 19, 2002 http://www.cutter.com/freestuff/apmupdate.html

8. Inadequate Quality of Software The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST Report, May 2002

9. Evolution of Testing Localization of errors Demonstration of errors Testing is the process of executing a program or system with the intent of finding errors [Myers, 1979] The purpose of testing is to show that aprogramhas bugs [Hetzel, 1988] Evaluation of quality Testing is the process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component [IEEE 90]

10. Co-verification Verification confirms that activity products properly reflect the requirements specified for them at the beginning of the activity. Co-verification process Perform verification of activity before proceeding to the dependent activities Prepare all the artifacts needed for verification concurrently with main activity products

11. Co-verification Development Process Business Modeling Deployment Requirements Integration Architecture Design Implementation Component Design

12. Main Part Overview Traditional approaches to verification UniTesK approach Example

13. Traditional Software Development Process Requirements Analysis Design Implementation-debugging Requirements Elicitation Test Case Design Test Implementation Test Execution Test Result Analysis Design and development Verification

14. Co-verification Process Requirements Analysis Design Implementation-debugging Requirements Elicitation Test Case Design Test Implementation Test Execution Test Result Analysis Design and development Verification

15. UniTesK Approach UniTesK Tools .N@T J@T CTesK VDM++TesK Uniform Specification Extension Uniform Test Architecture Integration with Development Environments Foundations Model Based Testing

16. Requirements Formalization Requirements Requirements Formal Specifications

17. Co-verification Support Formal Specifications Requirements Ambiguity? Incompleteness? Inconsistency?

18. Engineering Problems (1) Specification technique should support Easy transformation of requirements into specifications Easy automation of further test development Functional test coverage definition High reusability of specifications Specification notation should not require special education and skills

19. Specification Techniques ExecutableImperative state based specifications ConstraintsState based data type constraints, pre- and postconditions, internal invariants AxiomaticAction based axioms

20. Executable Specifications Are very close to some implementation Are easy to use in the industry Can be transformed into prototypes Are not close to requirements( √¯ = e½ln = lim(xn+1 = ½(xn +x/xn)) ) Unsuitable for test coverage measurement Can cause problems with conformance checkingHow to compare the results? Are highly reusable as executable code But are low reusable as criteria of correctness

21. Constraint Specifications Have the structure similar with implementation Are easy to use in the industry But have different form Are close to requirements in most cases Are easy to construct from requirements Suitable for test coverage measurement Counterexample: memory management subsystem Can be directly used in conformance checking Special constructs enabling reuse can be added

22. Axiomatic Specifications Are far from common implementations and have greatly different structure Can hardly be introduced in the industry Are usually far from requirements Are hard to develop in real-life projects Can hardly be used for coverage measurement But sometimes are the only appropriate form Can be used for conformance checking But provide poor error localization Reusability is a problem

23. Comparison of Specification Techniques

24. Specification Notation Specification language Suitable for capture abstract properties Has formal semantics Requires complex mediator layer for implementation Requires special education, mastering is enduring Extension of programming language Abstract concepts can be added by libraries Ambiguous parts of language can be excluded Complex mediators are not required Mastering can be made more effective Facilitates co-verification process

25. Constraint Specifications Preconditions and postconditions of operations Data type constraints Functional coverage description based on specification structure UniTesK Specification Technique UniTesK .N@T J@T CTesK VDM++TesK Uniform Specification Extension Uniform Test Architecture Integration with Development Environments Model Based Testing

26. UniTesK Specification Technique Constraint Specifications Preconditions and postconditions of operations Data type constraints specification Operation() preblock, returning Boolean value post block, returning Boolean value use @to refer to pre-value of expressions invariant Invariant()block, returning Boolean value

27. void enq (Object obj, int priority)priority  [Min_Priority, Max_Priority] Object deq () int size () Priority Queue Example 0 5 2 enq () enq () deq () enq () 5 5 4 3 1 1 size ()

28. J@va : Specification Extension of Java specification package pqueue; public class PQueueSpecification { specification publicvoid enq(Object obj, int prty) readsobj, prty updates items.?, priorities.? { pre { return obj !=null; } post { int i = 0; for(i = 0; i < items.size() && ((Integer)priorities.elementAt(i)).intValue() > prty; i++ ); ... UniTesK .N@T J@T CTesK VDM++TesK Uniform Specification Extension Uniform Test Architecture Integration with Development Environments Model Based Testing

29. Model State Definition specification package; public class PQueueSpecification { // List of elements of the queue public List items =new List(); // Accompanying list of their priorities public IntList priorities =new IntList(); }

30. Data Integrity Constraints : Lists are not Null public class PQueueSpecification { public List items =new List(); public IntList priorities =new IntList(); invariant ListsAreNotNull() { return items != null && priorities != null; } }

31. Data Integrity Constraints : Lists’ Sizes are Equal public class PQueueSpecification { invariant ListsSizesAreEqual() { return items.size()== priorities.size(); } }

32. Data Integrity Constraints : Priorities Lie in the Range public class PQueueSpecification { public static int Min_Priority; public static int Max_Priority; invariant PrioritiesLieInTheRange() { for(int i = 0; i < priorities.size(); i++) { if( priorities.get(i) < Min_Priority || priorities.get(i) > Max_Priority ) return false; } return true; } }

33. Data Integrity Constraints : Priorities do not Increase public class PQueueSpecification { invariant PrioritiesDoNotIncrease() { for(int i = 1; i < priorities.size(); i++) { if( priorities.get(i-1) < priorities.get(i)) return false; } return true; } }

34. Operation Specification : size() Method public class PQueueSpecification { specification public int size () reads this.? { pre { return true; } post { return size == items.size(); } } }

35. Operation Specification : enq() Method public class PQueueSpecification { specification public void enq (Object obj, int prty) readsobj, prty updates this.? { pre { return obj != null && Min_Priority <= prty && prty <= Max_Priority; } post { int i = 0; for(i = 0; i < items.size() && priorities.get(i) >= prty; i++ ); PQueueSpecification new_state =(PQueueSpecification)@clone(); new_state.items.add(i, obj); new_state.priorities.add(i, prty); returnthis.equals(new_state); } } }

36. Auxiliary Methods public class PQueueSpecification { public boolean equals (PQueueSpecification other) { return items.equals(other.items) && priorities.equals(other.priorities); } public Object clone () { PQueueSpecification result = new PQueueSpecification(); result.items = (List)items.clone(); result.priorities = (IntList)priorities.clone(); return result; } }

37. Operation Specification : deq() Method public class PQueueSpecification { specification public Objectdeq () updates this.? { post { if(items.isEmpty()) returndeq == null && items.isEmpty(); else { PQueueSpecification new_state =(PQueueSpecification)@clone(); Object result = new_state.items.first(); new_state.items.removeFirst(); new_state.priorities.removeFirst(); returnthis.equals(new_state) && deq == result; } } } }

38. Coverage Goals Definition Formal Specifications pre --------------------------- post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- --------------------------------------------- -------------------------------------------------- ---- ------------------------------------- Requirements Testable Specifications pre --------------------------- post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- -------------------------------------------------- ---- ------------------------------------- Test Case 1 --------------- --------------- -------- 1 2 3

39. Co-verification Support Formal Specifications pre --------------------------- post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- --------------------------------------------- -------------------------------------------------- ---- ------------------------------------- UniTesK supports this transformation with techniques and tools Software --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- Testable Specifications pre --------------------------- post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- -------------------------------------------------- ---- -------------------------------------

40. Engineering Problems (2) More than one coverage metric is needed Automatic extraction of coverage goals Enabling test designer to introduce additional goals

41. Several Levels of Coverage Metrics 1.2.1 1.2.2 1.1 1.2 1.3 Formal Specifications pre --------------------------- post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- -------------------------------------------------- ---- ------------------------------------- 1.3.1 1.3.2 1.3.3 1 2 2.1 2.2 3 3.1.1 3.1.2 3.1.3 3.1 3.2

42. Definition of Coverage Goals : Functionality Branches • Functional coverage description based on specification structure post if(a || b) ... branch“Case 1”; ... else if(!c && d) ... branch“Case 2”; ... else ... branch“Case 3”; ...

43. public class PQueueSpecification { specification public int size () reads this.? { post { branch“Single branch”; return size == items.size(); } } } Functionality Branches in size() Method

44. specification public void enq (Object obj, int prty) readsobj, prty updates this.? { pre { return obj != null; } post { int i = 0; for(i = 0; i < items.size() && priorities.get(i) >= prty; i++ ); branch“Single branch”; PQueueSpecification new_state =(PQueueSpecification)@clone(); new_state.items.add(i, obj); new_state.priorities.add(i, prty); returnthis.equals(new_state); } } Functionality Branches in enq() Method

45. specification public Objectdeq () updates this.? { post { if(items.isEmpty()) { branch “Empty queue”; returndeq == null && items.isEmpty(); } else { branch “Nonempty queue”; PQueueSpecification new_state =(PQueueSpecification)@clone(); Object result = new_state.items.firstElement(); new_state.items.removeFirstElement(); new_state.priorities.removeFirstElement(); returnthis.equals(new_state) && deq == result; } } } Functionality Branches in deq() Method

46. Additional Coverage Goals : enq() 1 7 5 4 2 0 5 5 4 3 1 1 1 1 1 1

47. Additional Coverage Goals : deq() 5 5 4 3 1 1 5 4 3 1

48. Definition of Coverage Goals : Marked Paths post if(a || b || c) { branch“Case 1”; ... } if(a) mark “a holds”;

49. Marked Paths in enq() Method post { int i = 0; for(i = 0; i < items.size() && priorities.get(i) >= prty; i++ ); if( items.isEmpty() ) mark "Insertion in the empty queue"; else if( prty > priorities.maximum() ) mark "Insertion in the head"; else if( prty == priorities.maximum()&& prty == priorities.minimum() ) mark "Insertion with single existing priority"; else if( prty == priorities.maximum() ) mark "Insertion with maximum priority"; else if( prty < priorities.minimum() ) mark "Insertion in the tail with new priority"; else if( prty == priorities.minimum() ) mark "Insertion with minimum priority"; else if( !priorities.contains(prty) ) mark "Insertion in the middle with new priority"; else mark "Insertion in the middle with existing priority"; branch“Single branch”; ... }

50. post { if(items.isEmpty()) { branch “Empty queue”; ... } else { if( items.size() == 1 ) mark "Single element in the queue"; else if( !(priorities.maximum() == priorities.get(1)) ) mark "Single element with maximum priority, several elements in the queue"; else if( priorities.toSet().size() > 1 ) mark "Several elements with maximum priority, there are other priorities"; else mark "Several elements in the queue with the same priority"; branch “Nonempty queue”; ... } } Marked Paths in deq() Method