Property-based Testing – ProTest FP7 Strep

1. Property-based Testing – ProTestFP7 Strep John Derrick University of Sheffield, and members of the ProTest team.

2. Consortium • University of Sheffield - USFD - UK • University of Kent - Kent - UK • Universidad Politécnica de Madrid - UPM - Spain • University of Göteborg - ITU1 - Sweden • Chalmers University of Technology - ITU2 - Sweden • Ericsson - Sweden • Erlang Training and Consulting Ltd - ETC - UK • Quviq AB - Sweden • LambdaStream - Lambda - Spain

3. Motivation • Communication networks, based on telephony, wireless and Internet are converging. • Increasing numbers of services are being added to these merging networks, and services are becoming more complex. • Telecoms industry provides 5-nines reliability, ie 99.999% reliability, of core systems. • How to maintain 5-nines reliability in future service-oriented networks and systems.

4. Ensuring reliability • The software for new services and network devices is rapidly growing in complexity. • Key ingredients for ensuring reliability of complex systems: suitable architecture and technologies, improving the software process, and thorough and efficient in testing. • Aim to support the European software industry in its testing methodology for software in network and service infrastructures.

5. Testing • Testing of complex systems is difficult and time-consuming. • Radical approaches are needed, and will be welcomed by industry — if they work. • In ProTest we build upon on the idea of using properties as objects for testing software. Eg random testing driven by LTL description of properties. • In order to deliver dynamic services and interoperable network applications with guaranteed properties, we focus testing around these properties.

6. Property-based testing • Property-based testing aims to deliver more effective tests, more efficiently, and so deliver economic benefits to the European software industry. • Testing with properties as objects improves the competitiveness of software developers, since they can deliver higher quality software for a lower price. • It also allows collaborating companies to improve the definition of their software interfaces and so improve the compatibility between their services. • We will deliver methods and tools to support property-based development of systems.

7. Project components • In order to deliver full benefits we need tools to integrate property-based testing into the development life cycle: • Property discovery. Current testing is based on sets of test cases embedded in test suites; • we are building techniques and tools to aid the software developers to extract properties from this test data. • Current specifications and models are often informal: we are developing specialised property languages to ease the formalisation of existing specifications.

8. Test and property evolution. All software systems are subject to change and evolution; we are building refactoring tools to support the evolution of tests and properties in line with the evolution of the system itself. • Property monitoring. Not all properties can be tested in advance of systems being executed; not all faults will be found during testing. • We will provide tools to support the post hoc examination of trace details for conformance to particular constraints. • Analysing concurrent systems. At the heart of service oriented systems is concurrency. We are designing model-checkers so that concurrent systems can be analysed for fundamental properties.

9. Erlang/OTP • Property-driven development can be used in a variety of programming languages and systems. • Chosen platform for initial implementation is Erlang/OTP, but dissemination into model-driven development arena (UML) and other languages (C/C++, Java, etc). • Erlang/OTPwas developed by Ericsson during the early nineties as a platform for rapid development of network applications. • Erlang is a concurrent functional language with specific support for the development of distributed, fault-tolerant systems with soft real-time requirements.

10. Erlang/OTP - robustness and reliability within the telecoms sector, eg, in the implementation of the AXD301 ATM telecoms switch by Ericsson. • Applications of Erlang include computer telephony, banking, TCP/IP programming (HTTP, SSL, Email, Instant messaging, etc) and 3D-modelling. • There is a wide user base which over the years has grown beyond the telecoms sector, and companies such as Amazon are interested in the suitability of Erlang for programming massively concurrent distributed systems. • Erlang is particularly well placed to exploit multi-core processor chips: the concurrency model of Erlang allows separate processes to run on the different cores with no change to the source code.

11. Consortium • Mix of Universities, SMEs, and a larger industrial company. • The academics bring experience of testing, formal verification, language development, and refactoring. • Quviq is a spin-off, founded to commercialise an innovative property-based testing tool (QuickCheck). • System builders (Ericsson, LambdaStream), consultants, and trainers (ETC), provide insights into what is required of practical tools, what properties will need to be checked, and ways of fitting the results from the project into practical software development methods.

12. Overview - 5 technical WPs

13. WP1 - Property discovery

14. Property discovery • Using QuickCheck - a tool combines random test case generation from properties written in Erlang, with a flexible language for specifying generators. Automatic simplification of test cases. • Develop Domain Specific Languages for dealing with several fragments of UML. Likely to include behavioural aspects such as sequence diagrams. • Developing a method and prototype tool to obtain properties from combining a number of test cases.

15. Developed a technique to reverse-engineer testable models from program execution traces. • Being integrated into an automated iterative process, where hypothesised models are refined by testing them against the implementation.

16. WP2 - System evolution and assurance

17. System Evolution and Assurance • Refactoring means changing the design orstructure of a program … without changing its behaviour. • Develop a method to refactor properties and test cases. Implemented in the Wrangler refactoring tool. • Refactoring of test cases and properties by connecting it to refactoring of the actual subject under test.

18. -module (test). -export([f/1]). add_one (N, [H|T]) -> [H+N | add_one(N,T)]; add_one (N,[]) -> []. f(X) -> add_one(1, X). Generalisation -module (test). -export([f/1]). add_one ([H|T]) -> [H+1 | add_one(T)]; add_one ([]) -> []. f(X) -> add_one(X). -module (test). -export([f/1]). add_int (N, [H|T]) -> [H+N | add_int(N,T)]; add_int (N,[]) -> []. f(X) -> add_int(1, X).

19. Wrangler Embedded in Emacs … and Eclipse. Structural, data type and module refactorings. Tree-based analysis and transformation. Work with multiple modules. Support undo of refactorings Preserve layout and comments. Respects aspects of the macro system.

20. Keeping track of names, including module names. Some refactorings carry over easily … … others require respecting of conventions. The role of makefiles and project management in system evolution … … and the question of integration: Erlide. Issues in refactoring and testing

21. WP3 - Property monitoring

22. Property monitoring • The Erlang run-time system has a built-in trace recording functionality. • How to check properties for a trace. • Develop methods for presenting trace information to the user in a comprehensible way, and also present minimal counter-examples in case a property does not hold for a trace. • By studying the trace we can see which test cases that are efficient and what aspects of the system is not exercised by a test case.

23. WP4 - Analysis of concurrent systems • Model checking is a complementary technique to testing for verifying concurrent systems. • Model checking provides the possibility to fully verify a system. • Develop two model-checkers for Erlang: McErlang, and etomcrl - based on an abstraction of Erlang into the process algebra, mCRL. • We are integrating the QuickCheck based testing framework with the model checker. This involves harmonizing the property specification language of QuickCheck and McErlang, and we plan to investigate the link between model based specification and testing using QuickCheck.

24. WP5 - Property driven development • Undertake a larger case study that will be used to evaluate and assess the results in an industrial setting. • LambdaStream will use the property-based development methodology supported by the QuickCheck tool and refactoring techniques, and evaluate the property monitoring methodology and the model-checking tools and techniques.

25. More information: www.protest-project.eu • Papers at Refactoring Workshop • Papers at ACM SIGPLAN Erlang Workshop • Papers at: • IEEE/ACM conf on Automated Soft. Eng. (ASE’08) • International Workshop on Model Checking and AI • FORTE 2008, etc, etc. • A number of video’s, tutorials and lectures available. See webpages.

26. Questions?