Efficient Techniques for Software Testing

1. Efficient Techniques for Software Testing Jeff Lei CSE@UTA April 13, 2007

2. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation, Pervasive Computing

3. Software Engineering • Software has become pervasive in modern society • Directly contributes to quality of life • Malfunctions cost billions of dollars every year, and have severe consequences in a safe-critical environment • All about building better software in better ways, especially for large-scale development • Requirements, design, coding, testing, maintenance, configuration, documentation, deployment, and etc.

4. Software Testing • A dynamic approach to detecting software faults • Alternatively, static analysis can be performed, which is however often intractable • Involves sampling the input space, running the test object, and observing the runtime behavior • Perhaps the most widely used approach in practice • Labor intensive, and often consumes more than 50% of development cost

5. Research Problems • How to detect as many faults as possible spend with as little effort as possible? • How to generate a good set of test inputs? • How to perform actual test executions? • How to evaluate test outcomes?

6. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation

7. Concurrent Programs • Consists of multiple threads of execution that proceed in parallel • multithreaded programs, multi-process (or distributed) programs • Advantages: • Better resource utilization, increased computation efficiency, and providing simplified solutions for many problem domains • Notoriously difficult to test due to their non-deterministic behavior

8. What Is Unique • Non-determinism • Multiple executions with the same data input might display different behaviors • Synchronization • How to ensure that concurrent events are exercised in an expected order? • Communication • How to exchange information between different threads?

9. Concurrency Testing Strategies • Non-deterministic testing - execute the same program with the same input for multiple times: • easy, but inefficient, and some errors cannot be detected • Deterministic testing - select a set of test sequences and then force them to be exercised: • can detect more subtle errors, but requires additional effort for test sequence selection and runtime control • State exploration – explore the state space in a systematic manner • suffers the well-known state explosion problem • state representation is difficult for programs written in a full-fledged programming language

10. Reachability Testing • Combines non-deterministicand deterministic testing • Generating test sequences dynamically, without constructing any static model • Dealing with partial orders directly, and thus no redundant interleavings! • Can also be considered as a state exploration technique • Exercising every possible synchronization sequence of a program with a given input

11. The Framework • Execute a program P with a given input non-deterministically to collect a trace Q • Identify the race conditions in Q and compute its race variants • For each variant, conduct a prefix-based test run to collect a new trace Q’ • Repeat 2, 3, and 4 for each newly collected trace Q’.

12. Example

13. Main Results • Developed three reachability testing algorithms • Exhaustive RT, Combinatorial RT, and Random RT • Built a reachability testing tool called RichTest • Portable implementation, applicable to most common synchronization constructs • Conducted an empirical evaluation • Performed significantly better than VeriSoft (from Bell Labs) • Publications: STVR07a, TSE06, ISSRE05

14. Dinning Philosophers

15. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation

16. Synchronization Components • Synchronization logic is often encapsulated into one or more components • Synchronization components need to be tested separately • Obtain necessary confidence prior to integration • helps to reduce complexity by incremental testing • A different problem than testing complete programs • The ways in which a component could be used is typically unknown

17. Java Monitor • A high-level synchronization construct that supports data encapsulation and information hiding • A Java class that defines one or more synchronized methods • Mutual exclusion automatically enforced by the Java runtime • Conditional synchronization implemented by the programmer, using wait and notify operations

18. Java Monitor Semantics • Consists of three components: an entry queue, a critical section, a waiting queue • A thread must enter the CS before it executes a synchronized method; it has to wait in the entry queue if the CS is not empty • The wait and notify operations can only be executed inside the CS • wait: puts the current thread in the waiting queue • notify: awakens one thread in the waiting queue

19. Example classBoundedBuffer { privateint fullSlots=0; privateint capacity = 0; privateint[] buffer = null; privateint in = 0, out = 0; publicBoundedBuffer(int bufferCapacity) { 1. capacity = bufferCapacity; 2. buffer = newint[capacity]; } publicsynchronizedvoid deposit (int value) { 3. while (fullSlots == capacity) { 4. try { wait(); } catch (InterruptedException ex) {} } 5. buffer[in] = value; 6. in = (in + 1) % capacity; 7. if (fullSlots++ == 0) { 8. notifyAll(); } } publicsynchronizedint withdraw () { 9. int value = 0; 10. while (fullSlots == 0) { 11. try { wait(); } catch (InterruptedException ex) {} } 12. value = buffer[out]; 13. out = (out + 1) % capacity; 14. if (fullSlots-- == capacity) { 15. notifyAll(); } 16. return value; } }

20. Graphic View W D Entry Queue Condition Queue Critical Section Assume that the buffer is initially empty.

21. Research Questions • What is unique to testing Java monitors? • Used to synchronize, and thus typically accessed by, multiple threads simultaneously • Can existing work on OO testing apply? • OO testing methods typically assume a single-threaded test driver • How to effectively test Java monitors?

22. Technical Challenges • To simulate possible scenarios in which a Java monitor can be accessed, multiple threads need to be created • Some faults can only be detected by a certain minimum number of threads. But how many? • Multiple threads can display non-deterministic behavior. How to describe and control thread behavior?

23. A State Exploration-Based Approach • Explores the state space of a monitor as an open system • Threads introduced on-the-fly, and as needed, to simulate race conditions • State abstraction used to ensure termination and to control state explosion • Thread behavior is controlled at finer granularity than method calls

24. Main Results • Developed an algorithm that implements the state exploration-based approach • Built a prototype tool, called MonitorExplorer, and conducted an empirical evaluation • Empirical results suggest that this approach can be very effective • Ongoing work: Extend to general monitors, and general synchronization components • Publication: ISSRE06

25. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation, Pervasive Computing

26. Combinatorial Testing • Creates tests by combining different parameter values • The input space of a test object is represented as a set of parameters and their values • Advantages: light specification, test input generation can be fully automated, requires no access to source code • How to deal with the combinatorial explosion problem?

27. T-Way Interaction Testing • Motivation: Not every parameter contributes to every fault • Many faults can be exposed by interactions involving a few parameters • Given any t parameters, every combination of values of these parameters be covered by at least one test • Allows faults triggered by at most t parameters to be detected • Makes an excellent trade-off between test effort and test coverage

28. T-Way Testing - Example P1 P2 P3 0 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1 P1 P2 P3 0 0 0 0 1 1 1 0 1 1 1 0 Three parameters, each with values 0 and 1 2-way testing exhaustive testing

29. State of the Art • Greedy construction: Each step tries to cover as many combinations as possible • Involves explicit enumeration of all possible combinations • Algebraic Construction: test sets are constructed using pre-defined rules • Most approaches focus on 2-way (or pairwise) testing

30. Strategy In-Parameter-Order • Input: A set of parameters and values; Output: a t-way test set • Construct a t-way test set for the first t parameters • Extend the test set to cover each of the remaining parameters one by one • Horizontal growth - extends each existing test by adding one value for the new parameter • Vertical growth – adds new tests, if needed, to make the test set complete

31. Example P1 P2 P3 P4 0 0 0 0 0 0 1 1 0 1 0 2 0 1 1 0 1 0 0 1 1 0 1 2 1 1 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 0 1 2 1 1 0 2 * 0 0 2 * 1 1 2 P1 P2 P3 0 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1 P1 P2 P3 P4 0 0 0 0 0 0 1 1 0 1 0 2 0 1 1 0 1 0 0 1 1 0 1 2 1 1 0 0 1 1 1 1 (a) (b) Horizontal growth Vertical growth (c) • Four parameters: P1, P2, P3, and P4 • P1, P2, and P3 have 2 values • P4 has 3 values 3-way test set

32. Main Results • Developed two algorithms, IPOG and IPOG-D, for general t-way testing • Has the lowest algorithmic complexity among the existing algorithms • Built a t-way testing tool called FireEye, and conducted an empirical evaluation • Performed significantly better than existing tools, e.g., ITCH, TConfig, Jenny, TestVector • Publications: TSE02, ECBS07, STVR07b

33. TCAS Example TCAS: Seven 2-value parameters, two 3-value parameters, one 4-value parameter, two 10-value parameters

34. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation, Pervasive Computing

35. Other Work • Model checking: How to deal with the state explosion problem? • True concurrency model vs interleaving-based concurrency model • Publication: FM05 • Protocol validation: A more specific problem than model checking • Allows multiple threads to proceed simulatenously during state exploration • Publication: CJ06 • Pervasive computing • Context-awareness: How to derive high level events from a sequence of low level events? • Power-constraints: How to reduce the power consumption on small mobile devices?

36. References • [STVR07a] Y. Lei, R. Carver, R. Kacker, and D. Kung. A Combinatorial Testing Strategy for Concurrent Programs. Accepted for publication in Software Testing, Verification, and Reliability, 2007. • [STVR07b] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. Two deterministic strategies for multi-way software testing. Software Testing, Verification, and Reliability, pending review. • [ECBS07] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. A general strategy for multi-way software testing. Int’l Conf. on Engineering of Computer-Based Systems, 2007. • [ISSRE06] Yu Lei, R. H. Carver, D. Kung, V. Gupta, M. Hernandez. A State Exploration-Based Approach to Testing Java Monitors. Int’l Symposium Software Reliability Engineering, 256-265, 2006. • [CJ06] Q. Ye, Y. Lei, and D. Kung. A Blocking-Based Approach to Protocol Validation, The Computer Journal, 49:541-553, 2006. • [TSE06] Y. Lei and R. Carver, Reachability testing of concurrent programs, IEEE Transactions On Software Engineering, 32(6):382-403, 2006. • [FM05] Y. Lei and P. Iyer. An Approach to Unfolding Asynchronous Communication Protocols, Proc. 13th Intl. Symp. on Formal Methods, pp. 334-349, 2005. • [TSE02] K. C. Tai and Y. Lei, "A test generation strategy for pairwise testing", IEEE Transactions on Software Engineering, Vol. 28, No. 1, pp. 109-111., Jan. 2002.