1 / 36

Efficient Techniques for Software Testing

Efficient Techniques for Software Testing. Jeff Lei CSE@UTA April 13, 2007. Outline. Software Testing Concurrency Testing Testing Concurrent Programs Testing Synchronization Components T-Way Interaction Testing Other Work Model Checking, Protocol Validation, Pervasive Computing.

khalil
Download Presentation

Efficient Techniques for Software Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Efficient Techniques for Software Testing Jeff Lei CSE@UTA April 13, 2007

  2. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation, Pervasive Computing

  3. Software Engineering • Software has become pervasive in modern society • Directly contributes to quality of life • Malfunctions cost billions of dollars every year, and have severe consequences in a safe-critical environment • All about building better software in better ways, especially for large-scale development • Requirements, design, coding, testing, maintenance, configuration, documentation, deployment, and etc.

  4. Software Testing • A dynamic approach to detecting software faults • Alternatively, static analysis can be performed, which is however often intractable • Involves sampling the input space, running the test object, and observing the runtime behavior • Perhaps the most widely used approach in practice • Labor intensive, and often consumes more than 50% of development cost

  5. Research Problems • How to detect as many faults as possible spend with as little effort as possible? • How to generate a good set of test inputs? • How to perform actual test executions? • How to evaluate test outcomes?

  6. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation

  7. Concurrent Programs • Consists of multiple threads of execution that proceed in parallel • multithreaded programs, multi-process (or distributed) programs • Advantages: • Better resource utilization, increased computation efficiency, and providing simplified solutions for many problem domains • Notoriously difficult to test due to their non-deterministic behavior

  8. What Is Unique • Non-determinism • Multiple executions with the same data input might display different behaviors • Synchronization • How to ensure that concurrent events are exercised in an expected order? • Communication • How to exchange information between different threads?

  9. Concurrency Testing Strategies • Non-deterministic testing - execute the same program with the same input for multiple times: • easy, but inefficient, and some errors cannot be detected • Deterministic testing - select a set of test sequences and then force them to be exercised: • can detect more subtle errors, but requires additional effort for test sequence selection and runtime control • State exploration – explore the state space in a systematic manner • suffers the well-known state explosion problem • state representation is difficult for programs written in a full-fledged programming language

  10. Reachability Testing • Combines non-deterministicand deterministic testing • Generating test sequences dynamically, without constructing any static model • Dealing with partial orders directly, and thus no redundant interleavings! • Can also be considered as a state exploration technique • Exercising every possible synchronization sequence of a program with a given input

  11. The Framework • Execute a program P with a given input non-deterministically to collect a trace Q • Identify the race conditions in Q and compute its race variants • For each variant, conduct a prefix-based test run to collect a new trace Q’ • Repeat 2, 3, and 4 for each newly collected trace Q’.

  12. Example

  13. Main Results • Developed three reachability testing algorithms • Exhaustive RT, Combinatorial RT, and Random RT • Built a reachability testing tool called RichTest • Portable implementation, applicable to most common synchronization constructs • Conducted an empirical evaluation • Performed significantly better than VeriSoft (from Bell Labs) • Publications: STVR07a, TSE06, ISSRE05

  14. Dinning Philosophers

  15. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation

  16. Synchronization Components • Synchronization logic is often encapsulated into one or more components • Synchronization components need to be tested separately • Obtain necessary confidence prior to integration • helps to reduce complexity by incremental testing • A different problem than testing complete programs • The ways in which a component could be used is typically unknown

  17. Java Monitor • A high-level synchronization construct that supports data encapsulation and information hiding • A Java class that defines one or more synchronized methods • Mutual exclusion automatically enforced by the Java runtime • Conditional synchronization implemented by the programmer, using wait and notify operations

  18. Java Monitor Semantics • Consists of three components: an entry queue, a critical section, a waiting queue • A thread must enter the CS before it executes a synchronized method; it has to wait in the entry queue if the CS is not empty • The wait and notify operations can only be executed inside the CS • wait: puts the current thread in the waiting queue • notify: awakens one thread in the waiting queue

  19. Example classBoundedBuffer { privateint fullSlots=0; privateint capacity = 0; privateint[] buffer = null; privateint in = 0, out = 0; publicBoundedBuffer(int bufferCapacity) { 1. capacity = bufferCapacity; 2. buffer = newint[capacity]; } publicsynchronizedvoid deposit (int value) { 3. while (fullSlots == capacity) { 4. try { wait(); } catch (InterruptedException ex) {} } 5. buffer[in] = value; 6. in = (in + 1) % capacity; 7. if (fullSlots++ == 0) { 8. notifyAll(); } } publicsynchronizedint withdraw () { 9. int value = 0; 10. while (fullSlots == 0) { 11. try { wait(); } catch (InterruptedException ex) {} } 12. value = buffer[out]; 13. out = (out + 1) % capacity; 14. if (fullSlots-- == capacity) { 15. notifyAll(); } 16. return value; } }

  20. Graphic View W D Entry Queue Condition Queue Critical Section Assume that the buffer is initially empty.

  21. Research Questions • What is unique to testing Java monitors? • Used to synchronize, and thus typically accessed by, multiple threads simultaneously • Can existing work on OO testing apply? • OO testing methods typically assume a single-threaded test driver • How to effectively test Java monitors?

  22. Technical Challenges • To simulate possible scenarios in which a Java monitor can be accessed, multiple threads need to be created • Some faults can only be detected by a certain minimum number of threads. But how many? • Multiple threads can display non-deterministic behavior. How to describe and control thread behavior?

  23. A State Exploration-Based Approach • Explores the state space of a monitor as an open system • Threads introduced on-the-fly, and as needed, to simulate race conditions • State abstraction used to ensure termination and to control state explosion • Thread behavior is controlled at finer granularity than method calls

  24. Main Results • Developed an algorithm that implements the state exploration-based approach • Built a prototype tool, called MonitorExplorer, and conducted an empirical evaluation • Empirical results suggest that this approach can be very effective • Ongoing work: Extend to general monitors, and general synchronization components • Publication: ISSRE06

  25. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation, Pervasive Computing

  26. Combinatorial Testing • Creates tests by combining different parameter values • The input space of a test object is represented as a set of parameters and their values • Advantages: light specification, test input generation can be fully automated, requires no access to source code • How to deal with the combinatorial explosion problem?

  27. T-Way Interaction Testing • Motivation: Not every parameter contributes to every fault • Many faults can be exposed by interactions involving a few parameters • Given any t parameters, every combination of values of these parameters be covered by at least one test • Allows faults triggered by at most t parameters to be detected • Makes an excellent trade-off between test effort and test coverage

  28. T-Way Testing - Example P1 P2 P3 0 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1 P1 P2 P3 0 0 0 0 1 1 1 0 1 1 1 0 Three parameters, each with values 0 and 1 2-way testing exhaustive testing

  29. State of the Art • Greedy construction: Each step tries to cover as many combinations as possible • Involves explicit enumeration of all possible combinations • Algebraic Construction: test sets are constructed using pre-defined rules • Most approaches focus on 2-way (or pairwise) testing

  30. Strategy In-Parameter-Order • Input: A set of parameters and values; Output: a t-way test set • Construct a t-way test set for the first t parameters • Extend the test set to cover each of the remaining parameters one by one • Horizontal growth - extends each existing test by adding one value for the new parameter • Vertical growth – adds new tests, if needed, to make the test set complete

  31. Example P1 P2 P3 P4 0 0 0 0 0 0 1 1 0 1 0 2 0 1 1 0 1 0 0 1 1 0 1 2 1 1 0 0 1 1 1 1 1 0 1 0 0 1 0 1 0 0 1 2 1 1 0 2 * 0 0 2 * 1 1 2 P1 P2 P3 0 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1 P1 P2 P3 P4 0 0 0 0 0 0 1 1 0 1 0 2 0 1 1 0 1 0 0 1 1 0 1 2 1 1 0 0 1 1 1 1 (a) (b) Horizontal growth Vertical growth (c) • Four parameters: P1, P2, P3, and P4 • P1, P2, and P3 have 2 values • P4 has 3 values 3-way test set

  32. Main Results • Developed two algorithms, IPOG and IPOG-D, for general t-way testing • Has the lowest algorithmic complexity among the existing algorithms • Built a t-way testing tool called FireEye, and conducted an empirical evaluation • Performed significantly better than existing tools, e.g., ITCH, TConfig, Jenny, TestVector • Publications: TSE02, ECBS07, STVR07b

  33. TCAS Example TCAS: Seven 2-value parameters, two 3-value parameters, one 4-value parameter, two 10-value parameters

  34. Outline • Software Testing • Concurrency Testing • Testing Concurrent Programs • Testing Synchronization Components • T-Way Interaction Testing • Other Work • Model Checking, Protocol Validation, Pervasive Computing

  35. Other Work • Model checking: How to deal with the state explosion problem? • True concurrency model vs interleaving-based concurrency model • Publication: FM05 • Protocol validation: A more specific problem than model checking • Allows multiple threads to proceed simulatenously during state exploration • Publication: CJ06 • Pervasive computing • Context-awareness: How to derive high level events from a sequence of low level events? • Power-constraints: How to reduce the power consumption on small mobile devices?

  36. References • [STVR07a] Y. Lei, R. Carver, R. Kacker, and D. Kung. A Combinatorial Testing Strategy for Concurrent Programs. Accepted for publication in Software Testing, Verification, and Reliability, 2007. • [STVR07b] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. Two deterministic strategies for multi-way software testing. Software Testing, Verification, and Reliability, pending review. • [ECBS07] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. A general strategy for multi-way software testing. Int’l Conf. on Engineering of Computer-Based Systems, 2007. • [ISSRE06] Yu Lei, R. H. Carver, D. Kung, V. Gupta, M. Hernandez. A State Exploration-Based Approach to Testing Java Monitors. Int’l Symposium Software Reliability Engineering, 256-265, 2006. • [CJ06] Q. Ye, Y. Lei, and D. Kung. A Blocking-Based Approach to Protocol Validation, The Computer Journal, 49:541-553, 2006. • [TSE06] Y. Lei and R. Carver, Reachability testing of concurrent programs, IEEE Transactions On Software Engineering, 32(6):382-403, 2006. • [FM05] Y. Lei and P. Iyer. An Approach to Unfolding Asynchronous Communication Protocols, Proc. 13th Intl. Symp. on Formal Methods, pp. 334-349, 2005. • [TSE02] K. C. Tai and Y. Lei, "A test generation strategy for pairwise testing", IEEE Transactions on Software Engineering, Vol. 28, No. 1, pp. 109-111., Jan. 2002.

More Related