Testing Concurrent/Distributed Systems

1. CEN 5076 Class 14 – 12/05 Testing Concurrent/Distributed Systems Testing Concurrent/Distributed Systems Review of Final

2. Regression Testing Regression test approaches: • Retest All – rerun all baseline test • Retest Risky Use cases – choose baseline test to rerun by risk heuristics. • Retest by Profile – choose baseline tests to rerun by allocating time in proportion to the optional profile. • Retest Changed Segments – choose baseline tests to rerun by comparing code segments. CEN 5076 Class 14 - 12/05

3. Regression Testing Regression test approaches: • Retest Within Firewall – choose baseline tests to rerun by analyzing dependencies. Points to note: • Time and cost constraints may preclude running such a test, in which case a reduced regression test can be selected, supporting selective regression testing. • Reduction of a test suites should use a safe reduction approach. CEN 5076 Class 14 - 12/05

4. Regression Testing • A 100% safe regression suite consists of “all test that could possibly exhibit output when run on [the delta build]” [Rothermel ’94]. • Criteria for a safe regression test suite: • Inclusiveness – percentage of baseline tests that may show regression faults and that are selected for a reduced test suite. • Precision – percentage of baseline test cases in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test set. CEN 5076 Class 14 - 12/05

5. Regression Testing • Efficiency – the cost to identify a reduced regression test suite. • Generality – the range of application for the selection strategy. • A safe regression test suite is 100 percent inclusive. • A precise regression test suite does not include any tests that cannot cause different output as these tests can be safely removed. CEN 5076 Class 14 - 12/05

6. Testing Concurrent/Distributed Systems • Few systems today are designed to execute using a single process. • Computational models: • Concurrent • Parallel • Networked • Distributed CEN 5076 Class 14 - 12/05

7. Testing Concurrent Programs • Concurrent program P consist of several processes, • P runs on a single processor machine whose process scheduler suspends the running process randomly after any of it atomic instructions, and • Scheduler randomly chooses a process to run. Note that this model is equivalent to one in which the instructions are allowed to execute simultaneously. [Weiss S. N. 88] CEN 5076 Class 14 - 12/05

8. Testing Concurrent Programs • Sequential approach - one program run per test case. Problem for concurrent programs! Why? • The execution of a concurrent (centralized, parallel, or distributed) program P exercises a sequence of events called a synchronization sequence or SYN-seq. [Tai K.C. 89] Give an example of a SYN-seq. • The result of an execution of P with input X is determined by P, X and the SYN-seq of the execution. CEN 5076 Class 14 - 12/05

9. Testing Concurrent Programs cont • Note due to the unpredictable nature of concurrent processes and the use of nondeterministic statements, multiple executions of P with the same input may exercise different SYN-seqs and may produce different results. [Tai K. C. 89] • How would you go about testing concurrent programs? CEN 5076 Class 14 - 12/05

10. Testing Concurrent Programs cont • Two approaches: nondeterministic and deterministic testing. • Assume the specification for a concurrent program is A, which models valid program behavior. Let P be a concurrent program. Let (P, X) represent program P with input X. Defn: A SYN-seq S is said to be feasible (valid) for (P, X) if S is allowed by the implementation (specification) of (P, X). CEN 5076 Class 14 - 12/05

11. Testing Concurrent Programs cont Defn: P is said to have a synchronization fault if a feasible (valid) SYN-seq of (P, X) is invalid (infeasible). Nondeterministic testing of P against S: • Select a set of test cases • For each selected input X execute P one or more times and collect the output and SYN-seq of each execution • Determine the validity of each collected SYN-seq Any problems with nondeterministic testing? CEN 5076 Class 14 - 12/05

12. Testing Concurrent Programs cont Problems with deterministic testing: • Some feasible SYN-seqs may be executed many times. • Some feasible SYN-seqs of (P, X) may never be executed. • The existence of valid but infeasible SYN-seqs may never be detected. Any advantages? CEN 5076 Class 14 - 12/05

13. Testing Concurrent Programs cont Deterministic testing of P against S: • Select a set of test cases, each of the form (X, S) • For each (X, S) perform a “forced execution” of P according to X and S • Check results of the forced execution (S is feasible for (P, X) iff the forced execution exercises exactly S before P terminates) Any problems with deterministic testing? Any advantages of deterministic testing? CEN 5076 Class 14 - 12/05

14. Testing Concurrent Programs cont Problems: • Selection of (X, S) pairs • Need to modify some component of the implementation e.g., compiler, runtime system, or operating system, to force an execution Advantages: • Allows SYN-seqs to satisfy specific test criterion. • Can detect the existence of feasible, invalid SYN-seqs of P as well as valid infeasible SYN-seqs of (P, X). • Regression testing is possible after changing P. CEN 5076 Class 14 - 12/05

15. Testing Concurrent Programs cont • Nondeterministic testing and deterministic testing are extremes, there are several hybrid between: • Reachability testing [Hwang G. H. et al. 95] • Prefix-based testing [Tai K. C. et al. 89] • Temporal testing [Yang et al. 98] • Early work in testing concurrent programs focused on debugging. Why? CEN 5076 Class 14 - 12/05

16. Specification-Based Technique [Carver et al. 98] • Uses constraints to guide the selection of test sequences. • Constraints written using CSPE (Constraints on Succeeding and Preceding Events). • CSPE describes feasibility constraints – sequencing constraints that are satisfied by the feasible SYN-seqs of P. • Validity constraints are an extension of feasibility constraints – constraints expected to be satisfied by the feasibility SYN-seqs of P. CEN 5076 Class 14 - 12/05

17. Specification-Based Technique [Carver et al. 98] Example: n-slot bounded buffer (BB) deposit deposit “a” – always “~” – never “p” – possibly “#” – start of execution “$” – normal termination “→” – immediately after S0 S1 … S2 withdraw withdraw B1. a[#; → deposit]: Buffer BB can always accept deposit first. S0 has an outgoing transition for deposit. B2. ~[#;→ withdraw]: Buffer BB cannot accept withdraw first S0 has no outgoing transition for withdraw. B3. p[deposit; → deposit] (buffer is not full) CSPE can be formally defined as logical formulas. CEN 5076 Class 14 - 12/05

18. Specification-Based Technique [Carver et al. 98] Constraint-based testing methodology: • Select a sequencing constraint notation, define how to achieve coverage and detect violations of constraints in the notation, determine constraint coverage criterion. • According to P’s spec use the notation to derive a set of validity constraints. • Perform nondeterministic testing of P and collect the SYN-seqs . Analyze the SYN-seqs to measure coverage and detect violations of P’s validity constraints. CEN 5076 Class 14 - 12/05

19. Specification-Based Technique [Carver et al. 98] Constraint-based testing methodology: • Generate additional SYN-seqs of P with the intention of using these SYN-seqs to cover the validity constraints not yet covered. • Perform deterministic testing of P with the generated SYN-sequences. (if deterministic testing of P with a generated SYN-seq fails to cover a constraint then a constraint violation is detected.) • After corrections have been made to P perform deterministic (regression) testing with the SYN-seqs that previously uncovered violations. CEN 5076 Class 14 - 12/05

20. Program-Based Technique [Yang et al. 98] • Supports test data adequacy criteria. • Automatically generates all-definition-use-paths for parallel programs (really!). • Program model consists of: threads that communicate through shared variables; syn events – post and wait; and thread creation pthread_create. • Handles problem of nondeterminism by performing controlled execution through temporal testing i.e., CEN 5076 Class 14 - 12/05

21. Program-Based Technique [Yang et al. 98] • Temporal testing alters the scheduled execution time of a program segment in order to detect syn errors. • A temporal test case is a 3-tuple (P, X, D), P – program, X – input, D – timing changes. D is the schedule execution time for certain syn instructions n represented as t(n) will change for each temporal test the behavior of P observed. Example of n - delay points along the du-path being tested i.e., all process creation and syn-events in du-path. CEN 5076 Class 14 - 12/05

22. Program-Based Technique [Yang et al. 98] Steps of testing process: • Generate all-du-paths statically. • Execute the program multiple times without considering any timing changes. • Examine the trace results. Different paths executed indicates possibility of syn error. • Generate temporal test cases w.r.t the du-paths. • Perform temporal testing automatically. • Examine the results CEN 5076 Class 14 - 12/05

23. Program-Based Technique [Yang et al. 98] Test coverage classification: • acceptable – du-path in P free of infeasible paths • unacceptable – violation of acceptable criteria • w-runnable – du-path that does not cause an infinite wait in any thread (liveness or safety?) • non-w-runnable – violation of w-runnable CEN 5076 Class 14 - 12/05

24. Program-Based Technique [Yang et al. 98] Finding du-paths • Create Parallel Program Flow Graph (PPFG). • Annotate PPFG • DFS employed to cover required nodes (SYN-seqs nodes and thread creation node) • Use dominator trees (DT) and implied trees (IT) approach to cover optional nodes • After a path is found all nodes are annotated with a traversal control number (TRN) • Path generation actual path coverage is generated using the traversal control annotations CEN 5076 Class 14 - 12/05