Reachability Testing of Concurrent Programs

1. Reachability Testing of Concurrent Programs Richard Carver, GMU Yu Lei, UTA Reachability Testing of Concurrent Programs 1

2. Outline • Introduction • The Reachability Testing (RT) Process • Computing Race Variants • An Example Scenario • Conclusion and Future work Reachability Testing of Concurrent Programs 2

3. A concurrent program ... • Contains a set of cooperative threads/processes • multithreaded programs - threads synchronize by accessing shared memory • distributed programs – threads synchronize by exchanging messages • Is pervasive in modern software development • better resource utilization • increases computing efficiency • provides simple solutions to certain problems • Is notoriously difficult to test due to their non-deterministic behavior Reachability Testing of Concurrent Programs 3

4. Testing Strategies • Non-deterministic testing - execute the same program with the same input for multiple times: • easy, but inefficient, and some errors cannot be detected • Deterministic testing - select a set of test sequences and then force them to be exercised: • can detect more subtle errors, but requires additional effort for test sequence selection and runtime control • Prefix-based testing – a combination of non-deterministic and deterministic testing: • a test run executes deterministically at the beginning and then proceeds non-deterministically Reachability Testing of Concurrent Programs 4

5. Reachability testing • ... is based on prefix-based testing, and has the following main features: • Dynamic generation of test sequences - avoids constructing any static model • Dealing with partial orders directly – avoids generating redundant interleavings • Complete coverage on synchronization behavior – has important applications on program verification • Can be implemented in a platform-independent manner Reachability Testing of Concurrent Programs 5

6. Program Execution Model • In our paper, a general representation of concurrent executions is provided, which allows RT to be applied to different types of programs, including semaphore-based, monitor-based, and message-passing programs. Reachability Testing of Concurrent Programs 6

7. Outline • Introduction • The RT Process • Computing Race Variants • An Example Scenario • Conclusion and Future work Reachability Testing of Concurrent Programs 7

8. SYN-sequence • Records the relative ordering of events that occur on a synchronization object • e.g., P/V events on a semaphore, entry/reentry events on a monitor, lock/unlock events on a lock, send/receive events on a channel • Used to characterize the synchronization behavior of an execution • the result of an execution is determined by the programtext, the input data, and the SYN-sequence exercised by the execution Reachability Testing of Concurrent Programs 8

9. Concurrent Path • Port M; • T1 T2 T3 • (1) M.send(10) (2) M.send(4) (3) x = M.receive (); • (4) y = M.receive (); • (5) z = x - y Assume that an execution of the above program exercises the sequence of statements: (1, 2, 3, 4, 5). Can we determine the value of z? Reachability Testing of Concurrent Programs 9

10. T2 T3 T1 s1 r1 s2 r2 Space-Time Diagram T2 T3 T1 s1 s2 r1 r2 z = -6 z = 6 Reachability Testing of Concurrent Programs 10

11. The RT Framework • Execute a program P with input I non-deterministically to collect a SYN-sequence Q • Compute the race variants of Q, each of which is a prefix of one or more other SYN-sequences • For each variant, conduct a prefix-based test run to collect a new SYN-sequence Q’ • Repeat 2, 3, and 4 for each newly collected SYN-sequence Q’. Reachability Testing of Concurrent Programs 11

12. Tracing and Replay • Additional runtime control is needed to conduct a prefix-based test run: • Synchronization library with semaphores, monitors, and message passing • Library has built-in tracing and replay capabilities • No modifications to the thread scheduler Reachability Testing of Concurrent Programs 12

13. Outline • Introduction • The RT Process • Computing Race Variants • An Example Scenario • Conclusion and Future work Reachability Testing of Concurrent Programs 13

14. Race Condition • Refers to the phenomenon where a receiving event can be synchronized with two or more sending events (in different executions) • e.g., the messages sent by two send events can be received by the same receive event • The race set of a receive event r consists of all the send events that r could possibly be synchronized with (in different executions). Reachability Testing of Concurrent Programs 14

15. {s2} T2 T3 T4 T5 T1 {s4, s5} {} s1 s2 {s5} r1 s4 s3 r3 r2 s5 {} r4 r5 Example Note that s1 is not in the race set of r2, because r2 may not even exist if r1 receives a message other than s1. Reachability Testing of Concurrent Programs 15

16. Identifying Races • Assign each event a vector timestamp to indicate the happens-before relation • Whether there is a race between two events can be determined by comparing their timestamps • Two timestamp assignment schemes described in paper: • Thread-centric: keeps a logic clock for each thread • Object-centric: keeps a version number for each synchronization object Reachability Testing of Concurrent Programs 16

17. Race Variant • Represents the beginning portion of one or more SYN-sequences that could have happened but didn’t • Derived from a SYN-sequence by changing one or more race outcomes in the sequence • If we change a race outcome, then all the events that could potentially be affected by the new outcome need to be removed Reachability Testing of Concurrent Programs 17

18. rj i v j Race Table • Consists of one column for each receiving event with a non-empty race set • Each row can be used to derive a unique, partially-ordered race variant of Q. • Value v indicates how receiving event r in Q is changed to create variant V: • v = -1: r is removed from V; • v = 0: the send partner of r is left unchanged • in V; and • v > 0: in V, the send partner of r is changed • to the v-th event in race_set(r, Q), Reachability Testing of Concurrent Programs 18

19. {s2} T2 T3 T4 T5 T1 {s4, s5} {} s1 s2 {s5} r1 s4 s3 r3 r2 s5 {} r4 r5 v1 v5 T2 T3 T4 T5 T2 T3 T4 T5 T1 T1 s1 s2 s1 s2 r1 s4 s4 s3 s3 r3 r2 r1 r3 s5 r4 Example Reachability Testing of Concurrent Programs 19

20. Outline • Introduction • The RT Process • Computing Race Variants • An Example Scenario • Conclusion and Future work Reachability Testing of Concurrent Programs 20

21. T1T2T3T4T5T6 s1: send (T2, a); r1: x = recv (); r2: y = recv (); s3: send (T3, c); r4: u = recv (); s5: send (T5, e) s2: send (T3, b); r3: z = recv (); s4: send(T5, d); r5: v = recv (); if (u == e) { s6: send (T2, f); } (a) T1 T2 T3 T4 T5 T6 T1 T2 T3 T4 T5 T6 T1 T1 T2 T2 T3 T3 T4 T4 T5 T5 T6 T6 s1 s1 s1 s1 s3 s3 s3 s3 r1 r1 r1 r1 s2 s5 s2 s5 s2 s2 s5 s5 r2 r2 r2 r2 s4 s4 s4 s4 r4 r4 r4 r4 r3 r3 r5 r5 r3 r3 r5 r5 s6 s6 Q0 Q1 Q2 Q3 T1 T2 T3 T4 T5 T6 T1 T2 T3 T4 T5 T6 s1 s1 s5 s5 s3 s3 r4 r4 s4 s4 r5 r5 s6 s6 r1 r1 r2 s2’ s2 r2 r3 r3 r3’ r3’ Q4 Q5 (b) An Example Scenario Reachability Testing of Concurrent Programs 21

22. Outline • Introduction • The RT Process • Computing Race Variants • An Example Scenario • Conclusion and Future Work Reachability Testing of Concurrent Programs 22

23. Conclusion • RT generates test sequences dynamically, which is more scalable than many model-based approaches. Our latest tool has almost constant memory requirements. • Compared to JPF and VeriSoft, which try to reduce the chances of generating redundant interleavings, RT deals with partial orders directly. • While most tools for concurrent programs testing rely on access to the thread scheduler and are thus platform dependent, our RT tool is platform independent. Reachability Testing of Concurrent Programs 23

24. Future Work • Test oracle – automatically evaluate test runs against properties specified in some logic formalism • Coverage-based RT - use coverage measures to selectively exercise a subset of the SYN-sequences • Model extraction – use RT to extract a complete model of the synchronization/communication behavior of a program • Real-time testing – add explicit time into the RT framework Reachability Testing of Concurrent Programs 24