ConcJUnit: Unit Testing for Concurrent Programs

1. ConcJUnit:Unit Testing for Concurrent Programs PPPJ 2009 Mathias Ricken and Robert Cartwright Rice University August 28, 2009

2. Concurrency in Practice Brian Goetz, Java Concurrency in Practice, Addison-Wesley, 2006

3. Concurrency Practiced Badly *1

4. Unit Tests… • Occur early • Automate testing • Keep the shared repository clean • Serve as documentation • Prevent bugs from reoccurring • Allow safe refactoring

5. Existing Unit Testing Frameworks • JUnit, TestNG • Don’t detect test failures in child threads • Don’t ensure that child threads terminate • Tests that should fail may succeed

6. ConcJUnit • Replacement for JUnit • Backward compatible, just replace junit.jar file • Available at www.concutest.org • Detects failures in all threads • Warns if child threads outlive main thread • Warns if child threads are not joined

7. Sample JUnit Tests publicclass Test extends TestCase { public void testException() { thrownew RuntimeException("booh!"); } public void testAssertion() { assertEquals(0, 1); } } Both tests fail. Both tests fail. } if (0!=1) throw new AssertionFailedError();

8. JUnit Test with Child Thread Main thread publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); Child thread end of test spawns Main thread success! uncaught! Child thread

9. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); Uncaught exception, test should fail but does not! • By default, no uncaught exception handler installed for child threads

10. Changes to JUnit (1 of 3) • Thread group with exception handler • JUnit test runs in a separate thread, not main thread • Child threads are created in same thread group • When test ends, check if handler was invoked Reasoning: • Uncaught exceptions in all threads must cause failure

11. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); spawns and joins resumes Main thread failure! check group’s handler end of test Test thread uncaught! invokes group’s handler Child thread

12. Child Thread Outlives Parent publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); check group’s handler Main thread success! Test thread Too late! end of test uncaught! invokes group’s handler Child thread

13. Changes to JUnit (2 of 3) • Check for living child threads after test ends Reasoning: • Uncaught exceptions in all threads must cause failure • If the test is declared a success before all child threads have ended, failures may go unnoticed • Therefore, all child threads must terminate before test ends

14. Check for Living Child Threads publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); check for living child threads check group’s handler Main thread failure! Test thread end of test uncaught! invokes group’s handler Child thread

15. Correctly Written Test publicclass Test extends TestCase { public void testException() { Thread t = new Thread() { public void run() { /* child thread */ } }; t.start(); t.join(); } } Thread t = new Thread() { public void run() { /* child thread */ } }; t.start(); t.join(); // wait until child thread has ended /* child thread */ check for living child threads check group’s handler Main thread success! Test thread end of test Child thread *4

16. Changes to JUnit (3 of 3) • Check if any child threads were not joined Reasoning: • All child threads must terminate before test ends • Without join() operation, a test may get “lucky” • Require all child threads to be joined

17. Fork/Join Model • Parent thread joins with each of its child threads • May be too limited for a general-purpose programming language Main thread Child thread 1 Child thread 2

18. Example of Other Join Models • Chain of child threads guaranteed to outlive parent • Main thread joins with last thread of chain Main thread Child thread 1 Child thread 2 Child thread 3

19. Generalize to Join Graph • Threads as nodes; edges to joined thread • Test is well-formed as long as all threads are reachable from main thread Main thread MT Child thread 1 CT1 Child thread 2 CT2 Child thread 3 CT3

20. Unreachable Nodes • An unreachable node has not been joined • Child thread may outlive the test Main thread MT Child thread 1 CT1 Child thread 2 CT2

21. Constructing the Graph: start() // in mainThreadchildThread.start(); • Add node for childThread main Thread MT childThread CT

22. Constructing the Graph : join() // in mainThreadchildThread.join(); • When leaving join(), add edge from mainThread to childThread main Thread MT child Thread CT *3

23. Modifying the Java Runtime • Changing Thread.start()and join() • Need to modify Java Runtime Library • Utility to process user’s rt.jar file • Put new jar file on boot classpath:-Xbootclasspath/p:newrt.jar • Still works without modified Thread class • Just does not emit “lucky” warnings

24. Evaluation • JFreeChart • All tests passed; tests are not concurrent • DrJava: 900 unit tests • Passed: 880 • No join: 1 • Lucky: 18 • Timeout: 1 • Runtime overhead: ~1 percent

25. Limitations • Only checks chosen schedule • A different schedule may still fail • Example: Thread t = new Thread(…);if (nondeterministic()) t.join(); *2

26. Future Work • Insert sleeps or yields in critical code locations • Example: If a notify() is delayed, a wait() may time out. • Can detect a number of sample problems • Run tests several times on build server • Record schedule, replay if test fails • Makes failures reproducible if found *5

27. ConcJUnit Demo

28. Thank you! www.concutest.org

29. Notes

30. Notes • Probably not caused by concurrency problems; just a metaphor. ← • Also cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) ← • Only add edge if joined thread is really dead; do not add if join ended spuriously. ←

31. Spurious Wakeup publicclass Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } } } Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } thrownew RuntimeException("booh!"); Loop since join() may end spuriously ←

32. Notes • Have not studied probabilities or durations for sleeps/yields:One inserted delay may negatively impact a second inserted delayExample: If both notify() and wait() are delayed. ←

33. Extra Slides

34. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); invokes checks TestGroup’s Uncaught Exception Handler

35. Join with All Offspring Threads • Main thread joins with all offspring threads, regardless of what thread spawned them Main thread Child thread 1 Child thread 2

36. Join Graph Examples Main thread MT Child thread 1 CT1 Child thread 2 CT2 Main thread MT Child thread 1 CT1 Child thread 2 CT2

37. Thread Creation Context • In Thread.start()– also record stack trace of Thread.currentThread( • Easy to find source code of a child thread that is not joined • Also available for uncaught exception stack traces

38. Creation Context Example class Helper extends Thread { void m() { Assert.fail(); } public void run() { m(); } } class Main { void foo() { // which one? new Helper().start(); new Helper().start(); // ... } } AssertionError: at Helper.m(Helper.java:2) at Helper.run(Helper.java:3) Started at: at Main.foo(Main.java:4) at Main.bar(Main.java:15) at Main.main(Main.java:25)

39. Many Thanks To… • My advisor • Corky Cartwright • My committee members • Walid Taha • David Scott • Bill Scherer • NSF and Texas ATP • For providing partial funding

40. Concurrency Problems NotFound During Testing *1