Effective Unit Testing for Concurrent Java Programs: Overcoming Challenges and Applying Best Practices

1. Test-First Java Concurrency for the Classroom SIGCSE 2010 Mathias Ricken and Robert Cartwright Rice University March 12, 2009

2. Test-driven development Concurrent programming Two Trends Brian Goetz, Java Concurrency in Practice, Addison-Wesley, 2006

3. Unit Testing Benefits • Occurs early • Automates testing • Keeps the shared repository clean • Prevents bugs from reoccurring • Allows safe refactoring • Serves as documentation

4. Unit Testing in Assignments • Hand out test cases to students • Improves confidence and understanding • Instill good practices • Require students to extend test suites • Automated grading • Part graded automatically, part by hand

5. Moore’s Law Requires Concurrency Adopted fromSutter 2009

6. Concurrency Is Difficult Unit testing not effective in multi-threaded programs

7. Existing Unit Testing Frameworks • JUnit, TestNG • Don’t detect test failures in threads other than main thread • Failures in event thread not detected either • Don’t ensure that other threads terminate • Tests that should fail may succeed

8. Sample JUnit Tests publicclass SimpleTest extends TestCase { public void testException() { thrownew RuntimeException("booh!"); } public void testAssertion() { assertEquals(0, 1); } } Both tests fail. Both tests fail. } if (0!=1) throw new AssertionFailedError();

9. JUnit Test with Child Thread publicclass SimpleTest extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } Main thread new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); Uncaught exception, test should fail but does not! Child thread end of test spawns Main thread success! uncaught! Child thread

10. ConcJUnit • Backward compatible replacement for JUnit • Detects exceptions in all threads • Exception handler for all child threads and the event thread • Ensures that child threads have terminated and event thread is done • Enumerate live threads after test • Inspect event queue • Requires all child threads to be joined • Analyze join graph

11. Thread Creation Coordinates • In Thread.start() record stack trace of Thread.currentThread() • Easy to find where a thread that caused a failure was started • Also shows where threads that outlived the test were started

12. Creation Coordinates Example class Main { void foo() { // which one? new Helper(true).start(); new Helper(false).start(); // ... } } class Helper extends Thread { void m() { if (b) Assert.fail(); } public void run() { m(); } private boolean b; // … } AssertionError: at Helper.m(Helper.java:2) at Helper.run(Helper.java:3) Started at: at Main.foo(Main.java:4) at Main.bar(Main.java:15) at Main.main(Main.java:25)

13. ConcJUnit Demo

14. Concurrency Examples • In-class discussion • Multi-threaded counter: data races • Multi-threaded bank: deadlock • Homework • Bounded buffer • Readers-writer lock • Test suite handed out to help students • Multi-threaded Breakout

15. Example: Counter • Class that can increment an integer variable N times • Write test first public class CounterTest extends TestCase { final long PER_THREAD = 1000000; public void testSingle() { Counter c = new Counter(); c.incrementNTimes(PER_THREAD); assertEquals(PER_THREAD, c.getCount()); } }

16. Counter: Implementation • Write implementation public class Counter { private long count = 0; public long getCount() { return count; } public void incrementNTimes(long n) { for(long i=0; i<n; ++i) { ++count; } } } Test passes!

17. Counter: Multi-threaded Test • Write multi-threaded test public void testMulti() { final Counter c = new Counter(); for(int i=0; i<NUM_THREADS; ++i) { new Thread() { public void run() { c.incrementNTimes(PER_THREAD); } }.start(); } TestUtils.waitForOtherThreads(); assertEquals(NUM_THREADS*PER_THREAD,c.getCount()); } Test fails (most likely)!

18. Shared Data • Why does the multi-threaded counter test fail? • Thecount field is shared among threads • The ++count operation is not atomic • Thread may be interrupted after reading count, but before writing back to count count=0 regA=? regB=? A1 regA = count; 0 0 ? B1 regB = count; 0 0 0 A2 regA = regA + 1; 0 1 0 A3 count = regA; 1 1 0 B2 regB = regB + 1; 1 1 1 B3 count = regB; 1 1 1

19. Data Races • Definition • Two threads access the same data • At least one access is a write • Nothing prevents the order from changing • Would like code to execute atomically (without interruption) • Java does not support atomicity(for general code)

20. Java Locks & Synchronized • Java provides “lock objects” and synchronized blockssynchronized(lock) { ++count; } • Thread must compete for ownership of lock object before entering synchronized block • Synchronized block is not atomic • But once a thread has a lock object, no other thread can execute code protected by the same lock object

21. Counter: Re-Write • Rewrite implementation // ... private Object lock = new Object(); public void incrementNTimes(long n) { for(long i=0; i<n; ++i) { synchronized(lock) { ++count; } } } Test passes!

22. Concurrency Still Difficult • Even race-free, deadlock-free programs are not deterministic • Thread scheduling is essentially non-deterministic • Different schedules may compute different results • May or may not be acceptable, depending on the task

23. Multi-threaded Breakout • Uses ACM Java Task Force material • Based on “Breakout - Nifty Assignment” by Eric Roberts, SIGCSE 2006 • Multiple balls, each in its own thread • Atomicity assumption when removing bricks • Ends game before all bricks are removed • Other problems • X,Y coordinate changes not atomic • X,Y coordinates not volatile or synchronized, event thread may never see the updates • Correctly synchronized version still not deterministic

24. Future Work • Testing all schedules is intractable • Insert random delays/yields before synchronization operations • Must consider volatile variable accesses to comply with Java Memory Model • Re-run program several times • Can detect a number of sample problems • Record schedule, replay if test fails • Makes failures reproducible if found *3

25. Conclusion • Unit testing has important benefits in industry and in the classroom • Concurrent programming is becoming more important, and it’s difficult • ConcJUnit helps… www.concutest.orgwww.drjava.org

26. Notes

27. Notes • Also cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) ← • Only add edge if joined thread is really dead; do not add if join ended spuriously. ← • Have not studied probabilities or durations for sleeps/yields:One inserted delay may negatively impact a second inserted delayExample: If both notify() and wait() are delayed. ←

28. Spurious Wakeup • ← publicclass Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } } } Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } thrownew RuntimeException("booh!"); Loop since join() may end spuriously

29. Image Attribution

30. Image Attribution • Left image on Two Trends: Test Driven Development, Damian Cugley. • Right image on Two Trends: adapted from Brian Goetz et al. 2006, Addison Wesley. • Graph on Moore’s Law:Adapted from Herb Sutter 2009 • Image on Concurrency Is Difficult:Caption Fridays

31. Extra Slides

32. Changes to JUnit (1 of 3) • Thread group with exception handler • JUnit test runs in a separate thread, not main thread • Child threads are created in same thread group • When test ends, check if handler was invoked Reasoning: • Uncaught exceptions in all threads must cause failure

33. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); invokes checks TestGroup’s Uncaught Exception Handler

34. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); spawns and joins resumes Main thread failure! check exception handler end of test Test thread uncaught! invokes exception handler Child thread

35. Child Thread Outlives Parent publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); check exception handler Main thread success! Test thread Too late! end of test uncaught! invokes exception handler Child thread

36. Changes to JUnit (2 of 3) • Check for living child threads after test ends Reasoning: • Uncaught exceptions in all threads must cause failure • If the test is declared a success before all child threads have ended, failures may go unnoticed • Therefore, all child threads must terminate before test ends

37. Check for Living Child Threads publicclass Test extends TestCase { public void testException() { new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); } } new Thread() { public void run() { thrownew RuntimeException("booh!"); } }.start(); thrownew RuntimeException("booh!"); check for living child threads check group’s handler Main thread failure! Test thread end of test uncaught! invokes group’s handler Child thread

38. Correctly Written Test publicclass Test extends TestCase { public void testException() { Thread t = new Thread() { public void run() { /* child thread */ } }; t.start(); t.join(); } } Thread t = new Thread() { public void run() { /* child thread */ } }; t.start(); t.join(); // wait until child thread has ended /* child thread */ check for living child threads check group’s handler Main thread success! Test thread end of test Child thread *4

39. Changes to JUnit (3 of 3) • Check if any child threads were not joined Reasoning: • All child threads must terminate before test ends • Without join() operation, a test may get “lucky” • Require all child threads to be joined

40. Fork/Join Model • Parent thread joins with each of its child threads • May be too limited for a general-purpose programming language Main thread Child thread 1 Child thread 2

41. Example of Other Join Models • Chain of child threads guaranteed to outlive parent • Main thread joins with last thread of chain Main thread Child thread 1 Child thread 2 Child thread 3

42. Modifying the Java Runtime • Changing Thread.start()and join() • Need to modify Java Runtime Library • Utility to process user’s rt.jar file • Put new jar file on boot classpath:-Xbootclasspath/p:newrt.jar • Still works without modified Thread class • Just does not emit “lucky” warnings

43. Join with All Offspring Threads • Main thread joins with all offspring threads, regardless of what thread spawned them Main thread Child thread 1 Child thread 2

44. Generalize to Join Graph • Threads as nodes; edges to joined thread • Test is well-formed as long as all threads are reachable from main thread Main thread MT Child thread 1 CT1 Child thread 2 CT2 Child thread 3 CT3

45. Join Graph Examples Main thread MT Child thread 1 CT1 Child thread 2 CT2 Main thread MT Child thread 1 CT1 Child thread 2 CT2

46. Unreachable Nodes • An unreachable node has not been joined • Child thread may outlive the test Main thread MT Child thread 1 CT1 Child thread 2 CT2

47. Constructing the Graph // in mainThreadchildThread.start(); • Add node for childThread main Thread MT childThread CT

48. Constructing the Graph // in mainThreadchildThread.join(); • When leaving join(), add edge from mainThread to childThread main Thread MT child Thread CT *2

49. Example: Multi-threaded Bank • Program simulating checking accounts • Account balances are shared data • To avoid data races, use synchronized • Need access to two accounts for transfers synchronized(locks[from]) { synchronized(locks[to]) { accounts[from] -= amount; accounts[to] += amount; } } Test hangs!

50. Deadlock • Thread A transfers from account 0 to 1 • Thread B transfers from account 1 to 0 • Thread A gets interrupted after acquiring locks[0] // thread A // thread B synchronized(locks[0]) { synchronized(locks[1]) { synchronized(locks[0]) // can’t continue, locks[0] // is owned by thread A */ synchronized(locks[1]) // can’t continue, locks[1] // is owned by thread B */