Deadlock Hunter Final Presentation

1. Deadlock HunterFinal Presentation Software Engineering Lab. Winter 07/08 Supervised by: Yonatan Kaspi Introduced by: Jamil Shehadeh Husam Khshaiboun

2. Agenda • Introduction (10 min) • Gaston & Alphonso (5 min) • The Deadlock Hunter (20 min) • Script & Statistics (5 min) • One More Example (3 min) • Conclusions (2 min)

3. Agenda • Introduction (10 min) • Gaston & Alphonso (5 min) • The Deadlock Hunter (20 min) • Script & Statistics (5 min) • One More Example (3 min) • Conclusions (2 min)

4. IntroductionDeadlock - Refreshment • Deadlock refers to a specific condition when two or more processes are each waiting for another to release a resource, or more than two processes are waiting for resources in a circular chain • This situation may be likened to two people who are drawing diagrams, with only one pencil and one ruler between them. • Deadlocks are more frequent in multithreaded programs because different threads use the same parent process resources. • They are hard to find due to the low probability for the deadlock execution path to occur, since the threads are executed in the same “secure” path most of the time.

5. IntroductionDeadlock Hunter Project • Our goal is to build a tool that will force the operating system scheduler to make the execution path be truly random. • We will do this by examining the byte code of a Java program and adding small code sections in places that effect the execution path. • Our job is to locate the deadlock, not to fix it.

6. IntroductionStrategy • Dealing with threads in any language can be done using specific commands. • Using java enables us to be Platform independent. • The specific commands will be searched for in java byte code using a parser. • Code sections will be added automatically in these places in order to force randomization in the execution path so that the probability for a deadlock to occur increases. • A Java Code with a hidden deadlock will be used as a client. • The deadlock hunter will build a new instrumented byte code according to the code sections that the parser find interesting.

7. IntroductionApache BCEL • BCEL stands for Byte Code Engineering Library • Intended to give users a convenient possibility to analyze, create, and manipulate (binary) Java class files. • Classes are represented by objects which contain all the symbolic information of the given class: methods, fields and byte code instructions, in particular. • Using it we can: • Get an abstraction of the byte code • Manipulate instruction lists • Manipulate methods • Look for keywords in the byte code • Define new classes

8. IntroductionMain Obstacles • BCEL is a problematic tool • Complicated user interface. • Unintuitive usage. • Lets you deal with the corners. • No debugging ability after code instrumentation. • Eclipse debug on a class file is not possible (overwrites the instrumented class file). • Debug with prints only. • Very few helping sources. • Impossible to work continuously • Apache manual and FAQ don’t always help • Even Google search is not always effective

9. Agenda • Introduction (10 min) • Gaston & Alphonse (5 min) • The Deadlock Hunter (20 min) • Script & Statistics (5 min) • One More Example (3 min) • Conclusions (2 min)

10. Gaston & AlphonseA classical deadlock • Gaston & Alphonso are two gentle and friendly fellows. • They must bow one to the other anytime they meet. • Each one of them bows and continue bowing until the other bows back. • What if both of them bow at the same time?

11. Gaston & AlphonseThe Original Code public void bow(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name +" has bowed to me!"); bowed.bowBack(this); }} public void bowBack(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name+" has bowed back to me!"); }} public static void main (String[] args) { final Client alphonse = new Client("Alphonse"); final Client gaston = new Client("Gaston"); new Thread(new Runnable() { public void run() { alphonse.bow(gaston); } }).start(); new Thread(new Runnable() { public void run() { gaston.bow(alphonse); } }).start(); }

12. Gaston & Alphonse Hide The Deadlock public void bow(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name +" has bowed to me!"); bowed.bowBack(this); }} public void bowBack(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name+" has bowed back to me!"); }} public static void main (String[] args) { final Client alphonse = new Client("Alphonse"); final Client gaston = new Client("Gaston"); new Thread(new Runnable() { public void run() { alphonse.bow(gaston); } }).start(); wait(10); new Thread(new Runnable() { public void run() { gaston.bow(alphonse); } }).start(); }

13. Gaston & AlphonseHow to find the deadlock • The idea is to enter other wait functions randomly anywhere before a synchronize keyword appears. • Then, we can shake the stability of the regular execution path. • We always enter the same byte code of an invariant function that includes a call of other modifiable function. • To do that. “Some” BCEL is needed.

14. Agenda • Gaston & Alphonso (5 min) • The Deadlock Hunter (20 min) • Script & Statistics (5 min) • One More Example (3 min) • Conclusions (2 min)

15. The DeadLock Hunter Entered method Client byte code • “Enter” method: • public void enter(){ • entered(1000);} • “Entered” method: • synchronized void entered(int time) { • try{ • if(time != 10){ • Random rand = new Random(); • double rnd = rand.nextDouble(); • if(rnd > 0.3) • wait(time);} • else • wait(10); • } • catch(Exception e) {e.printStackTrace();} • } “Enter” Function Byte Code

16. The Deadlock Hunter Main code flow Get the Client class file Loop over its methods Create new Instruction list for each method Insert the “enter” function call where needed Deal with branch instructions Update exception table handler Dump new class file

17. The Deadlock Hunter Original byte code of bow method 0: aload_0[42](1)    1: dup[89](1)   2: astore_2[77](1)   3: monitorenter[194](1)   4: getstatic[178](3) 23   7: new[187](3) 29  10: dup[89](1)  11: aload_1[43](1)  12: invokevirtual[182](3) 31  15: invokestatic[184](3) 33  18: invokespecial[183](3) 39  21: ldc[18](2) 41  23: invokevirtual[182](3) 43  26: aload_0[42](1) 27: getfield[180](3) 13   30: invokevirtual[182](3) 43 33: ldc[18](2) 47  35: invokevirtual[182](3) 43  38: invokevirtual[182](3) 49  41: invokevirtual[182](3) 52  44: aload_1[43](1)  45: aload_0[42](1)  46: invokevirtual[182](3) 57  49: aload_2[44](1)  50: monitorexit[195](1)  51: goto[167](3) -> return  54: aload_2[44](1)  55: monitorexit[195](1)  56: athrow[191](1)  57: return[177](1)

18. The DeadLock HunterGetting Started • Get the class object using its path: • JavaClass clazz = Repository.lookupClass(args[0]); • Extract the methods: • Method[] methods = clazz.getMethods(); • Get “bow” method: • Method bow=methods[2];

19. The DeadLock Hunter Get and Parse instruction lists • Create a constant pool generator: • ConstantPoolGen cp = new ConstantPoolGen(clazz.getConstantPool()); • Get instruction list of a method: • InstructionList ilbow=new MethodGen(bow, clazz.getClassName(), cp).getInstructionList(); • Get InstructionHandle: • InstructionHandle[] bowhandle=ilbow.getInstructionHandles(); • Parse and find “monitorenter”: • if(bowhandle[i].getInstruction().toString().startsWith("monitorenter")) • Find modification line “aload0”: • aloadIdx=find_string(i+insertions*ilwork_func.getLength(), il, "aload_0");

20. The DeadLock HunterInstruction list instrumentation • Add a single instruction to a instruction list: • il.append(bowhandle[i].getInstruction()); • Add an instruction list to other instruction list: • il.insert(il.getInstructionHandles()[aloadIdx], ilwork_func.copy()); • Handling a branch instruction: • if(ins[i].getInstruction() instanceof BranchInstruction){ • BranchInstruction branch=(BranchInstruction)ins[i].getInstruction(); • newil.append(branch); }

21. The DeadLock Hunter Remove Return in the Entered code After removing return: public InstructionList removeReturn(InstructionList Il){ InstructionList newIl= new InstructionList(); InstructionHandle[] ins=Il.getInstructionHandles(); for(int i=0; i<Il.getLength()-1; i++) if(ins[i].getInstruction() instanceof BranchInstruction){ BranchInstruction branch=(BranchInstruction)ins[i].getInstruction(); newIl.append(branch);} else if(ins[i].getInstruction() instanceof INVOKEVIRTUAL){ INVOKEVIRTUAL invoke=(INVOKEVIRTUAL)ins[i].getInstruction(); newIl.append(invoke);} else newIl.append(ins[i].getInstruction()); return newIl;} Before removing return: 0: aload_0[42](1) 1: sipush[17](3) 2000 4: invokevirtual[182](3) 15 7: return[177](1)

22. The DeadLock Hunter Exception table update • The exception table marks handlers, i.e., code chunks, to be responsible for exceptions of certain types that are raised within a given area of the byte code. When there is no appropriate handler the exception is propagated back to the caller of the method. • This table must be updated because adding instructions make an offset to the handler pointers and range line definitions. • Get exception table: • CodeException[] exp=methods[2].getCode().getExceptionTable(); • Modify exception table according to the number of entered code lines for(int j=0; j<exp.length; j++) { int start=exp[j].getStartPC() int end=exp[j].getEndPC(); if (end>bc_length){ exp[j].setEndPC(exp[j].getEndPC()+ilprint_length); exp[j].setHandlerPC(exp[j].getHandlerPC()+ilprint_length); if (start>bc_length) exp[j].setStartPC(exp[j].getStartPC()+ilprint_length); } }

23. The DeadLock Hunter Instrumented byte code of bow method 0: aload_0[42](1)   1: sipush[17](3) 1000   4: invokevirtual[182](3) 86   7: aload_0[42](1)   8: dup[89](1)   9: astore_2[77](1)  10: monitorenter[194](1)  11: getstatic[178](3) 23  14: new[187](3) 29  17: dup[89](1)  18: aload_1[43](1)  19: invokevirtual[182](3) 31  22: invokestatic[184](3) 33  25: invokespecial[183](3) 39  28: ldc[18](2) 41  30: invokevirtual[182](3) 43  33: aload_0[42](1) 34: getfield[180](3) 13  37: invokevirtual[182](3) 43  40: ldc[18](2) 47  42: invokevirtual[182](3) 43  45: invokevirtual[182](3) 49  48: invokevirtual[182](3) 52  51: aload_1[43](1)  52: aload_0[42](1)  53: invokevirtual[182](3) 57  56: aload_2[44](1)  57: monitorexit[195](1)  58: goto[167](3) -> return  61: aload_2[44](1)  62: monitorexit[195](1)  63: athrow[191](1)  64: return[177](1)

24. Agenda • The Deadlock Hunter (20 min) • Script & Statistics (5 min) • One More Example (3 min) • Conclusions (2 min)

25. Script and Statistics Script Code Initialize #!/bin/tcsh set count=0 set sum1=0 set till=100 set curr=0 while($count<$till) @ count++ echo $count java -classpath ~/.eclipse/test -noverify pack.Client > a.out & sleep 1 set curr = `wc -l a.out |cut -c 1` if($curr<4)then @ sum1++ endif End mv -f Client.class pack/ set count=0 set sum2=0 set curr=0 while($count<$till) @ count++ echo $count java -classpath ~/.eclipse/test -noverify pack.Client > a.out & sleep 3 set curr = `wc -l a.out |cut -c 1` \rm a.out if($curr<4)then @ sum2++ endif end echo "$sum1 deadlocks occured before code instrumentation" echo "$sum2 deadlocks occured after code instrumentation" Loop N times Run original class Update sum if deadlock happens Get the instrumented class file Loop N times Run instrumented class Update sum if deadlock happens Print output

26. Script and Statistics Script Output • Before code instrumentation, 0 deadlocks occurred in 100 runs. • After adding all waits randomly (probability 70%) in bow function only, 65 deadlocks occurred in 100 runs. • After adding all waits randomly (probability 20%) in bow function only, 35 deadlocks occurred in 100 runs. • Adding waits randomly in all functions resulted in 100 deadlocks in 100 runs !

27. Agenda • Script & Statistics (5 min) • One More Example (3 min) • Conclusions (2 min)

28. One More Example The Client private Object lock1 = new Object (); private Object lock2 = new Object (); public void instanceMethod1 () { synchronized (lock1) { synchronized (lock2) { System.out.println("inside instanceMethod1"); }}} public void instanceMethod2 (){ synchronized (lock2){ synchronized (lock1){ System.out.println("inside instanceMethod2"); }}} public static void main (String[] args){ final Client2 first = new Client2(); final Client2 second = new Client2(); new Thread(new Runnable() { public void run() { first.instanceMethod1();} }).start(); wait(10); new Thread(new Runnable() { public void run() { second.instanceMethod2(); } }).start(); }

29. One More Example Statistics • Before manipulating, 5 deadlocks occurred in 100 runs. • After code instrumentation with probability 70% in all methods, 65 deadlocks occurred in 100 runs.

30. Agenda • One More Example (3 min) • Conclusions (2 min)

31. Conclusions • The Client Execution path was randomized. • A hidden deadlock has a higher probability to happen after code instrumentation. • Project goal achieved: “Our goal is to build a tool that will force the operating system scheduler to make the execution path be truly random” • What next? • Be unrelated with the java code completely: • Entered function should be an outer function • Count deadlocks without relying on the clients prints • Generalize to other interesting situations. (not only Sync. Statements)

32. Thanks…