jFuzz – Java based Whitebox Fuzzing

1. David Harvison Adam Kiezun jFuzz – Java based Whitebox Fuzzing

2. Summary Problem • Generating interesting test inputs for file reading programs takes time. Approach • Create a smart fuzzer to generate inputs that cause programs to crash. Results • jFuzz generates a many input files and creates a base for others to expand upon.

3. Problem • Bugs in a program may cause crashes for specific input files. • A compiler with buggy code, a media player with a corrupt file, etc. • Creating input files by hand takes time. • Some of the files may exercise the same code. • Want a way to automatically generate input files that crash the program.

4. Idea • Generate inputs that cause crashes by generating many inputs that exercise unique execution paths.

5. File reading code. Program Example if (car0 == '-') { neg = true; } else { neg = false; cnt++; } if (car1 >= '0' && car1 <= '5') { val = car1 - '0'; } else { val = car1; }

6. if (car0 == '-') { neg = true; } else { neg = false; cnt++; } if (car1 >= '0' && car1 <= '5') { val = car1 - '0'; } else { val = car1; } Program Example • File reading code. • Want to generate files which exercise different program paths.

7. Program Example if (car0 == '-') { neg = true; } else { neg = false; cnt++; } if (car1 >= '0' && car1 <= '5') { val = car1 - '0'; } else { val = car1; } • File reading code. • Want to generate files which exercise different program paths. car0 == '-' car1 == '5'

8. Program Example if (car0 == '-') { neg = true; } else { neg = false; cnt++; } if (car1 >= '0' && car1 <= '5') { val = car1 - '0'; } else { val = car1; } • File reading code. • Want to generate files which exercise different program paths. car0 == '-' car1 == '7'

9. Program Example if (car0 == '-') { neg = true; } else { neg = false; cnt++; } if (car1 >= '0' && car1 <= '5') { val = car1 - '0'; } else { val = car1; } • File reading code. • Want to generate files which exercise different program paths. car0 == '+' car1 == '3'

10. Program Example if (car0 == '-') { neg = true; } else { neg = false; cnt++; } if (car1 >= '0' && car1 <= '5') { val = car1 - '0'; } else { val = car1; } • File reading code. • Want to generate files which exercise different program paths. car0 == '+' car1 == '9'

11. Related Tools • Cute, EXE, SAGE, catchconv, Apollo • Smart fuzzers - programs that generate interesting new inputs for programs. • Not for Java. • JCute • Smart fuzzer for Java. • Reinstruments code – Requires source files. • Has problems with the JDK.

12. Overall Idea • Input • A compiled (into bytecode) Java program. • A valid input file. • Output • New input files which exercise unique control paths. • Run the subject program in a modified JVM. • A logic predicate, the Path Condition, is formed as the program executes. • Describes control flow of execution. • New inputs are created by manipulating the path condition.

13. Example public void top(char[] input) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt > 3) crash(); } good

14. Example public void top(char[] input) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt > 3) crash(); } I0 != 'b' I1 != 'a' I2 != 'd' I3 != '!' path condition good • Negate constraints in path condition. • Solve the new path condition to create new inputs.

15. Example public void top(char[] input) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt > 3) crash(); } I0 != 'b' I1 != 'a' I2 != 'd' I3 == '!' goo! good

16. Example public void top(char[] input) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt > 3) crash(); } I0 != 'b' I1 != 'a' I2 == 'd' godd good

17. Example bood public void top(char[] input) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt > 3) crash(); } gaod godd goo! good All paths are explored systematically.

18. Tool Used • NASA Java PathFinder • Dynamic analysis framework for Java implemented as a JVM. • Allows backtracking including saving and restoring the whole state of the VM. • Can execute all thread interleavings. • Can execute a program on all possible inputs.

19. Attributes • Additional state-stored information. • Associated with runtime values. • JPF propagates attributes across assignment, method calls, etc. • Allows us to keep track of how the variables relate to the input using symbolic expressions.

20. Concrete v. Concolic Normal With Attributes exp0 exp1 3 4 3 4 + + 7 Sum (exp0, exp1) 7 Concrete Concolic execution is both concrete and symbolic. Symbolic

21. public class IADD extends ...bytecode.IADD { public Instruction execute (... ThreadInfo th) { [1] int v1 = th.pop(); [2] int v2 = th.pop(); [3] th.push(v1 + v2, ...); [4] StackFrame sf = th.getTopFrame(); [5] IntExpr sym_v1 = sf.getOperandAttr(); [6] IntExpr sym_v2 = sf.getOperandAttr(); [7] if (sym_v1 == null && sym_v2 == null) return getNext(th); [7] IntExpr result = sym_v1._plus(sym_v2); [8] sf.setOperandAttr(result); [9] return getNext(th); } } Concrete v. Concolic public class IADD extends Instruction { public Instruction execute (... ThreadInfo th) { [1] int v1 = th.pop(); [2] int v2 = th.pop(); [3] th.push(v1 + v2, ...); [4] return getNext(th); } } Concrete Symbolic

22. jFuzz Architecture Subject and Input • Runs JPF many times on the subject program and input files. • Each run: • Collects the Path Condition (PC). • Negates each constraint, reduces, and solves. • Uses new PCs to generate new input files. • Keeps track of inputs which caused exceptions to be thrown. jFuzz JPF Subject and Original Input PC Negated PC Solver New Input Inputs which cause crashes

23. Creating New Inputs • For a given execution some parts of the input may not be read.

24. Creating New Inputs • For a given execution some parts of the input may not be read.

25. Creating New Inputs • For a given execution some parts of the input may not be read. • When the path condition is solved, only the read parts will have new values.

26. Creating New Inputs • For a given execution some parts of the input may not be read. • When the path condition is solved, only the read parts will have new values. • The changes are written over the original input, preserving the unused parts.

27. Reducing the Path Condition Subject and Input • Path Conditions can be very long. • Not all constraints are effected by negating the PC. • Constraints not effected can be removed from the PC. jFuzz JPF Subject and Original Input PC Negated PC PC Minimizer Solver New Input Inputs which cause crashes

28. Example Reduction Path Condition: [1] a + b < 10 [2] b > 6 [3] c < 15 [4] a < 3 [5] c + d > 7 [6] e != 1 [7] c – e = 5 [8] a == 2

29. Path Condition: [1] a + b < 10 [2] b > 6 [3] c < 15 [4] a < 3 [5] c + d > 7 [6] e != 1 [7] c – e = 5 [8] a != 2 Example Reduction • Start with fuzzing the last constraint.

30. Path Condition: [1] a + b < 10 [2] b > 6 [3] c < 15 [4] a < 3 [5] c + d > 7 [6] e != 1 [7] c – e = 5 [8] a != 2 Example Reduction • Start with fuzzing the last constraint. • Select all constraints which contain variables in that constraint.

31. Path Condition: [1] a + b < 10 [2] b > 6 [3] c < 15 [4] a < 3 [5] c + d > 7 [6] e != 1 [7] c – e = 5 [8] a != 2 Example Reduction • Start with fuzzing the last constraint. • Select all constraints which contain variables in that constraint. • If one of the constraints contains multiple variables, select all constraints which contain those variables.

32. Path Condition: [1] a + b < 10 [2] b > 6 [4] a < 3 [8] a != 2 Example Reduction • Start with fuzzing the last constraint. • Select all constraints which contain variables in that constraint. • If one of the constraints contains multiple variables, select all constraints which contain those variables. • All other constraints can be removed.

33. Path Condition: [1] a + b < 10 [2] b > 6 [3] a < 3 [4] a != 2 Example Reduction • Start with fuzzing the last constraint. • Select all constraints which contain variables in that constraint. • If one of the constraints contains multiple variables, select all constraints which contain those variables. • All other constraints can be removed. • Variables not in the new PC are left unchanged.

34. Reducing the Path Condition • Reductions are performed for every constraint that is negated. • jFuzz uses a UnionFind data structure to find which variables are connected to each other. • In our case study, the average reduction was from around 250 constraints to about 5 constraints.

35. Case Study test1.dimacs c test 3 single clauses and 2 c binary clauses p cnf 4 5 1 0 2 0 3 0 -2 4 0 -3 4 0 • Subject: Sat4J • SAT solver written in Java. • Takes inputs in dimacs files. • ~10 kloc. • Goals: • Create inputs that crash Sat4J. • Create a set of good inputs.

36. Results • After 30 minutes of execution: • 12,000 input files were created. • 70 crashes where found. • The crashes are actually normal for SAT4J • 38 Invalid DIMACS files. • 27 Contradictions. • 4 Assertion Errors. • A Java compiler would be more compelling. • Any crash is due to a bug in the compiler. • Much larger program.

37. Performance • Sat4J was run 100 times in each VM. • The times are average runtime. • Simplifying the Path Condition reduces the solving time by 30%.

38. Conclusions • jFuzz is the first concolic tester for Java which will work for any bytecode. • This opens the door for more advanced fuzzing techniques, such as grammar based fuzzing.