1 / 29

Automated Whitebox Fuzz Testing

Automated Whitebox Fuzz Testing. Patrice Godefroid (Microsoft Research) ‏ Michael Y. Levin (Microsoft Center for Software Excellence) ‏ David Molnar (UC-Berkeley, visiting MSR). Fuzz Testing. An effective technique for finding

marva
Download Presentation

Automated Whitebox Fuzz Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automated Whitebox Fuzz Testing Patrice Godefroid (Microsoft Research)‏ Michael Y. Levin (Microsoft Center for Software Excellence)‏ David Molnar (UC-Berkeley, visiting MSR)

  2. Fuzz Testing • An effective technique for finding • security vulnerabilities in software • Apply invalid, unexpected, or random • data to the inputs of a program. • If the program fails(crashing, access violation exception), the defects can be noted.

  3. Vulnerability Finding Today • One Security bug can bring $500-$100,000 on the open market • Good bug finders make $180-$250/hr consulting • Few companies can find good people, many don’t even realize this is possible. • Still largely a black art

  4. Whitebox Fuzzing • This paper present an alternative whitebox fuzz testing approach inspired by recent advances in symbolic execution and dynamic test generation • Run the code with some initial well-formed input • Collect constraints on input with symbolic execution • Generate new constraints (by negating constraints one by one) • Solve constraints with constraint solver • Synthesize new inputs

  5. Dynamic Test Generation void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } input = “good” • Running the program with random values for the 4 input bytes is unlikely to discovery the error: a probability of about • Traditional fuzz testing is difficult to generate input values that will drive program through all its possible execution paths.

  6. Depth-First Search I0 != ‘b’ I1 != ‘a’ void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I2 != ‘d’ I0 != ‘b’ I3 != ‘!’ I1 != ‘a’ I2 != ‘d’ I3 != ‘!’ good

  7. Depth-First Search void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I0 != ‘b’ I1 != ‘a’ I2 != ‘d’ I3 == ‘!’ good goo!

  8. Practical limitation void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I0 != ‘b’ I1 != ‘a’ I2 == ‘d’ I3 != ‘!’ good godd Every time only one constraint is expanded, low efficiency!

  9. Generational Search Key Idea: One Trace, Many Tests bood void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } gaod I0 == ‘b’ godd I1 == ‘a’ I2 == ‘d’ I3 == ‘!’ good goo! “Generation 1” test cases

  10. The Generational Search Space void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } 0 good 1 goo! 1 godd 2 god! 1 gaod 2 gao! 2 gadd 1 bood 2 boo! 2 bodd 2 baod gad! bod! baod badd bad!

  11. SAGE Architecture(Scalable Automated Guided Execution) Constraints Input0 Trace File Check for Crashes (AppVerifier)‏ Trace Program (Nirvana)‏ Gather Constraints (Truscan)‏ Solve Constraints(Disolver) Input1 Input2 … InputN

  12. Initial Experiences with SAGE • Since 1st MS internal release in April’07: dozens of new security bugs found(most missed by blackbox fuzzers, static analysis)‏ • Apps: image processors, media players, file decoders,… • Many bugs found rated as “security critical, severity 1, priority 1” • Now used by several teams regularly as part of QA process

  13. ExperimentsANI Parsing - MS07-017 RIFF...ACONLIST B...INFOINAM.... 3D Blue Alternat e v1.1..IART.... ................ 1996..anih$...$. ................ ................ ..rate.......... ..........seq .. ................ ..LIST....framic on......... .. RIFF...ACONB B...INFOINAM.... 3D Blue Alternat e v1.1..IART.... ................ 1996..anih$...$. ................ ................ ..rate.......... ..........seq .. ................ ..anih....framic on......... .. Crashing test case Initial input

  14. ANI Parsing - MS07-017 RIFF...ACONLIST B...INFOINAM.... 3D Blue Alternat e v1.1..IART.... ................ 1996..anih$...$. ................ ................ ..rate.......... ..........seq .. ................ ..LIST....framic on......... .. RIFF...ACONB B...INFOINAM.... 3D Blue Alternat e v1.1..IART.... ................ 1996..anih$...$. ................ ................ ..rate.......... ..........seq .. ................ ..anih....framic on......... .. Only 1 in 232 chance at random! Crashing test case Initial input Initial input : 341 branch constraints, 1,279,939 total instructions. Crash test cases: 7706 test cases, 7hrs 36 mins

  15. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 0 – initial input – 100 bytes of “00”

  16. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 00 00 00 00 00 00 00 00 00 00 00 00 ; RIFF............ 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 1

  17. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 00 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF....*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 2

  18. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 3

  19. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 00 00 00 00 ; ....strh........ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 4

  20. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 5

  21. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 00 00 00 00 ; ....strf........ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 6

  22. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 28 00 00 00 ; ....strf....(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 7

  23. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 28 00 00 00 ; ....strf....(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 C9 9D E4 4E ; ............ÉäN 00000060h: 00 00 00 00 ; .... Generation 8

  24. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 28 00 00 00 ; ....strf....(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 9

  25. Zero to Crash in 10 Generations • Starting with 100 zero bytes … • SAGE generates a crashing test: 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 B2 75 76 3A 28 00 00 00 ; ....strf²uv:(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... Generation 10 – bug ID 1212954973! Found after only 3 generations starting from “well-formed” seed file

  26. 1867196225 X X X X X 2031962117 X X X X X 612334691 X X 1061959981 X X 1212954973 X X 1011628381 X X X 842674295 X 1246509355 X X X 1527393075 X 1277839407 X 1951025690 X Different Crashes From Different Initial Input Files 100 zero bytes Bug ID seed1 seed2 seed3 seed4 seed5 seed6 seed7 Media 1: 60 machine-hours, 44,598 total tests, 357 crashes, 12 bugs

  27. Most Bugs Found are “Shallow” Generation

  28. Conclusion Blackbox vs. Whitebox Fuzzing • Cost/precision tradeoffs • Blackbox is lightweight, easy and fast, but poor coverage • Whitebox is smarter, but complex and slower • Recent “semi-whitebox” approaches • Less smart but more lightweight: Flayer (taint-flow analysis, may generate false alarms), Bunny-the-fuzzer (taint-flow, source-based, heuristics to fuzz based on input usage), autodafe, etc. • Which is more effective at finding bugs? It depends… • Many apps are so buggy, any form of fuzzing finds bugs! • Once low-hanging bugs are gone, fuzzing must become smarter: use whitebox and/or user-provided guidance (grammars, etc.)‏ • Bottom-line: in practice, use both!

  29. Thank you! Questions?

More Related