1 / 34

Software Verification Tools: Current State and the Nearest Future

Victor Kuliamin Institute for System Programming Russian Academy of Sciences Moscow. Software Verification Tools: Current State and the Nearest Future. Software development trends. Bugs per 1000 LOC. Growth of software complexity Degradation of software quality. Software Verification.

chin
Download Presentation

Software Verification Tools: Current State and the Nearest Future

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Victor Kuliamin Institute for System Programming Russian Academy of Sciences Moscow Software Verification Tools: Current State and the Nearest Future

  2. SYRCoSE 2009, May 28 Software development trends Bugs per 1000 LOC Growth of software complexity Degradation of software quality

  3. SYRCoSE 2009, May 28 Software Verification Development Processes Standards and Rules Requirements Design Source Code System Operation Checking consistency between different development artifacts, relevant standards, and between them all and real system operation

  4. SYRCoSE 2009, May 28 Traditional Verification methods • Review (inspection) • Static analysis • Correctness rules checking • Bug search • Dynamic analysis • Monitoring • Testing • Formal methods • Theorem proving • Model checking

  5. SYRCoSE 2009, May 28 Static and dynamic analysis   Analysis Tool Requirements & Rules Source Code System Operation   Test Generation Requirements & Rules Monitoring Environment Users Static analysis Dynamic analysis

  6. SYRCoSE 2009, May 28 Formal verification methods • Theorem proving [R. Floyd 1967, C. A. R. Hoare 1969] • Hoare logic – {Pre} Program {Post} • Inference rules • Model checking [E. M. Clarke & E. A. Emerson 1980, J. P. Queille & J. Sifakis 1982] • Analysis of reachable states

  7. SYRCoSE 2009, May 28 Hybrid Verification Techniques Testing Formal models Monitoring Static analysis • Model based testing • Extended static analysis • Runtime verification • Compound structured testing • Auxiliary • Symbolic execution • Abstract interpretation • Constraint inference • Constraint resolution

  8. SYRCoSE 2009, May 28 Symbolic execution if(x > 0) { y := x+2; } else if(x > -1) { y := x+1; } else { y := x; } [(x > 0)  (y = x+2)] & [(x ≤ 0 & x > -1)  (y = x+1)] & [[(x ≤ -1)  (y = x) ] [J. C. King, L. A. Clark 1976]

  9. SYRCoSE 2009, May 28 abstract Interpretation ... while ( (x == 0) && (2*f(x) <= z+g(y/2.5)) ) { ... x++; } ... ... while ( B ) { ... B = false; } ... [P.Cousot & R. Cousot 1977] • Abstract Domains • Octagons x  y ≤ a • Polyhedra • Heap structures • Bit vectors • …

  10. SYRCoSE 2009, May 28 Constraint inference • Daikon 1999 MIT M. D. Ernst http://groups.csail.mit.edu/pag/daikon/ • =, ≠, <, ≤, ax + by + cz + d = 0, x = y2, x % y = 0 • , , , , AB=, subsequence, no duplicates,reverse • Houdini 2001 Flanagan, Leino • DIDUCE 2002 StanfordUniversity • Agitator 2003 Agitar • DySy2007 Tillmann, Csallner, Smaragdakis

  11. SYRCoSE 2009, May 28 Constraint resolution SMT-LIB http://combination.cs.uiowa.edu/smtlib/ SAT solvers – DPLL algorithm 1962 SMT solvers – Satisfiability modulo Theory CVC 2002 StanfordD. L. Dill, C. W. Barrett, A. Stump Yices 2005 SRI InternationalB. Dutertre, L. de Moura Z3 2006 Microsoft ResearchL. de Moura, N. S. Bjørner

  12. SYRCoSE 2009, May 28 Hybrid Verification Techniques Model based testing Extended static analysis Runtime verification Compound structured testing

  13. SYRCoSE 2009, May 28 Model based testing (MBT) Test action generator 12% 36% 87% 57% Adequacy criteria Adequacy metric System under test Behavior model State model State model and oracle FSM-based testing [F. C. Hennie 1964, M. P. Wasilevsky 1973]

  14. SYRCoSE 2009, May 28 Model based testing Tools • Automata (FSM, LTS, ASM) • TGV 1997 INRIA T. Jéron et al.http://www.inrialpes.fr/vasy/cadp/man/tgv.html • TorX 1999 University of TwenteJ. Tretmans et al.http://fmt.cs.utwente.nl/tools/torx/introduction.html • Gotcha-TCBeans 1999 IBM ResearchA. Hartman et al. • Automata + Software Contracts • UniTESK 2000 ISP RASA. Petrenko et al.http://www.unitesk.com • SpecExplorer 2004 Microsoft ResearchW. Schulte et al.+ symbolic executionhttp://research.microsoft.com/en-us/projects/SpecExplorer/

  15. SYRCoSE 2009, May 28 NEXT generation of MBT tools ModelJUnit 2004 Waikato University M. Utting et al.http://czt.sourceforge.net/modeljunit/index.html NModel 2007 Univ. of Washington Microsoft ResearchJ.Jacky, M. Veanes et al.http://nmodel.codeplex.com/

  16. SYRCoSE 2009, May 28 Nmodel Example (part I) namespaceClientServer { [Feature]publicpartial classServer { publicstaticSocketserverSocket = Socket.None; publicstaticPhasephase = Phase.Send; publicstaticboolServerSocketEnabled() { return (serverSocket == Socket.None); } [Action]publicstaticvoidServerSocket() { serverSocket = Socket.Created; } publicstaticboolServerBindEnabled() { return (serverSocket == Socket.Created); } [Action]publicstaticvoidServerBind() { serverSocket = Socket.Bound; } publicstaticboolServerListenEnabled() { return (serverSocket == Socket.Bound); } [Action]publicstaticvoidServerListen() { serverSocket = Socket.Listening; } publicstaticboolServerAcceptEnabled() { return (serverSocket == Socket.Listening); } [Action]publicstaticvoidServerAccept() { serverSocket = Socket.Connected;} publicstaticboolServerReceiveEnabled() { return (serverSocket == Socket.Connected && phase == Phase.ServerReceive); } [Action]publicstaticvoidServerReceive() { phase = Phase.Send; } }

  17. SYRCoSE 2009, May 28 NMODEL EXAMPLE (Part II) [Feature]publicpartialclassClient { publicstaticSocketclientSocket = Socket.None; publicstaticdoubleclientBuffer = double.MaxValue; publicstaticboolClientSocketEnabled() { return (clientSocket == Socket.None); } [Action]publicstaticvoidClientSocket() { clientSocket = Socket.Created; } publicstaticboolClientConnectEnabled() { return (clientSocket == Socket.Created); } [Action]publicstaticvoidClientConnect() { clientSocket = Socket.Connecting; } publicstaticboolClientSendEnabled() { return (clientSocket == Socket.Connected); } [Action]publicstaticvoidClientSend() { phase = Phase.ServerReceive; } publicstaticboolClientReceiveEnabled() { return (clientSocket == Socket.Connected); } [Action]publicstaticdoubleClientReceive(double datum) { clientBuffer = datum;returndatum;} publicstaticboolClientCloseEnabled() { return (clientSocket == Socket.Created || clientSocket == Socket.Connected); } [Action]publicstaticvoidClientClose() { clientSocket = Socket.Closed; } }

  18. SYRCoSE 2009, May 28 NMODEL EXAMPLE (Part III) [Feature]publicpartialclassServer { publicstaticboolClientConnectEnabled() { return (serverSocket == Socket.Listening); } publicstaticboolClientSendEnabled() { return (phase == Phase.Send); } [Action]publicstaticvoidClientSend() { phase = Phase.ServerReceive; } publicstaticboolClientReceiveEnabled() { return (phase == Phase.ClientReceive); } [Action]publicstaticvoidClientReceive() { phase = Phase.Send;} } [Feature]class Values2 { readonlystatic Set<double> Values = new Set<double>(99.9, 100.0); [Action]staticvoidClientReceive([Domain("Values")] double datum) {} }

  19. SYRCoSE 2009, May 28 NMODEL EXAMPLE (Part IV) [Feature]public partial classClient { public static boolServerAcceptEnabled() { return (clientSocket == Socket.Connecting); } [Action]public static voidServerAccept() {clientSocket = Socket.Connected; } } }

  20. SYRCoSE 2009, May 28 Nmodel example – composition Server

  21. SYRCoSE 2009, May 28 Test targeting based on composition

  22. SYRCoSE 2009, May 28 Extended static checking Behavior model Analysis Tool Requirements & Rules Source Code Provers, Solvers, Model Checkers [G. Nelson & J.B. Saxe et al. 1991] Search for bugs Sound analysis

  23. SYRCoSE 2009, May 28 Extended static checking tools • ESC/Modula 3 1995 DECG. Nelson, J.B. Saxe, K. R. M. Leino, D. Detlefs • ESC/Java 2000 CompaqK. R. M. Leino, C. Flanagan • ASTREE 2002 CNRSP. Cousothttp://www.astree.ens.fr/ • ESC/Java 2 2004http://kind.ucd.ie/products/opensource/ESCJava2/ • Simplify • Spec# Checker 2004 Microsoft ResearchK. R. M. Leinohttp://research.microsoft.com/SpecSharp/ • Boogie • Calysto2008 University of British Columbia

  24. SYRCoSE 2009, May 28 Solvers integration Boolean satisfiability (SAT) (x1~x2) & (~x1x3) Linear integer arithmetics x1 < x2 + 3 Floating-point arithmetics x1 · x2 = 2.5 Polyhedra 0.2x1+3x3 > x2–3.7x4 Ellipsoids 2x12+1.3x22 ≤ 76.9 Heap structures x1→p→p ≠ x2→p …

  25. SYRCoSE 2009, May 28 Automatic model refinement do { nPacketsOld = nPackets; ... if(request) { ... nPackets++; } } while (nPackets != nPacketsOld); do { b = true; ... if(request) { ... b = b?false:*; } } while (!b); Counterexample guided abstraction refinement CEGAR [E. M. Clarke & O. Grumberg et al 2000, T. Ball & S. K. Rajamani 2000]

  26. 26/ 14 SYRCoSE 2009, May 28 Automatic model refinement II Model Checking Correctness Rules Behavior Model Behavior Model Counterexample Model Refinement Test Data and Scenario Code under check Refutation Approval Bug!

  27. SYRCoSE 2009, May 28 Automatic model refinement tools • SLAM 2001 Microsoft ResearchT. Ball, S. K. Rajamani et al.http://research.microsoft.com/en-us/projects/slam/ • Static Driver Verifier 2007 Microsofthttp://www.microsoft.com/whdc/devtools/tools/sdv.mspx • BLAST 2003 UC BerkeleyT. A. Henzinger, R. Jhala, R. Majumdar, G. Sutrehttp://mtc.epfl.ch/software-tools/blast/ • MAGIC 2003 SCS Carnegie MellonE. M. Clarke, S. Chaki et al.http://www.cs.cmu.edu/~chaki/magic/

  28. SYRCoSE 2009, May 28 Runtime verification System under check Monitoring Environment State model and oracle Behavior model [K. Havelund & W. Visser 1999] Java Path Finder + symbolic execution + test generation http://javapathfinder.sourceforge.net/

  29. SYRCoSE 2009, May 28 Compound Structured testing [??? 2003-2004] • Targeting to cover various paths in source/byte code • Test oracle • No exceptionsNullPointer, IndexOutOfBounds, ClassCast, DivideByZero, IllegalArgument • Annotations and formal models • Test data and sequences generation • Random • Symbolic execution + constraint resolution • State abstraction • Heuristic search

  30. SYRCoSE 2009, May 28 crashers Daikon ESC/Java 2 solver [Y. Smaragdakis, C. Csallner] JCrasher 2004 Check-n-Crash 2005 DSD-Crasher 2006http://ranger.uta.edu/~csallner/dsd-crasher/

  31. SYRCoSE 2009, May 28 Directed automated random testing Execution Program Looking for new paths Symbolic Execution Tests [P. Godefroid, G. Agha, K. Sen 2005] CUTE2005 Consolic testing (concrete + symbolic) jCUTE 2006

  32. SYRCoSE 2009, May 28 DART II 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 28 00 00 00 ; ....strf....(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 B2 75 76 3A 28 00 00 00 ; ....strf²uv:(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 00 00 00 00 ; ....strh........ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 00 00 00 00 ; ....strf........ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 00 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF....*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 00 00 00 00 00 00 00 00 00 00 00 00 ; RIFF............ 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 28 00 00 00 ; ....strf....(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000040h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000060h: 00 00 00 00 ; .... 00000000h: 52 49 46 46 3D 00 00 00 ** ** ** 20 00 00 00 00 ; RIFF=...*** .... 00000010h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................ 00000030h: 00 00 00 00 73 74 72 68 00 00 00 00 76 69 64 73 ; ....strh....vids 00000040h: 00 00 00 00 73 74 72 66 00 00 00 00 28 00 00 00 ; ....strf....(... 00000050h: 00 00 00 00 00 00 00 00 00 00 00 00 C9 9D E4 4E ; ............ÉäN 00000060h: 00 00 00 00 ; .... Generation 6 Generation 0 – initial input – 100 bytes of “00” Generation 1 Generation 2 Generation 3 Generation 4 Generation 7 Generation 8 Generation 9 Generation 10 – bug ID 1212954973! Found after only 3 generations starting from “well-formed” seed file Generation 5 SAGE 2007 Pex 2007N. Tillmann, W. Schulte, J. de Halleuxhttp://research.microsoft.com/Pex/default.aspx

  33. SYRCoSE 2009, May 28 State abstraction EXE 2005 Stanford Univ.D. Dill, D.Engler et al. Randoop 2007 MIT + MS ResearchT. Ball, M. D. Ernst, C. Pacheco, S. Lahirihttp://people.csail.mit.edu/cpacheco/randoop/1.2/doc/ …

  34. SYRCoSE 2009, May 28 Conclusion – NEW Landscape • Modern verification tools • Use basic services of component technologies • Integrate a lot of techniques • Become highly modular • Can be modules of each other • Next step – integration frameworks ? • Java PathFinder • Bogorhttp://bogor.projects.cis.ksu.edu/ • Microsoft RiSEhttp://research.microsoft.com/en-us/um/redmond/groups/rise/

More Related