1 / 33

Verification Modulo Versions T owards usable verification

Verification Modulo Versions T owards usable verification. F. Logozzo , S. K. Lahiri, Microsoft Research, Redmond M. Fahndrich, now at Google S. Blackshear, Univ. Colorado, Boulder. Background. CodeContracts in VS. Compile-time verification 100% Abstract interpretation.

hosea
Download Presentation

Verification Modulo Versions T owards usable verification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Verification Modulo VersionsTowards usable verification F. Logozzo, S. K. Lahiri, Microsoft Research, Redmond M. Fahndrich, now at Google S. Blackshear, Univ. Colorado, Boulder

  2. Background

  3. CodeContracts inVS Compile-time verification 100% Abstract interpretation Used internally at MSFT > 120K external downloads Failed precondition

  4. Too many warnings

  5. Too many warnings: Now what? • False myth: Analysis too imprecise, let’s improve the analysis • In practice, the analysis is precise enough • Problem: It lacks information on external libraries, environment, physical world … • Academic solution: Address each warning • Add assumes, out-of-bands contracts … • Practice: No one is going to do it • Too expensive • Pragmatic: Syntactic baseline • Unsound and unreliable

  6. Disclaimer: over-simplified schema Syntactic baseline New version (k Renamed) Base version /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:var k = c + 273.15; 3:Contract.Assert(k >= 0); 4: returnk; } /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:varkelv = c + 273.15; 3:Contract.Assert(kelv>= 0); 4: returnkelv; } “mask assertion at line 3” Valid, but can’t prove assertion Masked: No alarm

  7. Syntactic baseline: Fail to spot regression New version (wrong constant) Base version /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:var k = c + 273.15; 3:Contract.Assert(k >= 0); 4: returnk; } /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:var k = c + 273.0; 3:Contract.Assert(k >= 0); 4: returnk; } “mask assertion at line 3” Masked regression!

  8. Syntactic baseline: Resurrect alarm New version (value rounded) Base version /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:var k = c + 273.15; 3:Contract.Assert(k >= 0); 4: returnk; } /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:vark = c + 273.15; 3: var round = Math.Round(k, 2); 4:Contract.Assert(round >= 0); 5: return round; } “mask assertion at line 3” Alarm Resurrected

  9. Straw man? • Of course, we can refine the former syntactic baseline technique • Add the assertion name, have a window instead of line number… • However, they are always brittle, and provide no guarantee • Suppress too little • Is the alarm resurrected from the old one? • Suppress too much • Is the masked alarm a new alarm?

  10. Verification Modulo Versions Intuition

  11. VMV: Verification modulo versions P’ New version P Base version P’+C Instrumented Analyze Instrument Analyze Analyzer is a black box Reduce alarms up to 70% With semantic guarantees!!! Extract semantic conditions C

  12. VMV(S): Finding regressions Base version Instrumented new version /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:var k = c + 273.15; 3:Contract.Assert(k >= 0); 4: returnk; } /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); Contract.Assume(c >= -273.15); 2:var k = c + 273.0; 3:Contract.Assert(k >= 0); 4: returnk; } Result(ReadTemperature) ≥ -273.15 Correct ⇔ c≥ -273.15 Report regression!

  13. VMV(N): Relative proofs Base version Instrumented new version /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); 2:var k = c + 273.15; 3:Contract.Assert(k >= 0); 4: returnk; } /// Get temperature in Kelvin doubleGetCurrTemperature() { // In Celsius 1:varc = ReadTemperature(); Contract.Assume(c >= -273.15); 2:vark = c + 273.15; 3: var round = Math.Round(k, 2); 4:Contract.Assert(round >= 0); 5: return round; } Result(ReadTemperature) ≥ -273.15 Correct ⇔ c≥ -273.15 Relative correctness

  14. Questions 2. How do we inject them? P’ New version P Base version P’+C Instrumented Analyze Instrument Analyze 3. What guarantees? Extract semantic conditions C 1. What conditions?

  15. Verification Modulo Versions Formalization

  16. Q1. What conditions? • Critical to distinguish between sufficient and necessary conditions • A sufficient condition S implies the program correctness • S ⟹ “program has no bad runs” • S can be under-approximated • At worst false • A necessary condition N holds in all correct runs • If R is a good program run, then R ⟹ N • N can be over-approximated • At worst true

  17. Example: Sufficient vs. Necessary publicvoid Foo() { int ret, x; if(*) { ret = API(); x = ret + 1; } else { ret = API(); x = ret - 1; } Contract.Assert(x > 100); } Property of good runs Strongest Necessary for API ret > 99 ∨ ret >101 = ret > 99 ret > 99 Removes all bad runs Weakest Sufficient for API ret > 99 ∧ret >101 = ret > 101 ret > 101

  18. Q2: Matching and injection publicvoid Foo() { int ret, x; if(*) { ret = API(); x = ret + 1; } else { ret = API(); x = ret - 1; } Contract.Assert(x > 100); } Match method calls publicvoidFoo() { intx; x = API(); Contract.Assert(…); } • Other matching solutions: • Program points, Sequences of method calls, … • See paper • In general • More refined matching ⟹ more precision • More refined matching ⟹ more brittle

  19. Q3: What semantic guarantees for P’? • Using sufficient conditions • Short VMV(S) • Theorem: All warnings in P’+ S are new alarms • S was sufficient to shut off all alarms in P • Each new alarm in P’+S is either • 1. in new code or • 2. S is not strong enough • Nothing can be said on the “verified” assertions • E.g., S can be too strong • VMV(S) useful for bug finding

  20. Example of VMV(S) publicvoid Foo() { int ret, x; if(*) { ret = API(); x = ret + 1; } else { ret = API(); x = ret - 1; } Contract.Assert(x > 100); } publicvoidFoo() { intx; x = SomeAPI(); Contract.Assume(x > 101); Contract.Assert(x > 105); } Recall Suff. condition: Result(API) > 101 Regression: Ask more to API

  21. Q3: What semantic guarantees for P’? • Using necessary conditions • Short VMV(N) • Theorem: All proven assertions in P’+ N are correct w.r.t. P • N holds in all good runs of P • Each proven assertion in P’+ N is either • 1. because of an assume from N, or • 2. absolute, i.e., without assumptions from P • Alarms may be new or old ones • E.g. N can be too weak • VMV(N) is useful for relative verification

  22. Example of VMV(N) publicvoid Foo() { int ret, x; if(*) { ret = API(); x = ret + 1; } else { ret = API(); x = ret - 1; } Contract.Assert(x > 100); } publicvoidFoo() { intx; x = SomeAPI(); Contract.Assume(x > 99); Contract.Assert(x > 0); } RecallNec. condition: Result(API) > 99 Relative verification: Holds for good runs of P

  23. Experiments

  24. Experimental setting Match method calls P’ New version P Base version P’+N Instrumented cccheck.exe Instrument Necessary ⟹ relative verification cccheck.exe Extract necessary conditions N Cccheck.exe checks safety properties Contracts and runtime exceptions null-pointers, division by zero, buffer overruns, arithmetic overflow, enums …

  25. Windows Azure ActorFX framework • Three months between P and P’ • VMV(N) dramatically reduces the number of alarms (roughly by 70%) • Eliminated great majority of the warnings from external non-annotated API • Added annotations to P’+ N to go to 0 warnings, i.e., relative correct • Less than 3 hours overall • Found bugs, reported to the developers

  26. Conclusions

  27. More in the paper • Abstraction of necessary/sufficient • Role of imprecision in the analyzer • Abstract regressions in VMV(S) • Impact of non-monotonic analyzers • i.e., all tools out there… • Self application • Analyze P+N • Related work • … Scenarios Greek letters Numbers

  28. Conclusions Match method calls P’ New version P Base version P’+C Instrumented Analyze Instrument Sufficient ⟹ bug finding Analyze Extract semantic conditions C Necessary ⟹ relative verification Sufficient or Necessary

  29. Backup slides

  30. Comparison with syntactic baseline • Our starting point: Syntactic baseline in cccheck.exe • Problems: • Suppresses too much • No semantic guarantee • VMV addresses those problems • In practice: Both options available to users

  31. Related work • Industrial tools: Coverity, Polyspace, Abductor… • They: Tool-specific syntactic matching and/or hashing • VMV: General, rigorous framework, semantic-based • Differential Assertion Checking [Lahiri et al. FSE’13] • They: Construct and analyze product program • VMV: Framework for augmenting any static analysis, lightweight(pre+post processing) Differential symbolic execution methods [Person et al. FSE’08, Ramos et al. CAV’11] • They: Special analyzers for finding differences, no semantic relative guarantees

  32. Related work • Enhanced Diff tools: Smart strategies to find syntactic changes among versions • Orthogonal to our work, can only enhance VMV • Annotation inference: • Our framework is independent of the inference technique • We consider both sufficient and necessary

  33. Experimental setting • Implemented VMV(N) in Clousot/cccheck • Developed for 7+ years at Microsoft Research • Used by .NET programmers, free download, VS integration • Internal and external adoption • Abstract interpretation-based • Analyze program to infer invariants at each program point • Use invariants to discharge proof obligations • Contracts, null-pointers, division by zero, buffer overruns, arithmetic overflow, enums …. • Infer necessary conditions • Run on very large code bases • Internal and open source

More Related