1 / 21

Software Verification Methods Rodeo: Enhancing Tool Efficacy

This project aims to assess tools based on formal methods for verifying real-world software through a series of competitions called the Rodeo. The goal is to motivate the use and improvement of such tools by measuring their efficacy. Steps involve developing a complexity measurement tool, selecting a test suite, running verifiers in competitions, assessing findings, and identifying research opportunities. The approach focuses on identifying and measuring complexity in test suites, calibration, and engagement with the software community to validate results. The project also involves setting up a software laboratory for analysis, manipulation, and transformation of software, enabling quick evaluation of new techniques and approaches. By engaging with the community, the project seeks to confirm or refute hypotheses regarding software properties and behaviors.

fsullivan
Download Presentation

Software Verification Methods Rodeo: Enhancing Tool Efficacy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Proposal For aSoftware Verification Formal Methods Rodeo Paul E. Black Lee Badger Chris Johnson Vadim Okun Barbara Guttman David Flater https://samate.nist.gov/FMSwVRodeo/

  2. The Goal: Measure the efficacy of tools based on formal methods for verification of real-world software to motivate the use and improvement of such tools.

  3. Steps of the Rodeo: • Develop a tool to measure “complexity” properties of production software. Use it to • Find a suite of test programs. • Run verifiers and proof validators against the suite in a competition or “rodeo.” • Assess findings to quantify improvement and identify research opportunities. Repeat steps 2 through 4

  4. Our Approach Software Complexity Photo by Jordan Heinrichs on Unsplash

  5. SV-COMP Test Suite Size & #

  6. A Complexity Measurement Tool • Identify quantities embodied in test suites of SV-COMP, RERS, VerifyThis, etc.

  7. A Complexity Measurement Tool • Identify quantities embodied in test suites of SV-COMP, RERS, VerifyThis, etc. • Build a tool to measure those quantities.

  8. A Complexity Measurement Tool • Identify quantities embodied in test suites of SV-COMP, RERS, VerifyThis, etc. • Build a tool to measure those quantities. • Calibrate tool against those test suites.

  9. A Complexity Measurement Tool of properties that are • automated, • objective, and • relevant.

  10. Our Approach Software Complexity Test Suites Photo by Jordan Heinrichs on Unsplash

  11. Test Suite • Realistic software (and specs), of selected size, with various properties to “prove.”

  12. Test Suite Criteria • Select from existing applications • possibly generate code from those • Computational software • Not embedded, distributed, real time, etc. • Open source, good quality, widely used • Some software not in C • Prefer software that has formal specs

  13. Our Approach Software Complexity Test Suites Verification & Proof Validation Photo by Jordan Heinrichs on Unsplash

  14. Stereotype Verification Source code Verify Pass (Fail)

  15. Two-Step: Verify and Check Source code Verify Witnesses Witness Checker Pass (Fail)

  16. Two-Step: Verify and Check Source code Verify Witnesses Witness Checker can be done by user fairly automatically with high assurance Pass (Fail)

  17. Two-Step: Verify and Check Source code Verify Witnesses Witness Checker can be done by user fairly automatically with high assurance Proof Checker

  18. Our Approach Software Complexity Test Suites Software Laboratory Verification & Proof Validation Photo by Jordan Heinrichs on Unsplash

  19. Set Up a Software Laboratory Photo by Windell Oskay Used under Creative Commons 2.0

  20. Set Up a Software Laboratory • Methods and programs to manipulate, transform, and analyze software … and the in-house expertise to use them. • In order to • Quickly get an idea of the potential of new techniques and approaches. • Confirm (or refute) hypothesis … would we find it if the code was like this?

  21. Our Approach Software Complexity Community Engagement Test Suites Software Laboratory Verification & Proof Validation Photo by Jordan Heinrichs on Unsplash

More Related