1 / 10

Software Assurance Metrics and Tool Evaluation (SAMATE)

Michael Kass National Institute of Standards and Technology http://samate.nist.gov/ Michael.kass@nist.gov. Software Assurance Metrics and Tool Evaluation (SAMATE). Outline . Overview of Software Assurance (SwA) tool testing at NIST Description of SAMATE project Follow-on.

walden
Download Presentation

Software Assurance Metrics and Tool Evaluation (SAMATE)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Michael Kass National Institute of Standards and Technology http://samate.nist.gov/ Michael.kass@nist.gov Software Assurance Metrics and Tool Evaluation (SAMATE)

  2. Outline Overview of Software Assurance (SwA) tool testing at NIST Description of SAMATE project Follow-on

  3. Dept Homeland Security Concern Do software assurance tools work as they should? Do they really find vulnerabilities and catch bugs? How much assurance does running the tool provide? Software Assurance tools should be: Tested (accurate and reliable) Peer reviewed Generally accepted

  4. Goals of SAMATE Develop metrics for the effectiveness of SwA tools and to identify deficiencies in software assurance methods and tools Perform SwA R&D to assess current methods and tools in order to identify deficiencies which can lead to software product failures and vulnerabilities Identify gaps in methods and tools and suggest areas of research

  5. The NIST SAMATE Project(Software Assurance Metrics and Tool Evaluation) Conduct surveys Tools Researchers and companies Host workshops & conference sessions Taxonomy of SwA functions and techniques Order of importance (cost/benefit, criticalities, …) Gaps and research agendas Studies to develop tool effectiveness metrics Evaluate tools Detailed specification Test plans Host reference dataset library

  6. A Taxonomy of Static Analysis Tool Functions • Language • Source/Binary analysis • Semantic checking (abstract syntax tree) • Interprocedural analysis • Strong type checking (type casting vulnerabilities, uninitialized variable use) • Memory allocation checking (memory leaks, deallocation of unallocated memory) • Logic checking (unnecessary code, unreachable code) • Interface checking (include file cycling) • Security checking • Buffer overflow/underflow • Stack overflows • Heap overflows • Integer overflow/underflow • Tainted data • Error path problems • Locking problems • Code metric generation (LOC, number of methods, levels of inheritance)

  7. SA Tool Effectiveness Metrics What constitutes a tool’s effectiveness metric? • Number of defects detected vs. total defects • Number of false positives • Number of false negatives • …

  8. Documenting tool effectiveness • Tool functional specification • Test plan • Reference dataset • Test report

  9. Workshop1 SA classes Workshop 2 fill gaps Workshop 3 Define Metric focus group class 1 focus group class 2 focus group class 2 focus group class 1 SAMATE Project Timeline T(mos.) 1 2 3 4 5 6 9 12 15 18 21 24 Tool Survey Function Taxonomy Survey Publication tool testing matrix Spec0 test reports select func test plan Spec1 strawman spec test plan draft test reports select func Spec0 test reports test plan Spec1 strawman spec test plan test reports draft

  10. Contact for SAMATE Participation Paul Black Project Leader, Software Diagnostics & Conformance Testing Division, Software Quality Group paul.black@nist.gov

More Related