1 / 14

NIST-developed Test Suites

NIST-developed Test Suites. David Flater National Institute of Standards and Technology http://vote.nist.gov. Context. Status quo Labs have been testing to 2002, 1.0 (2005) Lots of proprietary, custom tooling and review processes August 2007 draft of 2.0 (1.9)

marva
Download Presentation

NIST-developed Test Suites

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NIST-developed Test Suites David Flater National Institute of Standards and Technology http://vote.nist.gov

  2. Context Status quo Labs have been testing to 2002, 1.0 (2005) Lots of proprietary, custom tooling and review processes August 2007 draft of 2.0 (1.9) NIST defined test methods where appropriate for entire scope In case of omission or error, fall back to status quo is an option May 2009 draft of 1.1 (1.01) Test methods for new and changed material back-ported from 2.0 (1.9) Status quo prevails for everything else 12/9-10/2009 TGDC Meeting Page 2

  3. Materials 12/9-10/2009 TGDC Meeting Page 3

  4. General principles Ability to specify tests is limited by variability of design, unavailability of system-specific details, nature of requirements Avoid creating new requirements via test suite Measure when possible Measurable quantity Benchmark If not measurable, specify procedure for evaluating Test narratives Checklists Pass/fail criteria … Assessment by qualified experts 12/9-10/2009 TGDC Meeting Page 4

  5. Coverage strategies The VVSG contains many different sorts of requirements… some more testable than others Coverage strategies vary "as appropriate" to what is being tested One test per requirement, one requirement per test At least one test per requirement, many requirements per test Many tests for one requirement Incidental testing Derived requirements Tested by exception 12/9-10/2009 TGDC Meeting Page 5

  6. Usability & accessibility System-independent test narratives with pass/fail criteria CIF template & how-tos Common Industry Format for reporting usability test results Highly structured process surrounding the usability test(s) Voting Performance Protocol (VPP) is only in 2.0 Specified benchmarks Usability measured by test lab 12/9-10/2009 TGDC Meeting Page 6

  7. Security test suite Procedures for evaluating Cryptography Access control Event logging Other security features Secure configurations If X is used, then it should be configured thusly… Not OEVT 12/9-10/2009 TGDC Meeting Page 7

  8. Votetest Scope: basic, essential voting system logic Ability to define elections, capture, count, and report votes Voting variations Tests are intentionally simple… A volume test (mock election) is a significant test of all supported functions together …but they exercise the complete elections and voting process Election definition and test ballots in; reports out 92 tests formalized as SQL scripts 12/9-10/2009 TGDC Meeting Page 8

  9. Volume testing guidance A volume test is specified in Part 3 of 2.0 Additional guidance for conducting that test has been published in a separate document 12/9-10/2009 TGDC Meeting Page 9

  10. Reliability, accuracy, misfeed rate Improved test method replaces material that was historically included in the VSS/VVSG… hence, included in drafts Now evaluated using data collected during all tests, rather than a single, isolated test 12/9-10/2009 TGDC Meeting Page 10

  11. Hardware workmanship General build quality Durability Maintainability Temperature and humidity Operating humidity requirement added to 1.1 Transportation and storage 12/9-10/2009 TGDC Meeting Page 11

  12. Electromagnetic compatibility Coming soon for 2.0 Undergoing NIST internal review 12/9-10/2009 TGDC Meeting Page 12

  13. One more thing Source code analyzer tool assessment guide and test suite Test labs use source code analyzers to help assess conformity to the VVSG Source code analyzers vary in quality This guide and test suite help to determine which are fit for purpose 12/9-10/2009 TGDC Meeting Page 13

  14. Possible futures Voluntary use by test labs and/or manufacturers One input to a broader effort to reduce variability in testing Starting point for independent project(s) …manufacturer consortium? …jurisdictional testing? …general open source community? EAC requirements on test plans … not for NIST to determine 12/9-10/2009 TGDC Meeting Page 14

More Related