1 / 21

Conformance

Conformance. Mark Skall Lynne S. Rosenthal National Institute of Standards and Technology mark.skall@nist.gov lynne.rosenthal@nist.gov. Introduction . NIST works with industry to develop standards and tests to improve the quality of software and achieve interoperable solutions

keira
Download Presentation

Conformance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Conformance Mark Skall Lynne S. Rosenthal National Institute of Standards and Technology mark.skall@nist.gov lynne.rosenthal@nist.gov

  2. Introduction • NIST works with industry to develop standards and tests to improve the quality of software and achieve interoperable solutions • Many years experience with • Formal standards organizations and Consortia • Developing conformance test suites, tools, reference implementations • Developing validation and certification testing programs

  3. Today’s Objectives • Present basic information about conformance and conformance related topics • Achieve a common understanding • Lead a discussion about how conformance applies to ebXML • Present overview of TA Spec. Conformance Clause • Offer our assistance to WG in discussing and developing conformance for their Spec.

  4. Conformance clause Conformance Testing Test suites Test tools Certification Certificates Branding Validation Procedures, reporting Outline • Background • Terminology • Conformance Testing • Testing Methodology • Testing Ingredients • ebXML Issues • TA Conformance Guidance

  5. Background • Standards not enough to ensure interoperability • Standards only meaningful if implemented in a consistent way • Need to ensure that implementations adhere to the standard • What is expected of implementations in order to claim conformance – i.e., what are the requirements? • How will we know if an implementation conforms? • test suites, test tools • Different ideas of what conformance is • Past experience may have affected view of conformance

  6. Terminology Conformance Specification Conformance Clause Conformance Testing test suite, test tools Validation process, reporting Certification certificate, brand

  7. Terminology - Conformance • CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2) • These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification • CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

  8. Terminology - Conformance Testing • CONFORMANCE TESTING – a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites). • tests capture the technical requirements within the specification • testing is bound in scope by the specification • internal structure of product-under-test not accessible (black box testing) • Necessary, but not sufficient, for interoperability Note: referred to as conformity assessment in ISO Guide 2

  9. Testing Methodology • Falsification Testing • find errors by means of experimentation • show presence of errors not their absence • prove non-conformance; can never prove conformance • Tests built to check for: • required functionality has been implemented (basic tests) • logical errors by misunderstanding requirements • errors from boundary conditions and divergence • common programming errors

  10. Specification Requirements Implementation Conformance Tests Test programs ???? conforming 100% non conforming

  11. Terminology - Testing Process • VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite • ensures that testing can be repeatable and reproducible • ensures that conclusions are consistent with facts presented in the evaluation • CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

  12. Testing Ingredients • Certification • qualified bodies to do the testing and certification • control board - advisory and arbiter Validation Process - policy and procedures for testing • Conformance Testing • Test assertions • Test suite • (test software, test scripts, test criteria) • Standard • Conformance clause

  13. ebXML Conformance Issues • Conformance Clause should address? 1. What Needs to Conform? 2. Profiles and Levels 3. Extensions 4. Implementation Defined Features 5. Alternate approaches • Scope –what should be addressed? a. Assertions b. Test Suite c. Testing Service d. Certification

  14. ebXML Issues - Conformance Clause address: 1. What needs to conform • identify the ‘class of products’ that will be developed • what are the conditions that need to be met in order to claim conformance • Examples: registry, messaging service, business service interface, ebXML message, testing partner agreements

  15. ebXML Issues -Conformance Clause address: 2. Profiles and Levels • Are profiles and/or levels needed? • Should a profile have its own levels? • minimal requirements (core) Profile is a subset of the overall specifications that includes all of the functionality necessary to satisfy the requirements of a particular community of users. Levels are nested subsets of the specifications. Level 1 = core that all must implement; subsequent levels include lower level + added functionality

  16. ebXML Issues - Conformance Clause address: 3. Extensions • If allowed: • shall not re-define semantics of existing functions • shall not cause standard-conforming functions to execute incorrectly • Standardized method for extensions? • Define the mechanism for extending the specification, so extended in a standard manner • Register of extensions • document parallel to but distinct from specification • list of recognized, published extensions • may eventually migrate into future versions

  17. ebXML Issues - Conformance Clause address: 4. Implementation Defined Features • may want to address implementation dependent ranges, min/max allowed values 5. Alternative Approaches • specification may describe several different ways to accomplish an operation (e.g., choice of bindings) • specify whether or not all approaches need to be implemented

  18. ebXML Issues - Scope Should ebXML include or address: a. Assertions (test requirements) - Should assertions be part of the standard? b. Test Suite - Prescribe a test methodology? - How test suite will be delivered/used (e.g., web based, downloadable)? c. Testing Service - Guidelines for establishing a test service d. Certification - Guidelines for issuing certificates

  19. Testing Program - Considerations • How formal should testing be? • Self testing, 3rd party testing • Branding/certificates • Testing program • Who will ‘own’ the testing program • Policy and procedures • Testing laboratory • Issue Certificates • Control board • Test suite maintenance

  20. TA Conformance Guidance • General conformance guidelines for ebXML • All ebXML specs shall contain a conformance clause • Can have levels of ebXML conformance - hierarchical • Explains how conformance requirements can be expressed (mandatory, conditional, optional) • Encourage use of publicly available test suites

  21. Discussion

More Related