Conformance

1. Conformance Lynne S. Rosenthal Mary Brady National Institute of Standards and Technology Lynne.rosenthal@nist.gov Mary.brady@nist.gov

2. Outline • Terminology • Testing Methodologies • Benefits of Conformance Testing • Test Suites for XML Technologies • XML, DOM, XSLT, • XSL FO, XML Schema Conformance clause Conformance Testing Test suites Test tools Certification Validation Certificates Branding Procedures, reporting

3. Unscrambling the Terminology Conformance Specification Conformance Clause Conformance Testing test suite, test tools Validation process, reporting Certification certificate, brand

4. Terminology - Conformance • CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2) • These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification • CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

5. Terminology - Conformance Testing • CONFORMANCE TESTING – a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites) • tests capture the technical requirements within the specification • testing is bound in scope by the specification • internal structure of product-under-test not accessible (black box testing) • Necessary, but not sufficient, for interoperability Note: referred to as conformity assessment in ISO Guide 2

6. Input Output Testing Methodology • Falsification Testing • find errors by means of experimentation • show presence of errors not their absence • prove non-conformance; can never prove conformance • Tests Based on: • required functionality • logical errors by misunderstanding requirements • errors from boundary conditions and divergence • common programming errors

7. Specification Requirements Implementation Conformance Tests Test programs

8. Terminology - Testing Process • VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite • ensures that testing can be repeatable and reproducible • ensures that conclusions are consistent with facts presented in the evaluation • CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

9. Testing Ingredients • Certification • qualified bodies to do the testing and certification • control board - advisory and arbiter Validation Process - policy and procedures for testing • Conformance Testing • Test assertions • Test suite • (test software, test scripts, test criteria) • Standard • Conformance clause

10. Benefits of Conformance Testing • Increased probability that products are implemented correctly • Contains required functionality • Behaves as expected • Performs functions in a known manner • Increased likelihood of portability and interoperability • Portability - move software or applications among different systems • Interoperability – ability of 2+ systems to exchange and use information

11. Benefits of Conformance Testing cont. • For Developers of Specifications • Feedback loop • For Software Developers • Help to improve quality - identify areas in which they conform or not conform • For Buyers • Increases buyer’s confidence in a product and its ability to meet their needs • Substantiate seller’s claims • Not locked into purchasing from a single vendor

12. Test Harness Test Cases Test Scenario Test Assertions, Axioms, Semantic Requirements Specification Components of a Test Suite

13. OASIS XML Conformance • XML Conformance Committee • Develop tests to improve the quality of XML processors • Provide a forum to discuss issues related to interoperability • Provide input to W3C regarding errors/ambiguities • Deliverables • Test Suite containing over 2000 tests, Second Edition • Associated test report • Built using XML/XSLT

14. XML Test Suite • Types of XML Tests • Binary • Validating processors must check validity constraints. • Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read. • Output Tests • Requirements on XML processors to ensure that they pass the necessary information on to applications.

15. XML Test Matrix

16. XML Test Suite • Test Report • XSL Stylesheets • IE5, HTML renditions Testcases.dtd Conformance.xml • Test Profile • Test URI • Unique ID • Spec Section • Type • Entities • Description • NIST / OASIS • SUN Microsystems • James Clark • FUJI Xerox • IBM • XML.COM • Java application • Tests XML processors • Dave Brownell Test Case DTD XML Test Description Files Outputs <TESTCASES PROFILE=“SUN Microsystems XML Tests”> <TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”> Tests the Element Valid VC by including an undeclared child element.</TEST> </TESTCASES>

17. XML Test Suite Results, v1 • XML.COM Conformance Update – Dave Brownell • Validating Parsers • Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V2.0.2.7 • Pass Rate 82% - 99% • Non-validating Parsers • Tested all of the above plus XP 0.5 • Pass rate 86% - 99% • Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett • Non-validating: DOM: 95%, SAX: 97% • Validating: 97%

18. Test Harness – automatically runs available tests and presents user with a color-coded table of results; gives access to spec, assertions, and source code Test Scenario Load appropriate XML or HTML file, exercise DOM calls Test Assertions – Fundamental, Extended, HTML Interfaces DOM L1 Specification NIST DOM Test Suite Test Cases 200 Java – fundamental, extended 800 ECMAScript fundamental, extended, HTML

19. DOM Test Suite • Test Results: • Run tests against IE5, Mozilla (some fundamental interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98% • Implementations differ greatly in the way that parsers handle white-space and entity expansion issues – differences in DOM tree • DOM Level 2 • Working with W3C to identify resources • Modified Java tests to be consistent with SUN contribution • Will update NIST tests to L2 status

20. XSLT Test Suite • OASIS Technical Committee, chaired by G. Ken Holman • Member contributions include Lotus, NIST, Sun • Expands on XML Test Description file • Dave Marston, XSLT/XPATH Conformance Wed, 4pm, Delaware Room • Public Information Meeting, Wed, 8pm, Suite 8228

21. XSL FO Test Suite • W3C WG to test Candidate Recommendation • Develop test assertions for all basic formatting objects and properties • Develop test DTD and associated harness • Work with vendors to categorize available tests • Research translating available CSS and XSLT tests • Develop tests for additional spec components • Expected March 2001

22. XML Schema Test Suite • Test Control • Test Types • # schemas • # instances Test Generator Schema Tests • Constraints • Structure • Limits • Special info

23. Summary • Interesting URL’s • http://www.nist.gov/xml/ • http://www.w3.org/ • http://www.oasis-open.org/ • Questions???