Testing and Validating OAGi NDRs

1. National Institute of Standards and Technology 1 Testing and Validating OAGi NDRs Puja Goyal Salifou Sidi Presented to OAGi April 30th, 2008 Talking about work we�ve done with the testing and validating OAGi NDRsTalking about work we�ve done with the testing and validating OAGi NDRs

2. National Institute of Standards and Technology 2 Outline Naming and Design Rules Quality of Design Tool (QOD) OAGi NDR v9.0 Encoding Testability Observations Demo NDR Authoring/Sharing What�s Next help the industry achieve interoperability.help the industry achieve interoperability.

3. National Institute of Standards and Technology 3 NDR: Naming and Design Rules Rules applied when creating standards for the exchange of information Any of a series of rules documents since 2003 that restrict XML Schema flexibility Indicate features which should, may, may not, should not, must, or must not be used. Brief overview of what naming and design rules are.Brief overview of what naming and design rules are.

4. National Institute of Standards and Technology 4 What�s in an NDR? Naming Rules: Names (or tags) represent information to be exchanged Subject names to consistent naming practices for clarity and consistency in an XML schema. Design Rules Address how information is structured Superstructures: namespaces Microstructures: global vs. local elements Our focus is on the design rulesOur focus is on the design rules

5. National Institute of Standards and Technology 5 Benefits of Encoding Rules NDRs written in English Easy to misinterpret Schema testing and validation not automated Encoded NDRs Result in better rules Verifies rule is clear Validates rule is being used as intended Can be computationally enforced Cycle time from requirements for a schema to production XML is reduced Problems in written rules are flushed out as they are developed � result in better rules Problems in written rules are flushed out as they are developed � result in better rules

6. National Institute of Standards and Technology 6 QOD � Apply rules directly Interface to write Test Requirements (rules) and associate with Test Cases (encoded rules) Schematron Java-Expert System Shell (JESS) Automate testing and validation against schema(s) Displays and stores results from execution against Schematron/JESS Repository to store rules Easy to reuse Group rules to create Test Profile ** explain QOD terminology and how it maps to NDR terms** explain QOD terminology and how it maps to NDR terms

7. 7 QOD � Testing Environment

8. National Institute of Standards and Technology 8 QOD Tool Overview Test Requirements Page Test Case Page Test Profile Page * Results Page (Updated) * Import an NDR profile (New) * Export a QOD test profile (New)

9. National Institute of Standards and Technology 9 OAGi NDR v9.0 Encoding Rule Testability Additional observations Demo I will talk about the lesson that we learned while encoding the OAGi NDR. I will start first to talk about the potential problems that we found in the NDR Second I will talk about the rules testability and the reason why the rules are not testable Then I will talk about the overlaps and links between rules And finally the classification of OAGi rule within the QOD tool and I will make a demo. I will talk about the lesson that we learned while encoding the OAGi NDR. I will start first to talk about the potential problems that we found in the NDR Second I will talk about the rules testability and the reason why the rules are not testable Then I will talk about the overlaps and links between rules And finally the classification of OAGi rule within the QOD tool and I will make a demo.

10. National Institute of Standards and Technology 10 Rule Testability Fully Testable Rule can be encoded without any assumptions Partially Testable Some assumptions are needed Not Testable unable to encode the rule In the process of NDRs encoding, we classify the rules in 3 type categories : � fully testable, partially testable, and not testable. �fully testable� means that the test case associated with the rule can capture/represent the assertion stated in the rule without any assumption, i.e., the test case can always identify non-conformance to the assertion; �partially testable� means that the test case associated with the rule cannot totally capture/represent the assertion stated in the rule or that assumptions have to be made in order to capture the assertion, i.e., the test case, in some circumstances, cannot identify non-conformance to the assertion; �not testable� means that a script cannot be written to capture/represent the assertion stated in the rule. In the process of NDRs encoding, we classify the rules in 3 type categories : � fully testable, partially testable, and not testable. �fully testable� means that the test case associated with the rule can capture/represent the assertion stated in the rule without any assumption, i.e., the test case can always identify non-conformance to the assertion; �partially testable� means that the test case associated with the rule cannot totally capture/represent the assertion stated in the rule or that assumptions have to be made in order to capture the assertion, i.e., the test case, in some circumstances, cannot identify non-conformance to the assertion; �not testable� means that a script cannot be written to capture/represent the assertion stated in the rule.

11. National Institute of Standards and Technology 11 Testability of OAGi Rules This diagram shows the breakdown of OAGi NDR v 9 rules. 102 based on their testability The NDR has in total 102 rules among those rules 51 are fully testable, 12 partially testable and32 are not testable There is 7 rules without assertions in order word a rule ID has been affected but no rule assertion or wording is providedThis diagram shows the breakdown of OAGi NDR v 9 rules. 102 based on their testability The NDR has in total 102 rules among those rules 51 are fully testable, 12 partially testable and32 are not testable There is 7 rules without assertions in order word a rule ID has been affected but no rule assertion or wording is provided

12. National Institute of Standards and Technology 12 Not Testable Rules � From Our Perspective QOD Limitations Related to XML instance Lack of clarity Nondeterministic Lack of information in the OAGi schemas Documentation or Guidance Supplementary information Lack of clarity � need more information to ensure correctness of encoding (out of context assumption has been made)Lack of clarity � need more information to ensure correctness of encoding (out of context assumption has been made)

13. National Institute of Standards and Technology 13 Examples: OAGi-76, OAGi-70 - Ambiguous: OAGi-76 : �xsd:notation MUST NOT be used.� Which notation must not be used? xsd:notation element or xsd:NOTATION type ? - Nondeterministic: OAGi-70 �Minor version schema MUST incorporate all XML constructs from the immediately preceding major or minor version schema� - No minor version information in OAGi schemas Doesn�t precise which notation element or type For example : OR-76 is an example of rule which is not clear, it says that the xsd:notation must not be used but it doesn�t precise exactly what shouldn�t be use the xsd:notation element or the xsd:NOTATION type. OR-79 is an example of erroneous rule it says that xsd:any attribute must not be used but the problem is that there is no such attribute in the XML schema spec. _______________________________________________________________________________________________________________________________________ For the rule OR-79 it seems erroneous because there is no xsd:any attribute in the XML Schema spec.Doesn�t precise which notation element or type For example : OR-76 is an example of rule which is not clear, it says that the xsd:notation must not be used but it doesn�t precise exactly what shouldn�t be use the xsd:notation element or the xsd:NOTATION type. OR-79 is an example of erroneous rule it says that xsd:any attribute must not be used but the problem is that there is no such attribute in the XML schema spec. _______________________________________________________________________________________________________________________________________ For the rule OR-79 it seems erroneous because there is no xsd:any attribute in the XML Schema spec.

14. National Institute of Standards and Technology 14 Additional Observations Overlaps �Fully Overlapped� Generalization vs. Restriction Generic rule may be removed for same results �Partially Overlapped� Occasionally can combine rules if same type of schema Links (one rule refers to another) For testing purposes, all referenced rules have to be included The overlaps between rules occur when one rule is covered (partially or totally) by another rule. This usually happens when one rule is a generalization or a restriction of another rule. Identifying the overlaps between rules can simplify the NDR thus improve its readability and quality. The overlaps between rules occur when one rule is covered (partially or totally) by another rule. This usually happens when one rule is a generalization or a restriction of another rule. Identifying the overlaps between rules can simplify the NDR thus improve its readability and quality.

15. National Institute of Standards and Technology 15 Overlaps between rules Example : OAGi-11 �XML element, attribute and type names constructed from dictionary entry names MUST NOT include periods, spaces, or other separators; or characters not allowed by W3C XML 1.0 for XML names.� OAGi-10 �Element, attribute and type names MUST be drawn from the following set: a � z and A � Z.� OAGi-10 is restriction of OAGi-11 OAGi-11 can be deleted from the NDR without affecting the original intention

16. National Institute of Standards and Technology 16 OAGi Profiles in QOD General Profile Rules that can be applied to all OAGi schemas BOD Profile Rules specific to BOD schemas Categorized rules into 3 profiles. IN OAGi schemas, don�t know what is ABIE, BBIE, and ASBIE, so didn�t encode for these. Not testable. Treated as non deterministic rule. ABIE, BBIE, ASBIE � come UNCEFACT It is important classify rule into profile since they may apply to different schema type. For example the rule about the BODs can only apply to a BOD schema.Categorized rules into 3 profiles. IN OAGi schemas, don�t know what is ABIE, BBIE, and ASBIE, so didn�t encode for these. Not testable. Treated as non deterministic rule. ABIE, BBIE, ASBIE � come UNCEFACT It is important classify rule into profile since they may apply to different schema type. For example the rule about the BODs can only apply to a BOD schema.

17. National Institute of Standards and Technology 17 Demo

18. National Institute of Standards and Technology 18 OAGi Developer BOD Testing : Materials Schemas tested - Developer BOD schemas version 9.0 : 434 files - Developer BOD schemas version 9.1 : 434 files - Developer BOD schemas version 9.2 : 494 files Profiles tested - General Profile : 25 rules - BOD Profile : 5 rules

19. National Institute of Standards and Technology 19

20. National Institute of Standards and Technology 20 NDR Sharing Environment Facilitate reuse of rules Different groups have different requirements but some overlap Group relevant tests to create customized test profile Upload new tests to share with others Now we�ll move over to a high level idea of a sharing and authoring environment I think that the points to make are 1) that the use of an XML Schema--the NDRProfile-- is a key technical component that enables the authoring and sharing pieces, and 2) that we have developed some tools around it to facilitate both authoring and rendering of "test suites" or "test profiles". The question for discussion is whether people see a use for creating their own NDRs presumably based on the OAGi NDR. First they need to understand the idea of how that would work before they can really discuss it, otherwise they may get lost in the discussion of whether it's feasible or worth the effort. We've made the effort-- now is it something they can use so we are speculating that OAGi members may want to do this:- create their own test suites tailored from the OAGi NDR,- publish those test suites in multiple forms,- supplement the documentation of the tests Now we�ll move over to a high level idea of a sharing and authoring environment I think that the points to make are 1) that the use of an XML Schema--the NDRProfile-- is a key technical component that enables the authoring and sharing pieces, and 2) that we have developed some tools around it to facilitate both authoring and rendering of "test suites" or "test profiles". The question for discussion is whether people see a use for creating their own NDRs presumably based on the OAGi NDR. First they need to understand the idea of how that would work before they can really discuss it, otherwise they may get lost in the discussion of whether it's feasible or worth the effort. We've made the effort-- now is it something they can use

21. National Institute of Standards and Technology 21 NDR Authoring Environment Capability to export/import rules Edit rules offline Display in multiple ways HTML, XML, PDF, Spreadsheet, DocBook, etc Customize NDR to fit your needs Based on NDR Profile Schema Standard structured vocabulary for describing NDR rules Available: http://ecc02.eccnet.com/ndr/

22. National Institute of Standards and Technology 22 QOD � Authoring and Sharing

23. National Institute of Standards and Technology 23 OAGi NDR Sharing Based on UN/CEFACT 26 totally reusable from UN/CEFACT 59 modifications of UN/CEFACT rules and can be partially reuse 17 rules are unique

24. National Institute of Standards and Technology 24 NDR Profile Schema Purpose of NDR Schema Standard set of rules for describing NDR rules Use of open standards when developing NDRs Standard structured vocabulary for describing NDR rules Benefits of NDR Schema Enables exchange of NDR profiles between parties. Facilitates import/export of NDR rules with QOD. Can be developed using standard-based tools Is machine (computer) readable Multiple presentation methods for human visualization of rules HTML/PDF/Excel, etc. Puja � on this slide I want to describe the purpose and benefits of the NDR schema. I don�t think the NDR schema is described earlier in your presentation. Puja � on this slide I want to describe the purpose and benefits of the NDR schema. I don�t think the NDR schema is described earlier in your presentation.

25. National Institute of Standards and Technology 25 NDR Profile Authoring Ways to create NDR Profile XML Exchange Documents conforming to the NDR Profile Schema Export from QOD XForms creation Excel Spreadsheet Template XSLT transform from Excel to NDR Profile Any Validating XML Editor Xforms creation Client Side � Mozilla XForms Server Side � Orbeon XForms Server (prototype) xml editor - XMetal, Arbortext Editor, XML Spy, Oxygen, Syntext, etc. CSS and XSLT stylesheets have been developed that can be used by many XML Editor to provide Word-like interface in many tools. Xforms creation Client Side � Mozilla XForms Server Side � Orbeon XForms Server (prototype) xml editor - XMetal, Arbortext Editor, XML Spy, Oxygen, Syntext, etc. CSS and XSLT stylesheets have been developed that can be used by many XML Editor to provide Word-like interface in many tools.

26. National Institute of Standards and Technology 26 What�s next Create stand alone system Enhance support for rule sharing by including rule traceability Pursue similar support for developing tests for constraints on instance data Incorporate feedback into our tools! Sophisticated language to better encode business rules for instance data.Sophisticated language to better encode business rules for instance data.

27. National Institute of Standards and Technology 27 Discussion Points Authoring/Sharing Environment Do OAGi members see it useful for their needs? Instance testing Stand alone vs. web based system

28. National Institute of Standards and Technology 28 Contacts KC Morris , NIST � kcm@nist.gov Simon Frechette, NIST � simon.frechette@nist.gov Josh Lubell, NIST � lubell@cme.nist.gov Puja Goyal, NIST �puja.goyal@nist.gov Salifou Sidi, NIST � salifou.sidi@nist.gov Betty Harvey, ECC, Inc. -- harvey@eccnet.com Websites QOD online http://syseng.nist.gov/b2bTestbed/projects/QOD/ Developer�s site for QOD tool and executable rules: http://qod.sourceforge.net/ XML Tools available from MSID: http://www.mel.nist.gov/msid/xml_related.htm DAS XSI Working Group: https://www.collab.core.gov/CommunityBrowser.aspx?id=2234

29. National Institute of Standards and Technology 29 Welcome page

30. National Institute of Standards and Technology 30

31. National Institute of Standards and Technology 31

32. National Institute of Standards and Technology 32

33. National Institute of Standards and Technology 33 TP results

34. National Institute of Standards and Technology 34 OAGi 31 results with schema

35. National Institute of Standards and Technology 35 NDR Authoring Environment Based on XML Schema � NDR Profile User�s Guide Prototype Authoring and Rendering tools Display NDR Document Edit NDR Profile The Bridge to Testing: QOD Import/Export

36. National Institute of Standards and Technology 36 Mark-Up of Guidance <Guidance guidanceID="OR-90"> � <GuidanceName>Check the usage of empty elements</GuidanceName> � <Testability>Fully-Testable</Testability> � <GuidanceText>Empty elements MUST NOT be used.</GuidanceText> <TestCases ruleType="Schematron"> <TestCase> <CodeExample> <![CDATA[ <?xml version="1.0" encoding="UTF-8"?> <sch:schema xmlns:sch="http://www.ascc.net/xml/schematron"> <sch:ns uri="http://www.w3.org/2001/XMLSchema" prefix="xsd"/> <sch:pattern name="Check the usage of empty elements"> <sch:rule context="xsd:element"> <sch:report test="count(@type)=0 and count(@ref)=0 and count(.//xsd:element) =0"> [OAGi 90] Error : Empty elements MUST NOT be used. </sch:report> </sch:rule> <sch:rule context="xsd:complexType"> <sch:report test="count(@name)!=0 and count(.//xsd:element) =0"> [OAGi 90] Error : Empty elements MUST NOT be used. </sch:report> </sch:rule> </sch:pattern> </sch:schema>� ]]> � </CodeExample> � </TestCase> � </TestCases> � </Guidance>

37. National Institute of Standards and Technology 37 Demonstration of Editing

38. National Institute of Standards and Technology 38 Editing (cont.)

39. National Institute of Standards and Technology 39 Demonstration of Rendering: NDR in PDF

40. National Institute of Standards and Technology 40 NDR in HTML with links to Profile Data and QOD

41. National Institute of Standards and Technology 41 Profile Depicted as HTML