1 / 19

TTCN CONFORMANCE TEST SUITE TTCN-3 Part 1 & Part 9 STF 4 8 7 MILESTONE 2

This document provides an overview of the results, technical details, and proposed methods of the TTCN-3 Conformance Test Suite for Part 1 and Part 9. It covers new and changed features, upgrades to test execution tools and validation of tests using TTCN-3 compilers.

sherryn
Download Presentation

TTCN CONFORMANCE TEST SUITE TTCN-3 Part 1 & Part 9 STF 4 8 7 MILESTONE 2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 1 TTCN CONFORMANCE TEST SUITE TTCN-3 Part 1 & Part 9STF 487 MILESTONE 2 Bogdan Stanca-Kaposta/ Testing Technologies Tomas Urban / Elvior Andras Kovacs / Broadbit

  2. Contents • Introduction • Results • Technical details • Proposed Method • ATS

  3. Objectives • Detect changes of the TTCN-3 Standard • Part 1 4.7.1 • Part 9 (XML Schema) 4.6.1 • Cover new/changed Part 1 and Part 9 Features • Upgrade test execution tools • pre-processing, test adaptation • Validation of tests with at least two TTCN-3 Compilers • Tool vendors feedback (Elvior, Testing Technologies) • Raising of CRs • Document and publish the results • ICS, TSS/TP, IXIT, ATS, Progress Reports

  4. Objectives • Detect changes of the TTCN-3 Standard (done) • Part 1 4.7.1 • Part 9 (XML Schema) 4.6.1 • Cover new/changed Part 1 and Part 9 Features (wip) • Upgrade test execution tools (nearly done) • pre-processing, test adaptation • Validation of tests with at least two TTCN-3 Compilers • Tool vendors feedback (Elvior, Testing Technologies) (wip) • Raising of CRs (open) • Document and publish the results (draft done) • ICS, TSS/TP, IXIT, ATS, Progress Reports

  5. Results • Part 1 4.7.1 (184 new tests, 15 modified) • 84 (25%) changed sections (±36% done) • Enhanced converage of older sections • Part 9 (XML Schema) 4.6.1 • 11 (8%) changed sections (±10% done) • Administrative work • Updated SVN structure to ETSI standards • Compacted ATS folder structure to match windows 260 chars path limitations • Execution tools updated • Documents & reporting • Effort • 36 days used out of 100 by end of May 2015 • Updating existing tests to new requirements is the most of the work

  6. Validation • 4 teams • Team members: Testing Technologies, Elvior (wip) • External: IBM, Devoteam • Issues • Revalidating sections we usually find untested requirements, tests exists but not all details are tested • 1 bug in XmlDiff –1 acknowledged • Caused by incorrect handling of xsi:type attribute in Xerces XML processor – bug report filed, not fixed yet • Triggers on 2 TCs for union type • Hampered 2 vendors

  7. TECHNICAL DETAILS

  8. Part 1: Proposed Method • ATS = ∑ test cases • Test case = TTCN3 script • SUT = TTCN-3 tool • SUT executes TTCN3 script and • Produces some output: verdict, log • Output is validated against expected • SUT launch, execution of tests, collecting outputs is vendor specific.

  9. Part 1: Proposed Method TTCN-3 ATS Test case _ _ _ _ _ _ _ _ _ _ _ TTCN-3 Tool (SUT) Expected output _ _ _ _ _ _ _ _ _ _ _ _ Invalid TTCN-3 Execution results Execution error Evaluation PASS FAIL

  10. Overview: using XML Schema with TTCN3 • Direct import of definitions of types, elements, attributes as TTCN3 types from XML Schema Definition (XSD) file • Automagic transformation • Inbound XML message -> TTCN3 template • Sending TTCN3 template -> Outbound XML message • ES 201 873-9 “TTCN-3: Using XML schema with TTCN-3” references XSD imports TTCN-3 uses XML ATS Test case produces

  11. Part 9: Proposed Method • ATS = ∑ test cases • Test case = TTCN3 script + XSD to import + reference XML • Reference XML to validate encoding result • SUT = TTCN-3 tool + XSD import facility + XML codec • SUT executes TTCN3 script and • Produces some output: verdict, log • Output is validated against expected • SUT launch, execution of tests, collecting outputs is vendor specific.

  12. Part 9: Proposed Method references XSD imports TTCN-3 uses XML ATS Test case _ _ _ _ _ _ _ _ _ _ _ TTCN-3 Tool (SUT) Expected output _ _ _ _ _ _ _ _ _ _ _ _ Invalid XSD,TTCN-3 Execution results Execution error Evaluation PASS FAIL

  13. Part 1 & 9: Proposed method • Expected SUT output is encoded in the header comment of TTCN-3 script: • E. g.: @verdict pass accept, ttcn3verdict:pass • Same as TTCN-3 Conformance Test Suite • Reused tools to execute SUT & validate outputs • validation framework from TTCN-3 Conformance TS

  14. Test case construction If matches test case PASS, otherwise FAIL Types • Positive test: valid TTCN-3, valid XSD, must compile & execute with verdict PASS • Import definitions from XSD • Construct a template; encode template into string • Compare string with the reference XML document • Decode string • Check whether decoded value matches original template XSD TTCN-3 Template XML Data Decoded Value references encode decode XML Compare XML references

  15. Test case construction XSD TTCN-3 Template Test Case FAIL references encode • Negative test: valid TTCN-3, valid XSD, must compile & execute with verdict ERROR • Import definitions from XSD • Construct a template; • encode template into string • Encoding must fail -> TTCN-3 verdict error • If encoding succeeds, it is an SUT error Types Invalid XSD and TTCN-3 Template not compatible Test Case is PASS

  16. Sample TTCN-3 Script module Pos_060101_length_001 { import from schema_Pos_060101_length_001 language "XSD" all; /* target NS: “schema:Pos_060101_length_001”*/ template E1 m_msg := "length_010"; /* Aux. definitions */ testcase TC_Pos_060101_length_001() runs on C system C { /* loopback: sends back string with encoded XML*/ map(self:p, system:p); p.send(m_msg); /* encode */ alt { /* The actual matching happens here */ } } }

  17. Sample TTCN-3 Script: alt alt { // compare the encoded message with the XML file [] p.check(receive(universal charstring:?) -> value v_rcv) { if (matchFile(v_rcv, "Pos_060101_length_001.xml", { "Pos_060101_length_001.xsd" }, v_matchError) { alt { // match decoded value to pass test [] p.receive(m_msg) { setverdict(pass, "Decoded value matches"); } [] p.receive { setverdict(fail, "XML decoding failure"); } } } else { setverdict(fail, v_matchError); } } [] p.receive { setverdict(fail, “Failed to decode XML"); } }

  18. Sample XSD <?xml version="1.0" encoding="UTF-8"?> <schema xmlns=http://www.w3.org/2001/XMLSchema targetNamespace="schema:Pos_050101_namespaces_001“ xmlns:ns="schema:Pos_050101_namespaces_001"> <element name="MyType" type="integer"/> </schema>

  19. XML Comparison • Two XML files are considered equal if: • Tree of element nodes are equal, and qualified names of elements are the same for corresponding nodes. • NOTE: take care about for unordered subtrees corresponding to choice • Set of attributes for each element node are equal. Qualified names of attributes must match. • Literal values of attributes must match. • float type are compared by their numerical values. • Textual contents of elements must match, whitespaces are ignored. CDATA is considered equal to plain text with equal contents. • float type are compared by their numerical values. • Comments and processing instructions are ignored. • Invoked as external function • Reference implementation is based on XmlUnit toolkit, written in Java.

More Related