1 / 33

Testing Techniques

Testing Techniques. Conformance Testing Methodology and Framework ISO IS-9646. Overview. Conformance Testing Methodology and Framework ISO IS-9646 Literature: “An Overview of OSI Conformance Testing”. Overview: This Lecture. Protocol conformance testing: why and what Standardization

beyla
Download Presentation

Testing Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testing Techniques Conformance Testing Methodologyand FrameworkISO IS-9646

  2. Overview • Conformance Testing Methodology and FrameworkISO IS-9646 Literature: “An Overview of OSI Conformance Testing” TT 2004, ISO Conformance 2

  3. Overview: This Lecture • Protocol conformance testing: why and what • Standardization • Overview Standard IS-9646:Conformance Testing Methodology and Framework • Conformance: static and dynamic • Options, protocol implementation conformance statement • Test generation: test purposes, abstract-, executable tests • Test methods, test architectures, • Test classification and -structuring • Test notation • Test implementation and -execution TT 2004, ISO Conformance 3

  4. Types of Testing TT 2004, ISO Conformance 4

  5. Protocol Conformance Testing distributed systems exchange of information rules for communication : protocol standardized protocol specifications conforming implementations testing for protocol conformance TT 2004, ISO Conformance 5

  6. Protocol Conformance Testing International Standard ISO IS-9646 :Conformance Testing Methodology and Framework • How to test protocol entity implementationsfor conformance with respect to their specifications • Originally for OSI systems • Now more generally applied • Defines test language TTCN • Standard accepted by ITU-T, ETSI, ATM-Forum, …… TT 2004, ISO Conformance 6

  7. Protocol Conformance Testing • Starting point :a correct and valid (standardized) protocol specification • Original goal :standardization of test suites in TTCN • consistency between specification and test suite ? • Conformance testing  interoperability testing TT 2004, ISO Conformance 7

  8. The Conference Protocol TT 2004, ISO Conformance 8

  9. The Conference Protocol Conference Service CPE CPE CPE UDP Layer join leave send receive TT 2004, ISO Conformance 9

  10. Conference ProtocolTest Architecture Tester TorX A B C UT-PCO = C-SAP CPE = IUT U-SAP LT-PCO LT-PCO UDP Layer TT 2004, ISO Conformance 10

  11. Protocol Conformance Testingvs. Interoperability Testing (N)-ASPs (N)-ASPs (N)-service provider (N)-Protocol Entity (N)-Protocol Entity (N)-PDUs protocol (N-1)-ASPs (N-1)-ASPs (N-1)-service provider TT 2004, ISO Conformance 11

  12. Protocol Conformance Testingvs. Interoperability Testing  interoperability • Protocol Conformance Testing: • correct protocol (design and validation) • conforming protocol entities (testing) • Interoperability testing • test directly whether each pair of protocol entities can work together (i.e. deliver (N)-service) • Internet: • interoperability testingwith (standard) reference entities TT 2004, ISO Conformance 12

  13. Standard Conformance Testing • Testing by: • manufacturers • users organizations • telecom operators • independent test laboratories • Standard IS-9646 for • certification of implementations • accreditation of testing processes and test laboratories • international acceptance of test results TT 2004, ISO Conformance 13

  14. ISO IS - 9646 • Part 1 : Introduction and general concepts • Part 2 : Abstract test suite specification • Part 3 : TTCNTree and Tabular Combined Notation • Part 4 : Execution of tests • Part 5 : Requirements on test laboratories and administrative conformance assessment process TT 2004, ISO Conformance 14

  15. process of conformance testing standard protocol specification test generation implementation process standard conformance test suite protocol implementation IUT test implementation executabletest suite test execution verdict TT 2004, ISO Conformance 15

  16. Conforming Implementation • Specifications in natural language • What is a conforming implementation ? • “ … exhibits conformance if it complies with the conformance requirements of the applicable specification standard …” • Satisfaction of conformance requirements • Conformance requirements : “the implementation shall do ….. “ “the implementation shall not do …..” • Complication : options TT 2004, ISO Conformance 16

  17. Conforming Implementation • (P)ICS -(Protocol) Implementation Conformance Statement : • list of implemented options • supplied by manufacturer • (P)ICS proforma • fill-in questionnaire with all possible options • attachment to protocol standard • to be completed by manufacturer to make PICS • Restrictions on combinations of implemented options • documented in standard in static conformance requirements TT 2004, ISO Conformance 17

  18. Example : Static Conformance Requirement • Feature A is optional • Feature B is optional • But A may only be implementedif B is also implemented TT 2004, ISO Conformance 18

  19. Conforming Implementation • Requirements on dynamic behaviour:dynamic conformance requirements • Messages : • structure • coding • Ordering : • sequence of sent and receive messages • relation between information content of different messages • Example: • When in state Enable, upon receiving a Data-Request,the implementation shall send a Data-PDUor it shall enter the abort phase. TT 2004, ISO Conformance 19

  20. Conforming Implementation • A conforming implementation is onewhich satisfies both static and dynamic conformance requirements, consistent withthe capabilities stated in the PICS • Conformance testing: • checking static conformance by reviewing • checking dynamic conformance requirements by testing TT 2004, ISO Conformance 20

  21. Test Generation dynamic conformance requirement test purpose(s) generic test case abstract test case executable test case test case execution TT 2004, ISO Conformance 21

  22. Test Generation • Test purpose :precise description (natural language)of what is to be tested in order to decide aboutsatisfaction of a particular conformance requirement • Example: • When in state Enable, upon receiving a Data-Request,the implementation shall send a Data-PDUor it shall enter the abort phase. • Check whether in state Enable, after receivinga Data-Request, a Data-PDU is sent. (but implementation may also enter the abort phase!) TT 2004, ISO Conformance 22

  23. Test Generation • Generic test case:operationalization of test purpose;description of actions to be performedto reach test purpose • Example: • Check whether in state Enable, after receivinga Data-Request, a Data-PDU is sent. • Go to state Enable, send a Data-Request,and observe a Data-PDU, and if not check that theimplementation starts entering the abort phase TT 2004, ISO Conformance 23

  24. Test Generation • Abstract test case:take test method into account • Test method ( test architecture, test environment ) :abstract description of environmentin which testing will be performed,in terms of PCO - Points of Control and Observation • Example:Do Connect-Request,check Connect-Response (now in state Enable),send Data-Requestobserve Data-PDUor observe Start-Abort TT 2004, ISO Conformance 24

  25. Test Methods UT PCO (N)-ASPs IUT test coordination (N)-PDUs PCO (N-1)-ASPs LT Local test method TT 2004, ISO Conformance 25

  26. Test Methods test system SUT LT UT test coordination PCO (N)-ASPs (N)-PDUs IUT PCO (N-1)-ASPs service provider Distributed test method TT 2004, ISO Conformance 26

  27. Test Notation • Abstract test cases are written in standardized notation :TTCN : Tree and Tabular Combined Notation • Specification language for Abstract Test Cases • Behaviour specified as tree of sequences of possible events • Data specified in tables or in ASN.1 • ISO IS-9646 Part 3 • With: input ? , output ! (from tester perspective!) , timers / time out, ordering of events, default behaviour, alternative behaviours, tree attachment (=subroutines), parallel component testers, verdicts: pass, fail, inconclusive TT 2004, ISO Conformance 27

  28. Example : TTCN Test Case Dynamic Behaviour Test Case Name: Conn_Estab Group: transport/connection Purpose: Check connection establishment with remote initiative behaviour constraints verdict +preamble LT ! T-PDU-connect-request UT ? T-SP-connect-indication UT ! T-SP-connect-response LT ? T-PDU-connect-confirm OTHERWISE LT ? T-PDU-disconnect-request OTHERWISE pass fail inconclusive fail TT 2004, ISO Conformance 28

  29. Classification of Tests • Basic interconnection tests • Capability tests • Behaviour tests • Conformance resolution tests TT 2004, ISO Conformance 29

  30. Structuring of Test Suites Test group Test group Test group Test case Test case Test case Test step Test step Test step Test event Test event Test event PCO ? X PCO ! A Test suite TT 2004, ISO Conformance 30

  31. Test Implementation • Transform abstract test suite into executable test suite by • compilation • interpretation • a test slave ….. • Select test cases from abstract test suite based on PICS • Incorporate (P)IXIT -(Protocol) Implementation eXtra Information for Testing : • address information • timer information • specific parameter values • ….. • PIXIT proforma supplied by test laboratory TT 2004, ISO Conformance 31

  32. Test Execution • Execute / interpret executable test suite • Collect and analyse the resultsbased on verdicts pass, fail, inconclusiveof each test case execution • Collect all results of • static conformance review • dynamic testing into Protocol Conformance Test Report TT 2004, ISO Conformance 32

  33. standard protocol specification implementation process protocol implementation IUT PICS proforma test purposes test selection static conformance review generic test suite analysis of results test execution executable test suite verdict test report test notation (standardized) abstract test suite Basic interconn. capability behaviour standardized test methods test implementation dynamic conformance req’s static conformance req’s PIXIT proforma certification PICS PIXIT TT 2004, ISO Conformance 33

More Related