Abstract Test Case Development Phil Beecher (BCC)

1. Abstract Test Cases Edge / Enterprise Conformity Abstract Test Case Development Phil Beecher (BCC)

2. Abstract Test Cases Edge / Enterprise Conformity Abstract Test Cases • References • Prerequisites • Define System Under Test • Production Process • Identify Interfaces • Define Functionality and plan test coverage • Define Tests

3. Abstract Test Cases Reference Documents • ITU Recommendation X.291 – OSI Conformance Testing Methodology and Framework for Protocol Recommendations for ITU-T Applications – Abstract Test Suite Specification • This recommendation is particularly aimed at developing Abstract Test Suites for testing communications protocols but has relevance for Products (Systems and Devices) which comprise an application and one or more layers of a communications protocol

4. Abstract Test Cases Prerequisites Identify System Under Test / Interface(s)

5. Abstract Test Cases Prerequisites • Define Functionality • Implementation Conformance Statement • Specific Requirements are defined in an Implementation Conformance Statement (ICS) proforma • Format is typically a questionnaire • Each function / feature of the specification(s) should be clearly described with reference to the specific clause(s) in the specification(s) • All features / functions should be clearly described as mandatory or optional, with dependencies described • An IUT should have an accompanying ICS: this describes which features / functions of the specification(s) are implemented

6. Abstract Test Cases Abstract Test Suite Production Process • Specifications and ICS proforma(s) • Derive conformance requirements which need testing • Determine test groups • Understand/document test groups required to achieve conformance requirements • Develop Test Purposes • Description of tests to provide adequate coverage of conformance requirements • Determine Testing Context and Method(s) • Requirements for Upper and Lower Tester and Test Coordination Procedures (see next slide)

7. Abstract Test Cases Test Methods Overview • Testing Context: Single or Multi Party testing? • Single Party Testing Context is used when IUT is required (by test purpose) to communicate with only one real other system. • Multi Party Testing Context is used when IUT is required to communicate with multiple other real open systems concurrently. • Testing Functions • LT: Lower Tester • LTCF: Lower Tester Control Function • UT: Upper Tester • TCP: Test Coordination Procedures

8. Abstract Test Cases Testing Functions • Single Party Testing Context requires: • LT behaves as the peer real open system to the IUT and assigns the verdict for the test case • UT behaves as a user of the IUT • TCP coordinates the LT and the UT • Multi Party Testing Context requires: • A set of LTs which execute in parallel, each LT behaves as a peer real open system to the IUT. • LTCF coordinates the activity of the LTs and UT (if any) and assigns a verdict for the test case • Optionally, a set of UTs which execute in parallel, each behaving as a user of the IUT • TCP between each associated LT and UT, among the LTs between the LTs and LTCF among the UTs and between the UTs and LTCF

9. Abstract Test Cases Test Methods for Single Party Testing • Local • Distributed • Coordinated • Remote

10. Abstract Test Cases Single Party Testing – Local Method • Test events at the LT specified only in terms of X-ASPs and/or P1 to Pn PDUs • Test events at UT PCO specified in terms of Y-ASPs • Upper service boundary of the IUT shall be a standardised (hardware) interface which can be used for testing purposes – test suites place no additional requirements on realisation of interface in the SUT • The specification of the (hardware) upper interface of the IUT defines mapping between relevant ASPs and/or PDUs and their realisation at the interface • UT is located within the test system • Requirements for the TCP are specified in the Test Suite but are realised in the test system

11. Abstract Test Cases Local Test Method • LT – Lower Tester • UT – Upper Tester • IUT – Implementation Under Test • TCP – Test Coordination Procedures • SUT – System Under Test • ASP – Abstract Service Primitives • PCO – Point of Control and Observation Test System PCO ASPs UT TCP P1 to Pn PDUs LT SUT IUT X-ASPs X-Service-Provider

12. Abstract Test Cases Single Party Testing – Distributed Method • The test events at the LT PCO are specified only in terms of X-ASPs and/or P1 to Pn-PDUs • The test events at the UT PCO are specified in terms of Y-ASPs • The upper service boundary of the IUT shall be either a human user interface or a standardized programming language interface which can be used for testing purposes; the test suites shall not place any requirements on the realization of the interface in the SUT, additional to those in the standardized programming language interface specification, if applicable • There shall be a mapping between the relevant ASPs and their realization at the upper interface of the IUT • The UT is located within the SUT • The requirements for the TCP shall be specified in the ATSs, although the procedures themselves shall not be • If the upper interface of the IUT is a human user interface, then the human operator of the SUT fulfils the requirements of the TCP • If the upper interface is a standardized programming language interface, then the UT is realized in software and the UT and LT together fulfil the requirements of the TCP.

13. Abstract Test Cases Distributed Test Method • LT – Lower Tester • UT – Upper Tester • IUT – Implementation Under Test • TCP – Test Coordination Procedures • SUT – System Under Test • ASP – Abstract Service Primitives • Note: Duplicate UT, TCP for multi-user single party testing. Test System SUT LT UT TCP Y-ASPs P1 to Pn PDUs IUT X-ASPs X-Service-Provider

14. Abstract Test Cases Single Party Testing – Coordinated Method • The test events at the LT PCO are specified in terms of X-ASPs, and/or (P1 to Pn)-PDUs plus Test Management PDUs (TM-PDUs); • Y-ASPs are not used in the specification of the ATS; no assumption is made about the existence of an upper service boundary of the IUT; • the UT is located within the SUT; • The requirements for the TCP shall be specified in the ATS by means of a standardized Test Management Protocol (TMP) , referenced by the ATS; • The UT shall be required by the test suite specifier to implement the TMP and achieve the appropriate effects on the IUT; • Test cases shall be added to the ATS for the purpose of testing that the UT conforms to the requirements of the TMP specification; such test cases do not contribute to the conformance assessment of the IUT.

15. Abstract Test Cases Coordinated Test Method • LT – Lower Tester • UT – Upper Tester • IUT – Implementation Under Test • TCP – Test Coordination Procedures • SUT – System Under Test • ASP – Abstract Service Primitives • PDU – Protocol Data Unit Test System SUT LT UT TM-PDUs IUT P1 to Pn PDUs X-ASPs X-Service-Provider

16. Abstract Test Cases Single Party Testing – Remote Method • In this test method, provision is made for the case where it is not possible to observe and control the upper service boundary of the IUT. Also in this test method: • The test events at the LT PCO are specified only in terms of X-ASPs, and/or (P1 to Pn)-PDUs • Y-ASPs are not used in the specification of the ATS; no assumption is made about the existence of an upper service boundary of the IUT • Some requirements for TCP may be implied or informally expressed in the ATS but no assumption shall be made regarding their feasibility or realization • Abstractly the SUT needs to carry out some UT functions to achieve whatever effects of the TCP and whatever control and/or observation of the IUT are implied or informally expressed in the ATS for the given base specification(s); these functions are not specified nor are any assumptions made regarding their feasibility or realization • The LT should attempt to achieve the implied or informally expressed TCP in accordance with the relevant information in the IXIT(s).

17. Abstract Test Cases Single Party Testing – Remote Method (2) • Additional notes: • In order to overcome the lack of specification of behaviour above the IUT, where necessary, the required behaviour of the SUT shall be specified in terms of the X-ASPs or (P1 to Pn)-PDUs which need to be observed by the LT. This form of implicit specification shall be taken to mean “do whatever is necessary within the SUT in order to provoke the required behaviour”. • It is possible that some of the test cases in the ATS cannot be executed (e.g. transmission of consecutive unacknowledged Data PDUs, etc.). • With such implicit specification of control of the IUT, in this test method, it is possible to specify control but not observation above the IUT. This is a major difference between this and the other test methods.

18. Abstract Test Cases Remote Test Method • LT – Lower Tester • UT – Upper Tester • IUT – Implementation Under Test • TCP – Test Coordination Procedures • SUT – System Under Test • ASP – Abstract Service Primitives Test System SUT LT UT TCP IUT P1 to Pn PDUs X-ASPs X-Service-Provider

19. Abstract Test Cases Multi Party Testing Context • Used when the IUT is required to communicate with multiple other real open systems. • Only one IUT is tested, but multiple lower testers are used to test it. • Each Lower tester represents one of the real open systems with which the IUT needs to communicate. • Each LT communicates with appropriate part of IUT, by observing and controlling ASPs and PDUs • LTs provide preliminary results but not assign verdict. • Lower Tester Control Function (LTCF) • coordinates activity of Lower Testers. • LTCF assigns verdict to test case • 0 or more Upper testers may be used.

20. Abstract Test Cases General Model for Multi-Party Testing Lower Tester Control Function TCPs LT3 UT3 LT2 UT2 LT1 UT1 PCO PCO PCO Y-ASPs IUT PCO (P) PDUs PCO PCO X-ASPs X-Service-Provider

21. Abstract Test Cases Test Methods Comparison • Add details comparing Test methods…. • Meanwhile, see part 11 of x291_04_95