1 / 29

AMHS Conformance Testing

AMHS Conformance Testing. AMHS Implementation Workshop Chennai, India, 15 – 16 December 2008 Rainer Hoffmann, AC-B GmbH. Presentation Contents. EUR AMHS testing strategy AMHS conformance test environment Test experience. EUR AMHS testing strategy AMHS conformance test environment

tovi
Download Presentation

AMHS Conformance Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AMHS Conformance Testing AMHS Implementation Workshop Chennai, India, 15 – 16 December 2008 Rainer Hoffmann, AC-B GmbH

  2. Presentation Contents • EUR AMHS testing strategy • AMHS conformance test environment • Test experience

  3. EUR AMHS testing strategy • AMHS conformance test environment • Test experience

  4. EUR AMHS Testing Strategy • ICAO EANPG (AFSG) developed standards for AMHS testing • What exactly will be tested? • Which tests are performed? • How are tests executed (under which conditions)? • -> Make tests fair and transparent to everybody • 3 Test phases are defined in the EUR AMHS Manual • Conformance testing • Interoperability testing • Pre-operational testing

  5. Test phases (1) • Conformance tests • wide test scope of standard (and optional) functions • soon after system installation(may be done in parallel with FAT/SAT) • Interoperability tests • limited test scope - basic and important functions • bilateral and trilateral tests • when the network connection is established • Pre-operational tests • a few check-ups close before cut-over • run AFTN and AMHS link in parallel

  6. examine one system against the specification (standard documents) can be performed locally examine two or more systems communicating with each other distributed over two or more locations IUT System A IUT System B IUT System A Test Tool IUT System C Test phases (2) Conformancetesting  Interoperabilitytesting IUT:= Implementation Under Test • test automation possible • includes „negative“ test cases • bilateral coordination of staff presence • only „positive“ test cases

  7. Number of tests 59 Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests 1 2 3 4 5 6 7 8 18 Conformance testing saves overall test effort (1) Example:59 interoperability tests with 8 partners (neighbour domains) In this example 472 tests would need to be coordinated and performed

  8. Start with conformance tests... Number of tests ...and reduce the number of interoperability tests 59 18 Interop. Tests Interop. Tests Interop. Tests Interop. Tests Interop. Tests Interop. Tests Interop. Tests Interop. Tests Conformance testing saves overall test effort (2) In this example the overall number has been reduced to 203 tests ( = 59 + 144 ) Conformance Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests Interoperability Tests 1 2 3 4 5 6 7 8

  9. Test effort comparison (1) All test cases have to be performed with each partner!

  10. Test effort comparison (2)

  11. Reasons for a conformance test tool • Improved test coverage, more test capabilities • overcome scenario limitations encountered in the FIRST interoperability trials • enforce fault situations (= negative testing) to check IUT robustness, e.g. erroneous AMHS parameters • Better fault analysis • test tool can analyse captured AMHS messages, reports and probes • test tool can act as a reference system • Easier regression testing • less manual interaction, tests can be repeated very exactly (and quickly) • no need to use AMHS links to neighbour domains to verify your software updates

  12. EUR AMHS testing strategy • AMHS conformance test environment • Test experience

  13. AMHS Test Tool Concept • Build a test tool with all necessary components to simulate an operational AMHS environment of a COM centre • Provide a test platform for loadable test scenarios • modification or enhancements of standard tests • reuse of test messages and test sequences • easy creation of new tests • Minimise manual intervention by test automation

  14. AFTN Environment COM Centre AFTN User (1) AFTN/AMHS Gateway AFTN User (2) AFTN Center Local AFTN User Terminal AFTN (CIDIN) interface AFTN User (s) Monitor & Control Position AMHS Environment UA 1 ATS Message Server (MTA) MTA-1 UA 2 AMHS (P1) interfaces Local AMHS User Agent (UA) UA 3 MTA-2 User Agents (UA) UA 4 MTA-n UA m Message Transfer Agents (MTA) UA o Generic COM-Center Environment (AMHS view)

  15. AFTN/AMHS Gateway Local AFTN User Terminal Monitor & Control Position ATS Message Server (MTA) Test MTA-1 Local AMHS User Agent (UA) Test MTA-2 Test MTA-3 AMHS Test Interfaces Test Interfaces AFTN Test Environment IUT AFTN Test Appl. AFTN (CIDIN) interface AMHS Test Environment AMHS (P1) interfaces

  16. AMHS Test Tool AFTN/AMHS Gateway Local AFTN User Terminal Testlog Monitor & Control Position Test Control and Evaluation Application MessageSource/Sink ATS Message Server (MTA) Local AMHS User Agent (UA) Viewer/Editor User Interface AMHS Test Tool Architecture Test Interfaces IUT AFTN Test Appl. AFTN (CIDIN) interface Test MTA-1 AMHS (P1) interfaces Test MTA-2 Test MTA-3

  17. Features of AC-B‘s AMHS Test Tool • Test interfaces • AMHS over TCP/IP • AMHS over ATN (option) • AFTN over CIDIN/X.25 PVC • AFTN over X.25 (option) • AFTN over TCP/IP (option) • Test cases • 59 conformance test scenarios according to EUR AMHS Manual (App. D) • More than 200 prepared AMHS and AFTN test messages • Functions • Generation of IPMs, IPNs, reports, probes with user-defined parameters • Verification of IPMs, IPNs, reports, probes received from the IUT • Automatic response to received messages and probes: DR, NDR, RN, NRN

  18. AMHS Conformance Test Scope • Test groups • Submission operations • Delivery operations • Transfer operations • Gateway operations • AMHS -> AFTN • AFTN -> AMHS • Test classes • Verification of normal communication • Handling of communication errors (negative testing – robustness tests)

  19. Table tree view for test steps Run control buttons Details for a selected test step Indication of test step success / failure AMHS Test Tool - HMI examples (1)

  20. Test Control and Evaluation Application MTE IPM Report IPN Probe AMHS Test Tool - HMI examples (2) AMHS Viewer/Editor

  21. AC-B‘s AMHS Test Services • AC-B offers AMHS test services to ANSPs and AMHS suppliers • The basic AMHS conformance test service package includes • preparation of the AMHS test tool for the on-site test • on-site test session to run a set of standardized AMHS conformance test procedures • test evaluation and test report • test certificate (= conformance test statement) • Customized test service packages can be offered • to perform regression testing, or • to verify optional, customer-specific AMHS features

  22. EUR AMHS testing strategy • AMHS conformance test environment • Test experience

  23. AMHS conformance test sessions • 9 conformance test sessions in 2006, 2007 and 2008 • 5 initial test session and 4 regression test • regression tests were due to • normal (planned) product release upgrades, or • unsatisfying initial test results • 5 AMHS systems were tested • DFS (Germany), Austrocontrol (Austria), MATS (Malta), AFSBw (German Military), NAVIAIR (Denmark) • These 5 systems came from 5 different suppliers

  24. AMHS conformance test statistics

  25. Experiences made • The first (initial) test of a product takes time • Connection set-up is never easy b e p a t i e n t ... ! • Inital test runs never flow smoothly do not get frustrated! • A renewed (second) test run can be done very efficiently (even in one day) • Some major problems that occured were not specific to the AMHS standard • When we defined AMHS tests, we believed that base X.400 MHS products are mature that was wrong! • Tests revealed problems also on X.400 level and even ISO transport layer s u r p r i se ?

  26. Conclusions and recommendations (2) • Conformance testing is only one step in a series of tests • stepwise testing at supplier and customer premises required • conformance tests do not replace FAT/SAT and are no guarantee for perfect AMHS operation (due to limited scope and specific test parameters) • The system‘s configuration is an issue • the conformance test environment uses test parameters, e.g. for mta name, domain names, AFTN and AMHS addresses • later configuration changes may result in different behaviour (e.g. gateway use of address mapping tables) • configuration control is the supplier‘s responsibility • Recommendation: • Perform the official conformance tests at the target system, best at ANSPs premises

  27. Conclusions and recommendations (2) • Today‘s standardized conformance tests are absolutely necessary but not necessarily sufficient • In many cases non-conformant behaviour was detected by the test tool and the software could be updated (improved) before the official test run • but... • test scope limited mainly to AFTN/AMHS gateway and MTA • only few MTA submission and delivery tests • no standardized test for UA-MS/MTA and UA-UA communication • Recommendation: • Enhance the scope of standardized conformance tests

  28. Invitation • Please consider AMHS support from AC-B in your implementation projects • AMHS Conformance Test Services • AMHS Seminars • The AMHS Book

  29. www.ac-b.de/amhs

More Related