Table of Contents

1. ITS/CVO Architecture Conformance:Interoperability Testing StrategyTable of Contents 1 Introduction 2 Terminology 3 Scope 4 Roles and Responsibilities 5 Test Environment and Tools 6 Test Process 7 Key Criteria Appendix A Level 1 Interoperability Test Matrix Appendix B Example of Interoperability Testing Status Table of Contents The Johns Hopkins University Applied Physics Laboratory

2. What is the Intelligent Transportation System (ITS)/Commercial Vehicle Operations (CVO) Architecture? • Establishes a common framework for interoperability • Defines • the functions associated with ITS/CVO user services, • the physical entities or subsystems within which such functions reside, • the data interfaces and information flows between physical subsystems, and • the communications requirements associated with information flows. • Started with the National ITS Architecture • The Commercial Vehicle Information Systems and Networks (CVISN) Architecture and Design projects refined the ITS/CVO parts of the architecture • IBC Architecture project is focusing on International Border Crossing (IBC) functions Background The Johns Hopkins University Applied Physics Laboratory

3. ITS/CVO Architectural Conformance Critical Conformance Components • Implementation of the operational concepts and practices expressed in the National Architecture and CVISN Architecture • Capturing information efficiently and accurately at its source (e.g. ASPEN for inspections) • Use of standard data definitions, including standard identifiers, to facilitate the exchange of safety and credentials information between jurisdictions • Use of open standards (EDI & DSRC) to communicate between public and private systems, and between jurisdictions • Support of base state agreements through the established clearinghouses • Use of ITS/CVO sensor and control technologies as specified in the National Architecture (i.e., the part of ITS/CVO that is not under the CVISN umbrella), and the integration of that information into the information systems where appropriate Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

4. What Is “Interoperability”? Interoperable products/systems are . . . Different types of products/systems that cooperate to accomplish a shared function. For example, the DSRC readers and transponders are products that must interoperate to support various roadside-vehicle interactions. The CVISN systems in two jurisdictions will interoperate via SAFER to share safety information effectively. Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

5. Interoperability Testing Is a Checkpoint During ITS/CVO Lifecycle Deployment Cycle Checkpoints MOA & Requirements Definition Start Verify commitment to principles Project Planning, Review & Top-Level Design Definition Verify plan is sound and top-level design is consistent Top-Level Design Detailed design for Phase “n” - Verify detailed design is consistent Detailed Design Interoperability Testing at end of Phase “n” - Test that systems meet standards and are interoperable Implementation Test Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

6. ITS/CVO Interoperability Testing Objectives • Verify that interfaces between selected pairs of products/ systems meet the applicable standards (pairwise interface testing) • Verify dataflow and data usage among several selected products/ systems (end-to-end interface testing) Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

7. Verifying Pairwise Interfaces Means . . . • Focus on • Interfaces unique to ITS/CVO • Newly-developed standards (EDI, DSRC) • Verify pairs of interfaces between • Public and private entities • State and multi-state or national systems Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

8. Verifying End-to-End Interfaces Means . . . • Focus on • Geographic interoperability issues • Domestic electronic screening • International border crossing • Functional interoperability issues • Data meaning and interpretation • Multi-purpose transponder • Using interface specifications as a guide, verify end-to-end interfaces in • Snapshot usage • Electronic screening systems • Credentials application • Inspection reporting See Appendix A Level 1 Interoperability Test Matrix for information on potential tests. Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

9. Interoperability Testing Concepts(page 1 of 3) • APL develops the Interoperability Test Suite Package to support testing • Test Scenarios (managerial level descriptions) [Part 1] • Test Cases and Procedures [Part 2] • Test Data [Part 3] • Test Tools (e.g., emulators, simulators) [Part 4] • APL provides guidelines for tests leading up to interoperability testing • Component • Integration • System Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

10. Interoperability Testing Concepts(page 2 of 3) • APL equips and staffs CVISN Test Facility to support initial and on-going testing activities • Provides a stable test bed for systems coming on line to verify that they are ready to interact with “production” system (Learner’s Permit) • Reduces negative impact of CVISN expansion on existing systems (Liability Insurance) • Future Potential: As part of a separate testing program, FHWA authorizes one or more agents to certify selected vendor products to verify that they are actually compliant Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

11. Interoperability Testing Concepts(page 3 of 3) • APL provides criteria for interoperability testing via COACH Part 5 • Based on requirements in COACH Part 1 • System owners (states, motor carriers, and their vendors) perform the testing • As new standards emerge, APL adds them to the test suite, and removes tests related to obsolete standards Knowing what is expected keeps development on track Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

12. CVISN Project “Drivers” Product/System Planning Design OTHER REQUIREMENTS & GUIDING INFORMATION CONFORMANCE “MUSTs” • FHWA Policies • COACH • 1 Reqts & Op • Concepts • 2 Planning • 3 Detailed System • 4 Interface Specs • 5 Interoperability • Test Criteria • EDI Standards & Implementation Guides • DSRC Standards • Operational Practice Agreements Development • Process Guides • Technical Guides • Project plans • Product development plans • CVISN System Design Description • Product requirements & design specs • Prototype or Recommended practices Product/System Component Test Integration Test System Test Product/System Interoperability Test The Johns Hopkins University Applied Physics Laboratory

13. COACH Checklists Drive Interoperability Test Criteria Requirements in the Test Suite Package CVISN Operational and Architectural Compatibility Handbook (COACH) (POR-97-7067) Part 1 Operational Concept and Top-Level Design Checklists Interoperability Test Suite Package Part 4 Interface Specification Part 1 Test Specifications (POR-98-7122) Part 2 Test Cases and Procedures (POR-98-7123) Part 4 Test Data (POR-98-7125) Part 3 Test Tool Description (POR-98-7124) Part 5 Interoperability Test Criteria ITS/CVO Architecture Conformance: Interoperability Testing Strategy (POR-98-7076) CVISN Guide to Integration & Test Test Criteria Requirements Test Planning Information Maintained in Monthly Reports (See Appendix B for Example) The Johns Hopkins University Applied Physics Laboratory

14. APL Responsible for a Test Suite and Test Facility to Support Interoperability Testing Interoperability Test Suite Test Scenarios Test Tools Test Cases Test Data Test Procedures CVISN Test Facility Networks Simulators Transponders Translators Emulated state Readers & carrier products Other products Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

15. Interoperability Testing Approach • Develop guidelines for component, integration, and system testing for state, core infrastructure system, and carrier developers’/integrators’ use • Use test scenarios, test cases, and test data suite from early states’ system testing as the foundation for interoperability tests • Install stimulators, simulators, networks, translators, transponders, readers, state and carrier system emulations, and other test tools in CVISN Test Facility • Plan and execute standard tests as early as the integration test phase • Record test results and compare to expected • Repeat interoperability tests as part of regression testing after system upgrades • Feed back lessons learned to refine standards, COACH, test suites, guides, workshops, and training courses Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

16. Interoperability TestingSets 1 & 2 Defined • Set #1 Interoperability Tests: (June through August 1999) • Execute scheduled pair-wise (PW) & end-to-end (ETE) test scenarios at least once in the CVISN Test Facility between now and the end of August 1999. Some of these tests are expected to be run on the actual MD/VA systems. State personnel should run these tests. • Test Suite Package available in June 1999; describes the test scenarios, test cases & procedures, and test data for the Set #1 tests. The matrix in Appendix A indicates which tests are Set # 1. • APL will support interoperability tests for MD/VA in the CVISN Test Facility prior to running them on state systems. • Running these tests in the APL CVISN Test Facility (as soon as possible) will ensure that tests are valid as written. Interoperability test issues should also surface at that time. By the time these tests are run on the actual MD or VA systems the test procedures should work! • Next Test Suite Package update (October 1999) will describe Set #2. (September 1999 through March 2000). Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

17. Interoperability Testing Will Verify basic functionality Verify that selected transactions occur according to standards Verify interoperability among systems for selected functions Verify that basic interactions are successful Interoperability Testing Won’t Test performance extremes Verify that every transaction occurs according to standards (i.e. testing is not exhaustive across all data combinations) Verify interoperability for all functions (i.e. testing is not exhaustive across all functional scenarios) What Interoperability Testing Will and Won’t Do Overview of Conformance Testing Strategy The Johns Hopkins University Applied Physics Laboratory

18. FHWA supports the development and maintenance of the interoperability tests. States execute the tests. Interoperability Test Suite Package Develop CVISN Test Facility JHU/ APL Define Interoperability Test Scenarios Define Test Cases and Procedures Develop Test Tools Develop Test Data Validate Tests in CVISN Test Facility Schedule Interoperability Tests Tailor Test Procedures and Data Execute Tests Evaluate Tests States & Carriers A&S Expert A&S Expert A&S Expert: An Architecture & Standards Expert provides support to help assess whether the system is in conformance. Schedule CVISN Test Facility Support Test Execution JHU/ APL The Johns Hopkins University Applied Physics Laboratory