AUTOSAR Conformance Tests - Feedback on their development and utilization

1. Facing Up to the Future of Testing AUTOSAR Conformance Tests - Feedback on their development and utilization Alain Feudjio-Vouffo, Fraunhofer Fokus (Germany), alain.vouffo@fokus.fraunhofer.de Alexandra Faucheux, Kereval (France), alexandra.giret@kereval.com Mathieu Morvan, Kereval (France), mathieu.morvan@kereval.com Alain Ribault, Kereval (France), alain.ribault@kereval.com Ashish Jadhav, Patni Computer Systems (India), ashish.jadhav@patni.com Philippe Robin, Technoveo (France), philippe.robin@technoveo.com

2. Content of the presentation • Introduction: AUTOSAR CTSpecs release status • AUTOSAR CTSpecs & abstraction from hardware platform • Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment • Combination of conformance test and source code coverage • Looking into the future • Theoretical view on combination and reuse of TTCN-3 suites to create new ones • From ATS for module to ATS for stack and integrated software platform • Example of deployment in the automotive industry • Conclusion

3. Introduction - AUTOSAR CTSpecs release status • AUTOSAR conformance test specifications (CTSpecs) have been released on 15 April as part of Release4.0R0002 • Conformance tests are available for 43 modules (including the RTE) • A CTspec contains • the specification of the conformance test suite • the TTCN-3 scripts for dynamic test cases • the configuration sets to be used when executing the Conformance Test Suite • Number of test cases 6820 • Number of dynamic tests (TTCN3) 2754 • Number of static tests 4066

4. AUTOSAR CTSpecs & abstraction from hardware platform (1/2) TTCN-3 Executable • Adapters are generated • ETS is hardware independent TTCN-3 Compiler TTCN-3 files Adapters Adapters Generator COM Execution of the CTS ETS on PC COM COM COM COM HW Specific HW Specific HW Specific HW Specific Module Under Test Module Under Test Module Under Test Module Under Test Module Under Test Instrumented MUT on board KEREVAL' Platform

5. Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment Test Management System Reporting Source Code Inspection Configuration Inspection Operation Signature Tests Dynamic Tests Software, Hardware and Test design tool CUSTOMER Configurations and environment MANUAL EFFORT BSW preparation (configuration) BSW integration Sources CTSpec, TTCN-3 scripts and Configuration Set(s) CAPITALIZED or GENERATED Integration Reception Test suite execution Test suite execution results Attestation 5 5

6. AUTOSAR CTSpecs & abstraction from hardware platform (2/2) • Test Memory Management allows Test Suite to access memory of the board TTCN-3 files TTCN-3 Executable module FeeJo Allocate / Deallocate Adapters Read / Write / Compare /* ... */ var PointerAddr_v_AddrPointer; pt_TestMemoryAccess.call(TestMemoryAccess_Allocate:{v_Size}, nowait); alt { []pt_TestMemoryAccess.getreply(TestMemoryAccess_Allocate:{v_Size}) -> value v_AddrPointer { /* ... */ } } /* ... */ pt_Fee.call( Fee_Read:{v_BlockNumber, v_BlockOffset, v_AddrPointer, v_Size}, nowait ); /* ... */ COM COM Test Memory Management ETS on PC HW Specific Module Under Test Instrumented MUT on board

7. Combination of conformance test and source code coverage • A Code Coverage Analysis tool (RTRT) is used during Conformance Test execution • Provides • An indicator of tests relevance according to structure • An effort estimation to go over functional validation from conformance testing • Information on dead code and unspecified code • Required in Safety Validation context • ISO 26262 and DO178B • Thanks to coverage measurement, conformance testing can be part of validation process

8. Reuse of the TTCN-3 suites • Two types of reuse are envisaged for the module-level TTCN-3 ATS that are available for the 43 modules. • 1) Reuse of the suites for creating the ATS for the later revisions of the AUTOSAR standard during the maintenance phase of the current release, and • 2) Reuse of the suites for creating the ATS for the future releases of the AUTOSAR standard • The ATS has been created having maintainability as a major design criterion. TTCN-3 provides the high level artifacts in the ATS to enable the reuse. • Traceability between the specifications and the test cases and configuration parameters has been created to simplify the upgrade process. AUTOSAR Rel 4.0 AUTOSAR Rel 4.1 … … … Rev 2 Rev 0 Rev 3 Rev 1 RE-USE of Artifacts in the ATS

9. Combination of TTCN-3 suites to create new ones – extensibility • Different users of AUTOSAR might be interested in extending the existing ATS by having more tests that are vendor specific and do not exist in the current ATS for internal test compliance • The TTCN-3 suites are well structured and provide an ease for combining with other vendor specific test cases to extend the tests for internal test compliance • The TTCN-3 suite is well structured and provides ease of extensibility for test cases, configuration parameters and combinations of configuration parameters Extensibility of TTCN3 AUTOSAR ATS (for internal test compliance) AUTOSAR ATS Vendor Specific Module Tests Additional Tests Required by OEMs

10. Higher level Test Suites Full AUTOSAR STACK • The ATS is currently at module level • Using TTCN-3 provides ease of extensibility to combine test-cases and reuse code with additional test cases to obtain a test suite for a set of inter-connected modules • Some of the stubs and drivers would have to be replaced by actual modules and suites used for testing of stacks like the CAN stack, the Flexray stack, the communication stack, etc. can thus be prepared • In such type of higher level suites various AUTOSAR modules can be simultaneously tested • The test cases of the ATS can also be used to develop test suite for testing of a full AUTOSAR stack • The test cases of the ATS can also be used for developing test suites for testing of AUTOSAR compliant software components ATS for Combination of Modules Module ATS

11. Example of deployment in the automotive industry • Feedback on executions • Average of 10 non conformities detected per Module Under Test at first CT execution • Typical non conformities detected on each module • Such non conformities have not been detected during validation of the module

12. Example of deployment in the automotive industry Functional design Functional Tests SW Architecture design Integration Tests Coding Unit Tests Confirmation test and attestation delivery • Non-regression tests for any changes in: • Configuration or Implementation, • HW platform Module Conformance checking Conformancecheckingaccording to AUTOSAR description files (non-TTCN-3 tests) • Integrated with customers development plan • Multi run conformance tests

13. Conclusions • Hardware dependent ETS is obtained from ATS and is used for testing the MUT, this ensures that the AUTOSAR CTSpecs are portable • The use of AUTOSAR CTSpecs TTCN-3 code and test environment by CTAs need a very low percentage of manual activities and up to 95% of activities can be tool-based • AUTOSAR Conformance testing can be combined with other tests like source code coverage so that independent test runs can be reduced resulting in overall reduction in test execution time • AUTOSAR ATS has maintainability and extensibility enabled by use of TTCN3 for efficient reuse in creating new ATS for future releases, ehancements and higher level testing • AUTOSAR Conformance testing can be integrated in the overall AUTOSAR development process so that testing can be performed in early stages of the development and can result in detecting non-conformances even at design stage which would improve the development efficiency