BE-SECBS Benchmark Exercise of Safety Evaluation of Computer Based Systems

1. BE-SECBSBenchmark Exercise of Safety Evaluation of Computer Based Systems Safety Assessment of MADTEB-Limitation Functions at ISTec

2. BE-SECBSFISA 2003November 13th 2003 ASSESSMENT OBJECT DESCRIPTION MADTEB System • Part of the reactor limitation system to limit the allowed range of process variables (coolant pressure, pressurizer level) of the primary coolant loop of the reactor • Consisting of 8 limitation functions implemented in 4 redundant trains • Implemented in TELEPERM XS (TXS) Technology • Implementation comprising Data-Acquisition, Limitation-Functions, Priority-Functions, Output-Functions for all 4 trains consisting of about 300 Function Diagrams of TXS

3. BE-SECBSFISA 2003November 13th 2003 TXS – BASIC STRUCTURE • Specification based on a graphical user interface (SPACE) • I&C functions (functional diagrams FD) are constructed from prefabricated, normed basic elements (function blocks FB) and are organized as groups (functional diagram groups FDG) which are executed cyclically on single processing units. • The graphical specification is stored in a database. • Code generation is based only on database tables, and uses and includes only prefabricated FB modules and declaration files. Function Diagram

4. BE-SECBSFISA 2003November 13th 2003 ASSESSMENT OBJECT RELEVANT PROPERTIES Relevant Properties of Test Case BE-SECBS due to Implementation in TXS-Technology: • Strict formal character already in the first specification steps by use of Function Diagrams • Function Diagrams are unambiguous and can be checked in detail. • Resulting C-code is so called Normed Source Code with some predefined properties • Strictly cyclic and data independent execution of the code within a fixed time interval • Strictly linear control flow structure, i. e. the execution path ( and execution-timing) is predefined and not dependent on input data • Resulting C-code is based on a code-library, the so called Function-Blocks. Function-Blocks are type-tested software components that allow to rely on their compliance with the specification of their functional properties in a data sheet.

5. BE-SECBSFISA 2003November 13th 2003 BASIS FOR ASSESSMENT METHODOLOGY AT ISTEC Standards: • IEC 60880 • KTA 3503 („Type-Testing of electrical modules of the reactor protection system“) National Regulatory Rule Type-Test: • Pre-developed components (Function-Blocks, RTE) are tested application-independent • Subsequent applications of these components can refer to this Type-Test and do not need to assess these components again.

6. BE-SECBSFISA 2003November 13th 2003 ASSESSMENT ACTIVITIES • Adaptation of the Assessment Methodology to the Test Object being Normed Code (TXS) • Assessment Steps performed, according to Life-Cycle-Steps of IEC 60880 and Development Documentation: • Requirement Specification • System Specification • Detail Design • Coding / RETRANS-Analysis • Testing

7. BE-SECBSFISA 2003November 13th 2003 ADAPTATION OF THE ASSESSMENT METHODOLOGY Impact of TXS-Properties (Normed Code) on Assessment Methodology: • No need for static analysis (e.g. LDRA, SPADE, CANTATA) of the generated code in order to identify complexity of code structure, control flow paths or some other properties supporting the assessment activities, as it is the case with conventional (not normed) software • No need for dynamic analysis for measuring of test-coverage, because the test-coverage is predefined by the structure of the automatically generated code. There is only one main path that is executed in each cycle. • Amount of Testing determined only by functional aspects

8. BE-SECBSFISA 2003November 13th 2003 BASIC ASSESSMENT CONCEPT Overall picture of the activities performed in the BE-SECBS project together with the assessed documentation and the assessment outputs.

9. BE-SECBSFISA 2003November 13th 2003 OVERALL STRUCTURE OF THE ASSESSMENT ACTIVITIES-1 Basic Framework for Assessment-Steps • Basic Documents: • Development documents • Function-Diagrams in database BESECBS01 • Assessment-Method • Desk inspection of the respective documents • Analysis of the functionality of the Function-Diagrams in the database. • Application of SPACE-Tool for analysis of the Function-Diagrams in the project-database BESECBS01 for • navigating through the system • tracing the signal-paths • Application of RETRANS (developed at ISTec) checking the functional equivalence of automatically generated source code with its underlying specification. • Assessment-Output / LOPs • Document containing a List of Open Points (LOPs) • LOPs handed out for audit and clarification to the developing organisation (FANP) • Paper with comments on the LOPs sent back to ISTec

10. BE-SECBSFISA 2003November 13th 2003 OVERALL STRUCTURE OF THE ASSESSMENT ACTIVITIES-2 LOPS Main Types of Problems addressed in the LOPs • Inconsistencies between different Specification Levels • Inconsistencies between descriptions and Function Diagrams • Problems within Function Diagrams • Understanding of descriptions and of technical details Main Benefits of the LOPs • Identification of errors insufficiencies and weak points of the product  improving safety and quality of product. • Identification of inconsistencies and insufficiencies of the documentation  initiating revisions and updating of documentation  renders a consistent and complete status of documentation

11. BE-SECBS FISA 2003November 13th 2003 FINDINGS OF THE ASSESSMENT-STEPS-1 - VERIFICATION OF SYSTEM-SEPCIFICATION VS. REQUIREMENT-SPEC System Specification Check of a consistent and correct translation of the I&C functions description of the Process Requirements, into Prototype Function-Diagrams • Majority of the LOPs referring to changes and modifications • not explicitly documented or explained, • during translation of Process Requirements into FD-Templates • Identification of a Fault / Example of LOP-Procedure • ISTec-Question:Did the “inj. Valve (AA011) open” ever get the value “1”? This means: after an external event the inj. valve closes, but what happens in the event the coolant pressure decreases below 166 bars and the PRZ level below 11 m? • FANP-Answer:The injection valve is operated during normal volume control. As the functional requirements were not validated for the Benchmark Exercise, this is a fault in the requirement specification. AA011 has to be open for spraying as well as injection. • Remark: • Validation of Requirement-Spec. by process- engineer was out of scope of BE-SECBS test-case !

12. BE-SECBS FISA 2003November 13th 2003 FINDINGS OF THE ASSESSMENT-STEPS-2 - VERIFICATION OF DETAIL-DESIGN VS. SYSTEM-SPECIFICATION DETAIL-DESIGN Majority of LOPs referring to undocumented modifications between Template Function Diagrams and final implementation.Also one not intended modification was identified. • Redefinition of logic during Detail Design (shifting of tasks between different Function-Diagrams) • Modifications by SIVAT-Results, in order to • avoid triggering or spurious signaling during start-up of a CPU • improve the tolerance against faulty signals from the RPS • Priority Logic • Assessment identified insufficient implementation of Priority Logic within TXS Function-Diagrams • According to FANP-answer no fault because priority is implemented by an external logic downstream TXS outputs • Inconsistent Modification • Not intended inconsistency introduced while creating the Function-Diagrams from FD-Template for Analog Input. (Inhomogeneous usage of Flip-Flop-types within the DFD JEB00CS811 for the Input of RCP-Speed) • According to FANP-answer no impact on functional behaviour of the integrated system

13. BE-SECBSFISA 2003November 13th 2003 RETRANS ANALYSIS TOOL-STRUCTURE

14. BE-SECBSFISA 2003November 13th 2003 RETRANS ANALYSIS ANALYSIS RESULTS RETRANS Essential results of the software analysis tool: • Automatic comparison of the specification of the application programs stored in a database with the functionality of the automatically generated C-Source Code. • Hints for the analyser with respect to the plausibility of FB parameters in redundant channels. • Hints concerning inconsistencies in the database. • Hints concerning inconsistencies in the C-Source Code

15. BE-SECBS FISA 2003November 13th 2003 FINDINGS OF THE ASSESSMENT-STEPS-3 VERIFICATION OF CODING BY RETRANS-ANALYSIS RETRANS-Analysis Findings • The C-Code of the Benchmark-Test-Object is in accordance with the function diagrams in the related SPACE data base. • In SPACE the sequentialisation within the function diagrams is not unique, even if the Function Diagrams are totally equivalent from topological (i. e. optical) point of view. • Deviations make the verification of coding more complicated Deviations in sequentialisation, which were found, don‘t change the functionality of the resp. Function Diagrams • RETRANS plausibility check identified minor deficiencies of the benchmark test-case: • deviations within extensions of some block-numbering • incorrect explanatory text for some signal identifiers. Both deficiencies don’t violate the correctness of code, yet they have to be avoided within real safety applications. • The plausibility control in real applications yields more findings which finally are errors. Comparison and plausibility check for the whole C-Code of Benchmark-Testcase with the content of corresponding BE-SECBS database About 300 FDs, about 1000 pages of Diagrams .

16. BE-SECBS FISA 2003November 13th 2003 FINDINGS OF THE ASSESSMENT-STEPS-4 TESTING Testing • The amount of testing required for the assessment of this TXS-Code (Normed Code) is • only depending on the functionality of the system under test • independent of any code-coverage-measuring. • The testing strategy performed by the developer within the benchmark test case is in accordance with the properties (Normed Code) of the test-object.Functional Tests refer to • the basic functionality of the system • specific properties of the functionality • interface to its environment concerning I&C and process • system behaviour under failure condition for relevant ranges and combinations of input signals • Comprehensiveness and sufficiency of Testing: • In a real project, there will be a need for more effort concerning the understanding of the relevant process engineering topics and problems on basis of which the comprehensiveness of the functional tests can be assessed. • Validation of process engineering aspects was not foreseen and not performed within this benchmark-exercise.

17. BE-SECBS FISA 2003November 13th 2003 SUMMARY OF THE ASSESSMENT PERFORMED AT ISTEC SUMMARY • The property of the test-object being Normed Code, which is automatically generated, has an essential impact on the type of the required assessment activities. • Assessment method applied in the benchmark test-case provides the following important benefits for the V&V process: • Identification of inconsistencies and insufficiencies within and between the various development documents • Complete and comprehensive documentation representing the actual status of development • Tracing all the modifications implemented during the development process • Capability to find errors, weak points or insufficiencies that pass some quality assurance procedures and testing activities, as assessment is a diverse and supplementary method to testing

18. BE-SECBS FISA 2003November 13th 2003 CONCLUSION Normed-Code-Systems support the Assessment Activities and thus contribute to • SAFETY • Formal method (Function Diagrams) for the description of the system from the very beginning of the development • Approved components (Type-Testing) • Approved structure and properties (Type Testing) • Concentration of assessment on application-specific aspects • COST-EFFECTIVENESS • Some Assessment-Tasks are performed once for all by Type-Testing • Reduced set of aspects to check for an individual implementation • Checks can be reasonably automized (RETRANS) due tot he Normed Structure of the Code.