1 / 17

ETCS software testing How to avoid “Black swans”

ETCS software testing How to avoid “Black swans”. Istanbul 01-04/04/2014. Marc ANTONI, Senior Advisor Signaling. Summary. Aim ETCS shall be integrated in the railway system reality Computerized critical system and Black Swans Synthesis & Recommendations.

cala
Download Presentation

ETCS software testing How to avoid “Black swans”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ETCS software testing How to avoid “Black swans” Istanbul 01-04/04/2014 Marc ANTONI, Senior Advisor Signaling

  2. Summary • Aim • ETCS shall be integrated in the railway system reality • Computerized critical system and Black Swans • Synthesis & Recommendations ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  3. ETCS applies critically interconnected computerized systems. The complexity needs to consider a new paradigm regarding the assessment and the validation of the components. Until now the applied methods have notable disadvantages: 1. Classical methods only can check test cases:- it was required to identify the threats and to reduce they probabilities of occurrence- it was only possible to validate a system using “tests cases”… 2. Criticality check by computerized systems could be not affordable or sufficient: - the list of threats events is practically not countable- it is necessary to define the boundaries of all system states and be able to proof formally that the system never leave the defined boundaries – difficult to accept It is necessary to use “formal methods” to avoid the occurrence of “black swans” This is impossible with “test cases” applied on the integrated system. Aim: No “black swans” in the ETCS It is a non detected system state, with potential harm, which can appear under certain non imagined combination of inputs. The safety software shall avoid this – how to check?. 3 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  4. Currently no guarantee with conventional methods Feedback experience: • Some studies show that ¾ of accidents linked with computerized systems are due to “specification” issues (incomplete or false functional descriptions, initial or after SW maintenance...). Many examples can be given! If the computerized system is “complex” instead to be “complicated”, it will never be possible to proof its safety (validate it) ! Complex system   Complicated system 4 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  5. … Application of standards EN50128, DO178B, CEI61508 Application des normes EN50128, DO178B, CEI61508 Separate hardware from functional software Separation shall be done between: • the functional software what and why (or why and what) • the hardware and its integration software how Software of computerised systems Hardware of computerised systems Random HW failures External stress Random aspects Systematic failures of functional SW Probability studies N/P architecture Quality assurance >75% Formal Methods ! / - 5 5

  6. ETCS shall be integrated in the existing railway system ETCS is: a “block system”  On infrastructure a “cab signalling system” and a “speed control system”  On board ETCS integrated in the railway system (1) Men (skill, competences, education…) Environment (STI, corridors…) Operation rules (national) Gravity point Infrastructure (national and international) Rolling Stocks (national and international) ETCS impacts the five axles of the railway system 6 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  7. The signallingsystem uses different levels of specialised systems Each level has its own characteristics, life time et renewal criteria: Remote control centre: without safety function, “sequential”, central ETCS : block and speed control system, European, “combinatorial”, linear (for a corridor) IXL : Interlocking, national (operation and shunting rules, track layout…), “sequential”, nodal (bifurcation, station…) Field resources : national, “combinatorial”, punctual For safety proof and testing reasonsIXL and RBC shall never be merged in a single computerized system! ETCS integrated in the railway system (2) Formal functional Interfacesdefined by the IM 7 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  8.  Precise requirements, non-interpretable and trustful specifications of CCS systems (especially IXL, RBC and EVC) and of relevant interfaces are necessary to proof / validate the ETCS on an intended application area: ETCS integrated in the railway system (3) RBC computerized system Over system Operator Maintenance sensors Track layout IXL or Electrical system Operation rules EVC system Radio link  Formal functional specifications are necessary for a testable ETCS 8 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  9. ETCS integrated in the railway system (4) • For signalling technologies used (mechanical, electrical, computerized), each wrong side failure can lead in a direct way to a threaded event. • With computerized critical system the today's standards lead to: • the safety demonstration are now realized with a “mean obligation” (ex-post demonstration) and no more a “result obligation” (ex-ante demonstration) as with the previous technologies • the railway safety standards are build in terms of probabilities (Frequency x Severity) and Safety Integrity Level (level of confidence in the software) • initially written as an Arianne threat during the software development, the standards are today applied with documentation delivery objectives • the cross acceptance of systems increase the loos of the product and system mastery, increase the risks of use for certified products putting into service in a not adapted context (national rules…). 9 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  10. Black Swans may lead to not assumable events • Non assumable dangerous situations are not acceptable • They can be accepted by standards in terms of probabilities (severity matrix with log x log scales of Loss x Probability of occurrence index, risk index). • With previous technologies these not assumable situations are treated in a deterministic way. High severity with low probability of occurrence  are the consequences assumable? which threshold? Low severity with high probability of occurrence  are the consequences assumable? (availability) which threshold? Rare events to be eradicated 10 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  11. Black Swans can appear from ambiguous interpretable specification • The application of standards can induce “de facto” limitation of work in the single computer subsystem and the creation of a cluster formed by the system status (railway) and the software status (IT system) • If an error due to a misunderstanding of a system specification is introduced at the software specifications level, this error will be developed, tested and validated… leading to a possible wrong side failure > black swan is kicked off • The ways to ensure that the supplier has correctly applied the functional requirements are:- to formalize (with a formal language) the applicative software of the IT system, - to simulate its behaviour in an instantiated framework and - to validate the formal high-level specifications using formal proofs • Formalization and formal proof shall avoid “black swans” 11

  12. ETCS application approval: hunting the black swans in practice • Railways take measures to increase their mastering of safety, reduce costs and delays: •  Extendedtesting: All onboard units shall be tested with each field equipment / all field equipment shall be tested with each on board unit & assure transparent monitoring access of the National Safety Authority => in service after recover applicative bugs •  Formal modelling of RBC applicative functionalities and interfaces with the existing environment, simulation of event input combinations=> in service after recover of bugs from application environment, modelling reduces the future maintenance costs •  Formal validationof computerized critical system application software and the formal separation between the HW platform and the application software=> to avoid all deterministic wrong side failures and to master costs and delays, especially in case of HW obsolescence or SW evolution (SRS change) 12 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  13. Example of onboard EVC – No black swan from the specification model Formal Validation of the application specifications  critical parameters Speed (km/h)  Specification model taking into account all the parameters for a result obligation Distance (m) 13 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  14. Example of onboard EVC – No black swans from variation of input parameters Formal Validation of application specifications  critical parameters Speed (km/h) Trajectory leading to an overshoot • Identification of scenario leading to an overshoot (ex: 250m with -15% adherence) => evolution of the specifications to covers this situation Distance (m) 14 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  15. The use of formal methods (especially formal validation of the applicative software) is a key procedure for a railway to preserve the high level of safety, the durability and the maintenance costs of the critical computerized systems. Synthesis & Recommendations Three main recommendations: • Control the applicative software: to formalise and simulate specifications, to prove them formally before any real application. • Write the specifications in the professional manner: to formalize, capable of validation trough simulation and, provable formally under the environment system constraints. • Apply computer architectures to make distinction of applicative software and hardware platform: functional must be understandable by railway agent and controllable by the railway – the hardware platform is responsible for its interpretation and is under control by one or more suppliers. 15 ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  16. Recommendation: simple questions to face  Risk assessment models are based on incomplete knowledge of the phenomena involved, uncertainty analysis therefore plays an important role in the confidence of decision-making, in particular in the case of the unspecified expression 0 x infinite, • The “simple” application of standards and classical “test cases” for today’s complex system is not sufficient to conceive safe systems, without functional failures in accordance to the operational environment, • To face the question: Can “we” be able to consider the “black Swans” without to wait for collapses, to serious accidents ? Considering the standards, the today answer is clearly NO for computerized critical systems  Formal methods are the only way to prove that the non-assumable risks are really “eradicated by design”…no black swan by design! ETCS2 software testing – How to avoid Black Swans - 01/04/2014

  17. Thank you for your kind attention Marc ANTONI, Senior Advisor Signaling antoni@uic.org ETCS2 software testing – How to avoid Black Swans - 01/04/2014

More Related