Playing the Devil ’ s Advocate: Verifying Real-Time Systems

1. Playing the Devil’s Advocate: Verifying Real-Time Systems Jan Jürjens Software & Systems EngineeringTU Munich, Germany juerjens@in.tum.de http://www.jurjens.de/jan

2. Dependability Ability to deliver service that can justifiably be trusted. By definition issue supposed to include Reliability, Safety, Security attributes, the threats against them and the means to encounter the threats. (Although historically „security“ and „dependability“ communities have been largely disjoint.) Jan Jürjens, TU Munich: Critical Systems Development with UML

3. Safety Safety: software execution without contributing to hazards. Safety-critical systems: five failure condition categories: catastrophic, hazardous, major, minor, no effect. Corresponding safety levels A - E (DO-178B standards in avionics). Jan Jürjens, TU Munich: Critical Systems Development with UML

4. Reliability For safety need sufficient level of reliability: probability of failure-free functioning of software component for specified period in specified environment. Reliability goals: via the maximum allowed failure rate. For high degree of reliability, testing not sufficient (1 failure per 100,000 years). Jan Jürjens, TU Munich: Critical Systems Development with UML

5. Embedded Systems In particular, embedded software increasingly used in safety-critical systems (flexibility): • Automotive • Avionics • Aeronautics • Robotics, Telemedicine • … Our treatment of dependable systems in particular applies to embedded systems. Jan Jürjens, TU Munich: Critical Systems Development with UML

6. Fault-tolerance Redundancy modeldetermines which level of redundancy provided. Goal: no hazardsin presence of single-point failures. In the following treatment: • focus on reliability, in particular for safety • focus on fault-tolerance aspects of reliability Jan Jürjens, TU Munich: Critical Systems Development with UML

7. Faults vs. Failures Failures: perceived deviation of output values from expected values. Faults: possible cause of failures in hardware, code or other artefacts. For example, a faulty communication line may result in a communication failure. Failures relative to system requirements (real-time: inacceptable communication delay „failure“). Jan Jürjens, TU Munich: Critical Systems Development with UML

8. Faults vs. Failures II Faults in component cause failures of component. Are faults of subsystem containing component. Leads to failures of subsystem …  Faults / failures relative distinction. Can be both. Jan Jürjens, TU Munich: Critical Systems Development with UML

9. From UMLsec to UMLdep Reliability = „Security against stupid adversaries“ Security = „Reliability for paranoids“ Adversariesin security correspond to failuresin reliability. Replace adversary model in UMLsec by failure model to get UMLdep. Jan Jürjens, TU Munich: Critical Systems Development with UML

10. Fault Semantics Modeling For redundancy model R, stereotypes∊{¿crash/performanceÀ,¿valueÀ}, have set FaultsR(s)⊆{delay(t), loss(p), corrupt(q)}, with interpretation: • t: expected maximum time delay, • p: probability that value not delivered within t, • q: probability that value delivered in time corrupted (in each case incorporatingredundancy). Or use ¿riskÀstereotype with {fault} tag. Jan Jürjens, TU Munich: Critical Systems Development with UML

11. Example Suppose redundancy model R uses controller with redundancy 3and fastest result. Then can define: • delay(t): t delay of fastest controller, • loss(p): p probability that fastest result not delivered within t, • corrupt(q): q probability that fastest result is corrupted (each wrt. given fault semantics). Jan Jürjens, TU Munich: Critical Systems Development with UML

12. ¿guaranteeÀ Describe guarantees required from communication dependencies resp. system components. Tags: {goal} with value subset of {immediate(t), eventual(p), correct(q)}, where • t: expected maximum time delay, • p: probability that value is delivered within t, • q: probability that value delivered in time notcorrupted. Jan Jürjens, TU Munich: Critical Systems Development with UML

13. Reliable Architecture Is this a reliable architecture ? Jan Jürjens, TU Munich: Critical Systems Development with UML

14. ¿reliable linksÀ Physical layer should meet reliability requirements on communicationgiven redundancy model R. Constraint: For dependency d stereotyped ¿guaranteeÀ and each corresponding communication link l with stereotype s: • if {goal} has immediate(t) as value then delay(t‘) 2 FaultsR(s) implies t‘·t, • if {goal} has eventual(p) as value then loss(p‘) 2 FaultsR(s) implies p‘·1-p, and • if {goal} has correct(q) as value then corruption(q‘) 2 FaultsR(s) implies q‘·1-q. Jan Jürjens, TU Munich: Critical Systems Development with UML

15. Example ¿reliable linksÀ Given redundancy model none, ¿reliable linksÀ fulfilled iff T ≥ t where delay(t) 2Faultsnone(¿crash/performanceÀ). Jan Jürjens, TU Munich: Critical Systems Development with UML

16. Reliable Data Structure Assuming immediate(t) goals(realtime), data structure reliable ? Jan Jürjens, TU Munich: Critical Systems Development with UML

17. ¿reliable dependencyÀ Communication dependencies should respect reliability requirements on ¿criticalÀdata. For each reliability level {l} for ¿criticalÀ data, have goals(l){immediate(t), eventual(p), correct(q)}. Constraint: for each dependency dfrom C to D stereotyped ¿guaranteeÀ: • Goals on data in D same as those in C. • Goals on data inC that also appears inDmet by guarantees of d. Jan Jürjens, TU Munich: Critical Systems Development with UML

18. Example ¿reliable dependencyÀ Assuming immediate(t) goals(realtime), violates ¿reliable dependencyÀ, since Sensor and dependency do not provide realtime goal immediate(t) for measure() required by Controller. Jan Jürjens, TU Munich: Critical Systems Development with UML

19. ¿reliable behaviorÀ Ensures that system behavior in presence of failure model provides required reliability {goals}: For any execution trace h, any transmission of a value along a communication dependency stereotyped ¿guaranteeÀ, following constraints should hold, given the reliability goal: • eventual(p): With probability at least p, … • immediate(t): … every value is delivered after at most t time steps. • correct(q): Probability that a delivered value is corrupted during transmission is at most 1-q. Jan Jürjens, TU Munich: Critical Systems Development with UML

20. Dependable Interference Acceptable interference between safe and unsafe data ? Jan Jürjens, TU Munich: Critical Systems Development with UML

21. ¿containmentÀ Prevent indirect corruption of data. Constraint: Value of any data element d may only be influenced by data whose requirements attached to ¿criticalÀimply those of d. Make precise by referring to execution semantics (view of history associated with dependability level). Jan Jürjens, TU Munich: Critical Systems Development with UML

22. Example ¿containmentÀ Violates containment because a {safe} value depends on un{safe} value. Can check this mechanically. Jan Jürjens, TU Munich: Critical Systems Development with UML

23. Tool Support: Fault Models lqln: messages on link l delayed further n time units. phn: probability of fault at nth iteration in history h. For link l stereotyped s where loss(p)FaultsR(s), • history may give lql0:=; then append p to phn, • or no change, then append 1-p. For link l stereotyped s where corruption(q)FaultsR(s), • history may give lql0:={■}; then append q, • or no change; append 1-q. For link l stereotyped s with delay(t)FaultsR(s), and lql0, history may give lqln:=lql0 for n·t; append 1/t. Then distribute lql0; for each n, lqln:=lqln+1. Jan Jürjens, TU Munich: Critical Systems Development with UML

24. Other Checks Have other consistency checks such as • Is the software‘s response to out-of-rangevalues specified for every input ? • If input arrives when it shouldn't, is a response specified ? …and other safety checks from the literature. Jan Jürjens, TU Munich: Critical Systems Development with UML

25. Any Questions ? Jan Jürjens, TU Munich: Critical Systems Development with UML

26. Testing Real-Time Systems Very challenging. For high level of assurance, would need full coverage (test every possible execution). Usually infeasible (especially reactive systems). Have heuristics for trade-off between development effort and reliability. Need to ask yourself: • How complete is the heuristic ? • How can I validate it ? Jan Jürjens, TU Munich: Critical Systems Development with UML

27. Recent Trends in Academic Research Model-based Testing (e.g. based on Real-time UML). Advantages: • Precise measures for completeness. • Can be formally validated. Two complementary strategies: • Conformance testing • Testing for criticality requirements Jan Jürjens, TU Munich: Critical Systems Development with UML

28. Conformance Testing Classical approach in model-based test-generation (much literature). Can be superfluous when using code-generation [except to check your code-generator, but only once and for all]. Works independently of real-time requirements. Jan Jürjens, TU Munich: Critical Systems Development with UML

29. Conformance Testing: Caveats • Complete test-coverage still infeasible (although can measure coverage). • Can only test code against what is contained in model. Usually, model more abstract than code. May lead to „blind spots“. For both reasons, may miss critical test-cases. Want: „criticality testing“. Jan Jürjens, TU Munich: Critical Systems Development with UML

30. Criticality Testing: Strategies Internal: Ensure test-case selection from models does not miss critical cases: Select according to information on criticality. External: Test code against possible environment interaction generated from parts of the model (e.g. deployment diagram with information on physical environment). Jan Jürjens, TU Munich: Critical Systems Development with UML

31. Criticality Testing Shortcoming of classical model-based test-generation (conformance testing) motivates „criticality testing“. Goal: model-based test-generation adequate for critical real-time systems. Jan Jürjens, TU Munich: Critical Systems Development with UML

32. Internal Criticality Testing Need behavioral semantics of used specification language (precise enough to be understood by a tool). Here: semantics for simplified fragment of UML in „pseudo-code“ (ASMs). Select test-cases according to criticality annotations in the class diagrams. Test-cases: critical selections of intended behavior of the system. Jan Jürjens, TU Munich: Critical Systems Development with UML

33. External Criticality Testing Generate test-sequences representing the environment behaviour from the criticality information in the deployment diagrams. Jan Jürjens, TU Munich: Critical Systems Development with UML