Software Reliability

Software Reliability CIS 376 Bruce R. Maxim UM-Dearborn

Functional and Non-functional Requirements • System functional requirements may specify error checking, recovery features, and system failure protection • System reliability and availability are specified as part of the non-functional requirements for the system.

System Reliability Specification • Hardware reliability • probability a hardware component fails • Software reliability • probability a software component will produce an incorrect output • software does not wear out • software can continue to operate after a bad result • Operator reliability • probability system user makes an error

Failure Probabilities • If there are two independent components in a system and the operation of the system depends on them both then P(S) = P(A) + P(B) • If the components are replicated then the probability of failure is P(S) = P(A)n meaning that all components fail at once

Functional Reliability Requirements • The system will check the all operator inputs to see that they fall within their required ranges. • The system will check all disks for bad blocks each time it is booted. • The system must be implemented in using a standard implementation of Ada.

Non-functional Reliability Specification • The required level of reliability must be expressed quantitatively. • Reliability is a dynamic system attribute. • Source code reliability specifications are meaningless (e.g. N faults/1000 LOC) • An appropriate metric should be chosen to specify the overall system reliability.

Hardware Reliability Metrics • Hardware metrics are not suitable for software since its metrics are based on notion of component failure • Software failures are often design failures • Often the system is available after the failure has occurred • Hardware components can wear out

Software Reliability Metrics • Reliability metrics are units of measure for system reliability • System reliability is measured by counting the number of operational failures and relating these to demands made on the system at the time of failure • A long-term measurement program is required to assess the reliability of critical systems

Reliability Metrics - part 1 • Probability of Failure on Demand (POFOD) • POFOD = 0.001 • For one in every 1000 requests the service fails per time unit • Rate of Fault Occurrence (ROCOF) • ROCOF = 0.02 • Two failures for each 100 operational time units of operation

Reliability Metrics - part 2 • Mean Time to Failure (MTTF) • average time between observed failures (aka MTBF) • Availability = MTBF / (MTBF+MTTR) • MTBF = Mean Time Between Failure • MTTR = Mean Time to Repair • Reliability = MTBF / (1+MTBF)

Time Units • Raw Execution Time • non-stop system • Calendar Time • If the system has regular usage patterns • Number of Transactions • demand type transaction systems

Availability • Measures the fraction of time system is really available for use • Takes repair and restart times into account • Relevant for non-stop continuously running systems (e.g. traffic signal)

Probability of Failure on Demand • Probability system will fail when a service request is made • Useful when requests are made on an intermittent or infrequent basis • Appropriate for protection systems service requests may be rare and consequences can be serious if service is not delivered • Relevant for many safety-critical systems with exception handlers

Rate of Fault Occurrence • Reflects rate of failure in the system • Useful when system has to process a large number of similar requests that are relatively frequent • Relevant for operating systems and transaction processing systems

Mean Time to Failure • Measures time between observable system failures • For stable systems MTTF = 1/ROCOF • Relevant for systems when individual transactions take lots of processing time (e.g. CAD or WP systems)

Failure Consequences - part 1 • Reliability does not take consequences into account • Transient faults have no real consequences but other faults might cause data loss or corruption • May be worthwhile to identify different classes of failure, and use different metrics for each

Failure Consequences - part 2 • When specifying reliability both the number of failures and the consequences of each matter • Failures with serious consequences are more damaging than those where repair and recovery is straightforward • In some cases, different reliability specifications may be defined for different failure types

Failure Classification • Transient - only occurs with certain inputs • Permanent - occurs on all inputs • Recoverable - system can recover without operator help • Unrecoverable - operator has to help • Non-corrupting - failure does not corrupt system state or data • Corrupting - system state or data are altered

Building Reliability Specification • For each sub-system analyze consequences of possible system failures • From system failure analysis partition failure into appropriate classes • For each class send out the appropriate reliability metric

Examples

Specification Validation • It is impossible to empirically validate high reliability specifications • No database corruption really means POFOD class < 1 in 200 million • If each transaction takes 1 second to verify, simulation of one day’s transactions takes 3.5 days

Statistical Reliability Testing • Test data used, needs to follow typical software usage patterns • Measuring numbers of errors needs to be based on errors of omission (failing to do the right thing) and errors of commission (doing the wrong thing)

Difficulties with Statistical Reliability Testing • Uncertainty when creating the operational profile • High cost of generating the operational profile • Statistical uncertainty problems when high reliabilities are specified

Safety Specification • Each safety specification should be specified separately • These requirements should be based on hazard and risk analysis • Safety requirements usually apply to the system as a whole rather than individual components • System safety is an an emergent system property

Safety Life Cycle - part 1 • Concept and scope definition • Hazard and risk analysis • Safety requirements specification • safety requirements derivation • safety requirements allocation • Planning and development • safety related systems development • external risk reduction facilities

Safety Life Cycle - part 2 • Deployment • safety validation • installation and commissioning • Operation and maintenance • System decommissioning

Safety Processes • Hazard and risk analysis • assess the hazards and risks associated with the system • Safety requirements specification • specify system safety requirements • Designation of safety-critical systems • identify sub-systems whose incorrect operation can compromise entire system safety • Safety validation • check overall system safety

Hazard Analysis Stages • Hazard identification • identify potential hazards that may arise • Risk analysis and hazard classification • assess risk associated with each hazard • Hazard decomposition • seek to discover potential root causes for each hazard • Risk reduction assessment • describe how each hazard is to be taken into account when system is designed

Fault-tree Analysis • Hazard analysis method that starts with an identified fault and works backwards to the cause of the fault • Can be used at all stages of hazard analysis • It is a top-down technique, that may be combined with a bottom-up hazard analysis techniques that start with system failures that lead to hazards

Fault-tree Analysis Steps • Identify hazard • Identify potential causes of hazards • Link combinations of alternative causes using “or” or “and” symbols as appropriate • Continue process until “root” causes are identified (result will be an and/or tree or a logic circuit) the causes are the “leaves”

How does it work? • What would a fault tree look like for a fault tree describing the causes for a hazard like “data deleted”?

Risk Assessment • Assess the hazard severity, hazard probability, and accident probability • Outcome of risk assessment is a statement of acceptability • Intolerable (can never occur) • ALARP (as low as possible given cost and schedule constraints) • Acceptable (consequences are acceptable and no extra cost should be incurred to reduce it further)

Risk Acceptability • Determined by human, social, and political considerations • In most societies, the boundaries between regions are pushed upwards with time (meaning risk becomes less acceptable) • Risk assessment is always subjective (what is acceptable to one person is ALARP to another)

Risk Reduction • System should be specified so that hazards do not arise or result in an accident • Hazard avoidance • system designed so hazard can never arise during normal operation • Hazard detection and removal • system designed so that hazards are detected and neutralized before an accident can occur • Damage limitation • system designed to minimized accident consequences

Security Specification • Similar to safety specification • not possible to specify quantitatively • usually stated in “system shall not” terms rather than “system shall” terms • Differences • no well-defined security life cycle yet • security deals with generic threats rather than system specific hazards

Security Specification Stages - part 1 • Asset identification and evaluation • data and programs identified with their level of protection • degree of protection depends on asset value • Threat analysis and risk assessment • security threats identified and risks associated with each is estimated • Threat assignment • identified threats are related to assets so that asset has a list of associated threats

Security Specification Stages - part 2 • Technology analysis • available security technologies and their applicability against the threats • Security requirements specification • where appropriate these will identify the security technologies that may be used to protect against different threats to the system

Software Reliability