Operators Versus Automation

1. Operators Versus Automation Andrew Rae

2. A couple of warm-up questions • Would you voluntarily accept a device added to your car that detected the local speed limit and prevented you exceeding that limit in all circumstances? • Do you believe that trains should be fitted with vigilance devices which apply the brakes if the driver isn’t aware/awake/conscious? In different circumstances, people are willing to place different levels of trust in automation.

3. Chernobyl 4 versus USS Thresher • Chernobyl: Automatic Shut-Down System was disabled to allow special testing • RESULT: Explosion and fire • CONCLUSION: Operators should not be allowed to turn off vital safety systems • USS Thresher: Automatic Shut-Down System removed power at a time when propulsion power was the only thing that could save the submarine • RESULT: Loss of submarine with all crew • CONCLUSION: Operators should be allowed to override vital safety systems

4. So who are we going to trust? • For road vehicles: should speed limits be automatically enforced, or should drivers have ultimate control? • For military engines: should thermal limits be permitted to be overridden using “battle shorts”? • For aircraft: should “alpha” (flight envelope) protection be strict, or soft? • For railways: should a signal interlocking be flexible in order to move trains out of a dangerous situation?

5. The question of ultimate trust In a given situation, should automation be designed to prevent system states which the designers judge to be dangerous, or should the interface provide facility for the operator to execute any control at any time?

6. What conventional hazard analysis has to say • Hazard: Operator error in performing manual taskMitigation: Automation • Hazard: Automation ErrorMitigation: Operator in the loop • Hazard: Inappropriate operator interventionMitigation: Interlock • Hazard: Unforeseen circumstancesMitigation: Interlock override • Hazard: Inappropriate use of overrideMitigation: ???

7. Conventional Hazard Analysis (continued) • Conventional hazard analysis does not cope well with situations where: • Mitigations / Safety Requirements are in conflict • Hazards cannot be quantified and thus directly compared • The risk of human error can be quantified (but not very accurately without application-specific measurements) • The risk of automation error is very difficult to quantify • The risk of unforeseen circumstances (unknown-unknowns) cannot be quantified

8. The Fitts’ List Approach Humans are good at: • Ability to perceive patterns • Ability to improvise and use flexible procedure • Ability to reason inductively • Ability to exercise judgment

9. Fitts’ List Continued Machines are good at: • Ability to respond quickly to control signals • Ability to perform repetitive tasks • Ability to do many things at once • Ability to perform complex calculations quickly and accurately

10. The “Fitts’ List” approach isn’t perfect • In reality, humans and machines co-operate to perform tasks • The performance of humans and machines is shaped by environmental factors, and their interaction with each other

11. A “performance shaping” approach • In a given situation, the performance of humans and machines will be moderated by: • Whether the agent is innately suited to the task • The internal state of the agent • The ability of the agent to process inputs • The ability of the agent to decide upon outputs • The ability of the agent to execute outputs • We call each of these considerations “performance shaping factors”

12. Performance shaping (continued) • Performance shaping is a common concept in human-centered design • The idea that performance shaping factors can be applied to machines is less frequently addressed

13. Performance Shaping Factors for Humans • Task demands and characteristics • Instructions and procedures • Displays and controls • Stresses • Individual skill and capacity • Individual condition • Socio-technical factors

14. Assumptions RULE SET Outputs INTERNAL MODEL Inputs Performance shaping factors for machines

15. Performance shaping factors for machines • Incorrect Internal Model • Incorrect assumptions • Incorrect or insufficient inputs • Incorrectly Applied Rule Set • Incorrectly Specified Rule Set • Error in Implementing Rule Set • Error in Transmitting Outputs

16. Application – Automatic Train Operation • Train driving is a well-understood task • Most often performed by humans • Sometimes (e.g. Docklands Light Rail, Copenhagen Metro) performed by computer • Sometimes performed by computer with a driver in the cab “just in case” • A good candidate for testing our methodology

17. Performance Shaping Factors for Train Drivers Task demands are moderate Instructions and procedures are generally clear and well understood Displays and controls can be made simple and intuitive Stresses – job has some pressures, and also some boredom Driver condition can be a factor – remember Waterfall Performance Shaping Factors for Train Computers Task is well understood Inputs are generally sufficient – note important exception that drivers can see out of the windscreen and computers can’t Safety-conscious industry / development environment Mechanical reliability is high Application – Automatic Train Operation

18. Automatic Train Operation - Conclusion • So should humans or computers be used to drive trains? • Some supplementary points of interest • Adequate levels of safety and performance must be delivered in a cost effective way – sometimes performance and cost will tip the balance towards a particular solution • As automation increases, the drivers’ performance may decrease • Lower situation awareness • Greater boredom • Greater stress when things do go wrong • There comes a point when things are so automated that the driver lacks capacity to be an effective safeguard

19. Conclusion • It is important to make a conscious decision who (or what) is responsible for making safety decisions • Ultimately, humans will be responsible • Either as operators, or as designers/maintainers of the system • The nature of the task, and the task environment, dictate whether the operator or the automation is best suited to making safety decisions • A sound argument can be formulated for assigning the safety decision, based on performance shaping factors