FOR0383 Software Quality Assurance

1. FOR0383 Software Quality Assurance Lecture 3 London Ambulance System (LAS) 26 & 27 October and November 4 1992 Dr Andy Brooks

2. Facts about LAS ~ 1992 • LAS was the largest in the world, handling 1,300-1,600 ‘999 calls’ daily. • £69.7 million budget • 2,700 staff • 305 A&E ambulances (Accident and Emergency) • 445 PTS ambulances (Patient Transport Service) • 1 helicopter • 0.5 million A&E journeys per year Dr Andy Brooks

3. Deficiencies of manual system • Identification of exact location • “About half-way along London Road…” • Physical movement of paper • Maintaining proper vehicle status • Time consuming use of voice • “Can you repeat that address please…” • Identifying duplicated calls • “This sounds like a report of the same accident…” • Dealing with call backs • “It has been 25 minutes, the ambulance is not here yet…” • Identifying special incidents • “Could this be a tanker chemical spill?...” Dr Andy Brooks

4. With Computer Aided Dispatch (CAD) • a gazetteer provided telephone box identification • no more paper • timely updates of resource availability • intelligent help to identify duplicates • direct mobilisation of an ambulance on completion of an emergency call Dr Andy Brooks

5. Some Inquiry Conclusions • software was not complete and not fully tested and the backup was not tested • there were outstanding data transmission problems to/from MDTs (Mobile Data Terminals) • there was scepticism over AVLS accuracy (Automatic Vehicle Location System) • the ambulance crews had no confidence in the system and had not been fully trained “Train train and train again”Ian Tighe, 999 rescue, The Computer Bulletin October 1997 Dr Andy Brooks

6. Some Inquiry Conclusions • CAC (Central Ambulance Control) staff were placed in unfamiliar positions in the control room and could not easily work together to solve problems • the system relied on near perfect information of vehicle location and crew/vehicle status • no attempt was made to deal with the effects of inaccurate or incomplete data which led to an increase in the number of exception messages Dr Andy Brooks

7. Poor interface between crews, MDTs, and the system. • the system failed to capture all of the data • ambulance crews sometimes failed to press the correct status button due to the pressure of dealing with certain incidents • there were radio black spots • ambulance crews sometimes failed to press status buttons because of frustration • there were radio communication bottlenecks during busy periods Dr Andy Brooks

8. Poor interface between crews, MDTs, and the system. • missing or swapped callsigns • there were faults in handshaking routines: MDTs and the system showed different status! • sometimes ambulance crews intentionally did not press the correct status button or pressed buttons in an incorrect order (frustration) • ambulance crews sometimes took a different vehicle to the one they had logged onto • incorrect or missing vehicle locations • too few call takers Dr Andy Brooks

9. Unreliability, slowness and operator interface • the system fell over and screens locked • staff were advise to re-boot • there was a failure to identify all duplicate calls • exception messages were not prioritised • messages scrolled off the top of screens! • the software made resource allocation errors • there were slow response times for certain screen based activities Dr Andy Brooks

10. 26 and 27 October 1992 • the system went live • there were no paper records • there was no manual back up • the launch was across all of London • resource allocators were separated from radio operators and exception rectifiers • system proposed resource allocations were used Dr Andy Brooks

11. 26 and 27 October 1992 • it was extremely difficult for staff to intervene and correct problems • the system rapidly knew the correct location and status of fewer and fewer vehicles • multiple vehicles were sent to the same incident or not the closest vehicle was sent • the number of exception messages rapidly increased so that staff could not clear the queue • each effect quickly reinforced the others • see the Cause/Effect diagram in the Inquiry Report Dr Andy Brooks

12. Ambulance crew frustration • crew frustration may have led crews not to press the status buttons in the correct order • crew frustration may have led crews taking a different vehicle to the one allocated to them • crew frustration may have led to a different vehicle and crew responding to an incident Dr Andy Brooks

13. Crash on November 4 • after 27 October, it was decided to revert to a semi-manual operation • computer-based call taking and the gazetteer were for the most part reliable • radio voice channels were available to resolve any mobilisation misunderstandings • additional call taking staff were allocated to each shift • but shortly after 2am on November 4th, the system slowed significantly and then locked up • attempts to reboot the system failed • a decision was made to revert to a manual system with voice mobilisations Dr Andy Brooks

14. What went wrong on November 4? • a minor programming error left a piece of code which used up a small amount of memory within the file server every time a vehicle was mobilised • after 3 weeks all the available memory on the file server had been used! • normal testing would probably not have caught this particular error • the fallback to a second server was never implemented for the semi-manual operation! Dr Andy Brooks

15. Response times 26/27 October(The ORCON specified response time from taking the call and arriving at the scene is 14 minutes.) Dr Andy Brooks

16. Calls and average ring times 26 October(Average ring times peak at 10 minutes!) Dr Andy Brooks

17. Total A&E Patients Carried, October(October 26/27 were normal in terms of demands on LAS services. Excessive demands was not responsible for the poor response times.) Dr Andy Brooks

18. Calls recorded by call logger(On October 26/27 the number of calls were within the range within which it can be predicted 95% of calls will fall.) Dr Andy Brooks