Conference on Quality in the Space and Defense Industries - March 3/4, 2008 -

1. Conference on Quality in the Space and Defense Industries- March 3/4, 2008 - Lessons Learned from: The Big Dig Brian Hughitt NASA Headquarters Office of Safety and Mission Assurance

10. Bumper to bumper 6 – 8 hours per day

11. Plus, deteriorating structure

12. Logan Airport The solution to the overhead highway…. Downtown Downtown Bury it underground South Boston

14. The Big Dig • 7 1/2 milecorridor • 161 lane miles • 5 miles of tunnel • 6 interchanges • 200 bridges • 16 million cubic yards of dirt • 541,000 truckloads (4,612 miles of trucks lined up end to end) • 15stadiums filled to the rim with dirt • 3.8 million cubic yards of concrete (enough for a sidewalk from Boston to San Francisco and back 3 times)

15. Accident Synopsis At 11:00 pm on July 10, 2006 a 1991 Buick passenger car occupied by a 46 year-old male driver and his 38 year-old wife was traveling eastbound in the I-90 connector tunnel in Boston, MA, en route to Logan International Airport. As the car approached the end of the connector tunnel, a section of the tunnel’s suspended concrete ceiling (26 tons) detached from the tunnel roof and fell onto the vehicle, crushing its right side. The driver’s wife, occupying the right-front seat, was fatally injured. The driver escaped with minor injuries.

18. National Transportation Safety BoardAccident Report Proximate Cause Use of an epoxy anchor adhesive with poor creep resistance Post accident testing revealed that Fast Set epoxy had been supplied and that, while both Fast Set and Standard Set epoxy performed similarly in short term tests, they differed dramatically under long term load. Creep Epoxy is a polymer and its stiffness is time and temperature dependent. If a load is applied suddenly, the epoxy responds like a hard solid. But if that load is then held constant, the molecules within the polymer may begin to rearrange and slide past one another, causing the epoxy to gradually deform. As the deformation increases, it becomes irreversible.

19. Creep: The Unknown Known

20. National Transportation Safety BoardAccident Report Contributing Causal Factors • Design • System Safety • Project Management • Procurement • Supplier Documentation • Quality Assurance • Industry Standards • Governmental Oversight • Governmental Regulations

21. Defenses are never perfect -When Events Lines Up, the Consequences Can Be Devastating Requirements Hazard Design Quality Assurance Manufacturing Test Operations Mishap Adapted from : James Reason, Managing the Risks of Organizational Accidents, 1997, p. 12

22. Design Output Design Input Process Analysis Configuration Management Process Control Supplier Qualification Common Quality Threads Process Adequacy Documents Availability Control Verification Process Documentation Requirements Production & Service Design & Development Planning Validation Procedures Records Review Records Process Development Production Realization Internal Audits Process Measurement Calibration Requirement Flow Down Process Measurement, Analysis, Improvement And Control Maintenance Metrology Purchasing Corrective Action Process Improvement Subtier Control Preventative Action Supplier’s Ability To Provide Suitable Products or Services Quality Manual Design Adequacy Fabrication Control Maintenance Control Documentation Hardware Config. Mgt. Process Test Inspection & Evaluation Integrity Quality System Product Acceptability Commitment Planning Policy Objectives Skills Cleanliness Business System(s) Maturity Capability Work Environment Experience Software Human Resources Materials Lighting Material Control Training Authority Suitability Review Resource System(s) Responsibility Non-Con Article Control Customer Focus Communication Workspace Effectiveness Availability Process Equipment Resources Infrastructure Support Services System Product

23. NTSB Accident Report- Contributing Causal Factors - Design • Design specifications did not incorporate a provision for attaching a suspended ceiling, even though it was known that one would be needed. Consequently, the tunnel had no embedded ceiling supports. • The design consultant repeatedly recommended undercut anchors vs the adhesive anchors ultimately chosen. • In order to save costs/time, a change was made to use heavy precast concrete panels in lieu of custom-engineered laminated lightweight concrete panels. “The July 10, 2006, accident was a sudden, violent event, but the circumstances leading up to it developed over a period of more than 20 years, beginning with the design of the Ted Williams tunnel in the late 1980s.”

26. Epoxy Secured Bolts

27. Undercut Anchor System Stud is held in mechanically Anchor Sleeve is inserted and it spreads Hole is drilled with an undercut

28. NTSB Accident Report - Contributing Causal Factors - System Safety • No redundancy- the majority of U.S. tunnels have continuous ceiling panels that extend into the concrete wall. If the hangers fail, the ceiling is self-supported. • Incomplete Failure Modes and Effects Analysis- Creep not identified as a potential failure mode. Consequently, risk mitigation measures were not implemented.

29. Contributing Causal Factors (cont)Industry Standards ICC AC58: Either a design safety factor of 5.33 or a 120-day creep test is required for Fast Set epoxy. “Given that the ability to sustain a load over a period of time is a typical requirement for almost any type of fastener, the Safety Board is concerned that the ICC has allowed creep testing of epoxy adhesives to be optional. A design engineer should be provided with all of the relevant information about a product before it is used in a safety critical application.”

30. Consequently… - To support product qualification, the supplier provided an Evaluation Report (ER) which included bond strength tables specifying a safety factor of 5.33 for Fast Set epoxy- not the results of creep tests*. • Nothing in the ER tables or footnotes indicated that the Fast Set epoxy should be limited to use with short-term loads regardless of the safety factor employed (a restriction for use in short-term load applications was contained elsewhere in the ER, but could easily have been overlooked). [* The Safety Board learned during the investigation that Fast Set epoxy had been tested for creep performance in 1995 and 1996 and had failed to meet the standard]

31. Contributing Causal Factors (cont)Product Qualification No documentation was provided by the supplier which specified which epoxy formulation was supplied, and neither the contractor nor the design consultant questioned which epoxy was used. Both assumed that the epoxy provided by the supplier was suitable. The Safety Board found fault with the construction contractor and the design consultant for not adequately reviewing the product qualification documentation. But even more….

32. Contributing Causal Factors (cont) “The supplier should have made a clear distinction in all its literature between the relative capabilities of its Standard and Fast Set formulations. It did not do so, even though before the epoxy was provided, the company had conclusive evidence that its Fast Set epoxy was susceptible to creep.” “The Safety Board concludes that the information provided by the supplier was inadequate and misleading.”

33. The Gift… On September 9, 1999, a construction contractor employee installing ventilation ductwork over the tunnel ceiling noticed that several of the anchors had begun to pull out. …and the Smoking Gun On November 12, 1999, a proof load test was performed on one of the anchors that had shown significant displacement (9/16”). The engineer noted that “the bolt held for a few seconds, then began to pull out with almost no resistance”.

35. The Supplier’s Response When the supplier was called to examine the anchor displacements, they seemed surprised that the anchors that had been successfully proof tested only a few months before could be failing. Installation problems (e.g., excessive preload) were postulated as the cause. No evidence was found that the supplier took any follow-up action after the examination. - No further testing - No further research “At least some supplier officials were aware that their Fast Set epoxy was subject to creep, but this information was apparently not considered or was not known by the representatives who evaluated the failed anchors. Even if the information about poor creep resistance was not common knowledge, a reasonable amount of research would likely have revealed it. The Safety Board would have expected the supplier of a safety critical component to have been more proactive in determining why its product was failing.”

36. The Builder, Design Agent, and Project Manager Reply Increased proof load testing

37. The root cause for the hanger displacement was never clearly identified… …and surveillance monitoring inspections were never implemented “The project managers apparently accepted at face value the catalog load capacities provided by the supplier, and performed no independent testing to verify that … the anchors would perform similarly in this particular application.” NTSB Accident Report

38. LbF 20,000# Calculated design service load (2,600 Lb-Force) 15,000# 10,000# 5,000# 2,600 # 0 Design Service Load Post Installation Proof Test Finite Element Analysis

39. LbF 20,000# After each bolt was installed, a proof test was conducted at 25% higher than design service load (3,250 Lb.- Force) 15,000# 10,000# 5,000# 3,250 # 2,600 # 0 Design Service Load Post Installation Proof Test Finite Element Analysis

40. LbF 20,000# Later, after slippage was noted, bolts were proof tested to the maximum allowable load (6,350 Lb-Force) 15,000# 10,000# 6,350 # 5,000# 3,250 # 2,600 # 0 Design Service Load Post Installation Proof Test Finite Element Analysis

41. A finite element analyses determined that the load would be between 2,371 and 2,823 lb force LbF 15,000# 10,000# 6,350 # 5,000# 3,250 # 2,823 # 2,600 # 2,371 # 0 Design Service Load Post Installation Proof Test Finite Element Analysis

42. "You’ve noted the key piece of information that is missing. That is the cause of the anchor failure and how the repair procedure will overcome that… We are not trying to hold up construction, we are trying to make a determination that the installation is safe…” Design Manager e-mail concerning response to Deficiency Report Where is the voice of Quality? “Glaringly absent from the Deficiency Report is any explanation why the anchors failed and what steps are proposed to ensure that this problem does not reoccur.” Structural Engineer e-mail reply

43. The Gift (part 2) On December 17, 2001, a quality control inspector submitted a Noncompliance Report which stated: “Several anchors appear to be pulling away from the concrete. The subject anchors were previously tested to the revised value of 6350 lbs, all of which passed…. Reason for failure is unknown.” “At this point, it should have been obvious... that the remedy that had been developed in response to the anchor displacement in the HOV tunnel in 1999 had not been effective, as anchors that had passed proof testing at higher values were still displacing. This was another opportunity to …inspect all the installed anchors to determine the extent and, more importantly, the cause of the anchor displacement. Instead, the companies apparently considered the continuing failures as isolated instances and took no action to address the problem in a systemic way.” NTSB Accident Report

45. Contributing Causal Factors (cont)Standards and Protocols for the Testing of Adhesive Anchors The Safety Board concluded that proof test procedures are inadequate for adhesive anchors and recommended that, building on current test standards, the FHWA work jointly with industry to develop standards for the testing of adhesive anchors used in sustained tensile-load applications.[ The ICBO AC58 optional creep test was found to be overly simplistic/limited ] ASTM D2990-01, Standard Test Method for Tensile, Compressive, and Flexural Creep and Creep Rupture of Plastics Since the properties of viscoelastic materials are dependent on time…an instantaneous test result cannot be expected to show how a material will behave when subjected to stress or deformation for an extended period of time.

46. Contributing Causal Factors (cont)Lack of Awareness “This accident investigation revealed a striking lack of awareness among designers, contractors, managers, and overseers about the nature and performance of polymer adhesives, even as those adhesives were being approved for use an applications where a failure would present an immediate threat to the public. Even after being presented with evidence of anchor creep, project managers and overseers failed to recognize the inherent weakness in the epoxy adhesive – a weakness that could not be overcome even with the best installation practices or the most rigorous short-term proof testing.”

47. Cognitive Dissonance • A psychological term describing the uncomfortable tension that may result from having two conflicting thoughts at the same time, or from engaging in behavior that conflicts with one's beliefs, or from experiencing apparently conflicting phenomena. • In simple terms, it can be the filtering of information that conflicts with what you already believe, in an effort to ignore that information and reinforce your beliefs. • Social psychologistLeon Festinger first proposed the theory in 1957 after the publication of his book When Prophecy Fails, observing the counterintuitive belief persistence of members of a UFO doomsday cult and their increased proselytization after the leader's prophecy failed. Wikipedia

48. Contributing Causal Factors (cont)Tunnel Inspections • In November, 2003, the Design Agent published Inspection Manual for Tunnels and Boat Structures. The manual required each ceiling hanger component to be inspected visually or by NDT. • From the time the tunnel was opened to traffic until the day of the fatal accident, no tunnel inspections were performed. • Post accident inspection of the suspended ceiling displayed large numbers of anchors that had become displaced (~25%), and that the displacement was so obvious that even a cursory examination would have revealed that structural integrity was threatened.

50. Tunnel Inspections (cont) • “Investigators asked MTA officials why the inspection manual was not used... The officials stated that the inspection manual was not used because • a tunnel inspection database needed to be developed, • the inspection manual was being reviewed by the FHWA and the MTA, and • MTA personnel needed time to be trained on the manual. ” • NTSB Accident Report