Quality Management ffice Edward A. Sierra (631) 344-4080

1. Brookhaven National Laboratory Causal Analysis Tree Training2009 DOE Facility Representative WorkshopLas Vegas, NevadaMay 13, 2009 Quality Management ffice Edward A. Sierra (631) 344-4080

2. Brookhaven National Laboratory (BNL) • Established in 1947 • 6 Nobel Prizes and Counting! Long Island –NY

5. Why Do a Causal Analysis? “It’s impossible to solve significant problems using the same level of knowledge that created them!”

7. START HERE A1 DESIGN/ENGINEERING PROBLEM A6 TRAINING DEFICIENCY A4 MANAGEMENT PROBLEM A5 COMMUNICATIONS LTA A3 HUMAN PERFORMANCE LTA A2 EQUIPMENT / MATERIAL PROBLEMS B1 SKILL BASED ERROR B2 RULE BASED ERROR B3 KNOWLEDGE BASED ERROR B4 WORK PRACTICES LTA B1 DESIGN INPUT LTA B2 DESIGN OUTPUT LTA B3 DESIGN/ DOCUMENTATION LTA B4 DESIGN/ INSTALLATION VERIFICATION LTA B5 OPERABILITY OF DESIGN/ ENVIRONMENT LTA B1 WRITTEN COMMUNICATIONS METHOD OF PRESENTATION LTA B2 WRITTEN COMMUNICATION CONTENT LTA B3 WRITTEN COMMUNICATION NOT USED B4 VERBAL COMMUNICATION LTA B1 MANAGEMENT METHODS LTA B2 RESOURCE MANAGEMENT LTA B3 WORK ORGANIZATION & PLANNING LTA B4 SUPERVISORY METHODS LTA B5 CHANGE MANAGEMENT LTA B1 NO TRAINING PROVIDED B2 TRAINING METHODS LTA B3 TRAINING MATERIAL LTA B1 CALIBRATION FOR INSTRUMENTS LTA B2 PERIODIC/ CORRECTIVE MAINTENANCE LTA B3 INSPECTION/ TESTING LTA B4 MATERIAL CONTROL LTA B5 PROCUREMENT CONTROL LTA B6 DEFECTIVE, FAILED OR CONTAMINATED Level A nodes are underlined. Level B nodes are in ALLCAPS. LTA = Less than adequate A7 OTHERPROBLEM B1 EXTERNAL PHENOMENA B2 RADIOLOGICAL/HAZARDOUS MATERIAL PROBLEM Causal Analysis Tree

8. Purpose of the CAT Tree • Provides a Taxonomy of Causes • Facilitates Trending & Analysis • Recurring Type ORPS Reports: • Recurring Occurrence on Cause Code A5 - Communications LTA – Sandia • Recurring Occurrence Reports Associated with Performance Analysis of Cause Code A4B5C04– Sandia

9. Definitions: • Incidents generally have more than one causal factor, but the following is true for all Causal Factors: Causal Factor – an event or condition that either caused the occurrence under investigation or contributed to the unwanted result. If it were not for this event or condition, the unwanted result would not have occurred or would have been less severe.

10. Example: BNL Causal Factors • LTA Organizational Change Management (technical personnel, funding, documentation) • Operator not authorized to open capacitor cabinet • No evidence of analysis to specify inspection interval for capacitors

11. Definitions: • Apparent Cause - The most probable cause(s) that explains why the event happened, that can reasonably be identified, that local or facility management has the control to fix, and for which effective recommendations for corrective action(s) to remedy the problem can be generated, if necessary. • Root Cause • Senior management • Prevent specific recurrence of the problem • Preclude occurrence of similar problems

12. Apparent Cause Process • Identify Causal Factors (first step) • Use the Causal Analysis Tree (CAT) to Identify Apparent Causes • DOE Guide 231.1-2, Occurrence Reporting Causal Analysis Guide, August 2003.

13. CAT Tree Application Branch Branch Branch A1 - Design A2 - Equipment A4 - Management A5 - Communications A6 - Training A7 - Other A3 - Human Performance Input: Causal Factors COUPLETS Output:Apparent Causes

14. A1 – Design / Engineering Problem

15. A1 – Design / Engineering Problem B1 DESIGN INPUT LTA C01 Design input cannot be met C02 Design input obsolete C03 Design input not correct C04 Necessary design input not available B2 DESIGN OUTPUT LTA C01 Design output scope LTA C02 Design output not clear C03 Design output not correct C04 Inconsistent design output C05 Design input not addressed in design output C06 Drawing, specification, or data error C07 Error in equipment or material selection C08 Errors not detectable C09 Errors not recoverable B3 DESIGN / DOCUMENTATION LTA C01 Design / documentation not complete C02 Design / documentation not up-to-date C03 Design / documentation not controlled B4 DESIGN / INSTALLATION VERIFICATION LTA C01 Independent review of design / documentation LTA C02 Testing of design / installation LTA C03 Independent inspection of design / installation LTA C04 Acceptance of design / installation LTA B5 OPERABILITY OF DESIGN / ENVIRONMENT LTA C01 Ergonomics LTA C02 Physical environment LTA C03 Natural environment LTA

16. Apparent Cause(s) A serious event / incident occurred when glare caused by improper overhead lighting prevented an operator from detecting that an important annunciator tile was illuminated. • B4- Design/Installation Verification LTA • B3 - Design Documentation LTA • B5 - Operability of Design / Environment LTA ü • CO2 - Physical environment LTA • CO3 - Natural environment LTA • CO1 -Ergonomics LTA ü

17. A2 – Equipment/Material Problem

18. A2 – Equip/Material Problem B4 MATERIAL CONTROL LTA C01 Material handling LTA C02 Material storage LTA C03 Material packaging LTA C04 Material shipping LTA C05 Shelf life exceeded C06 Unauthorized material substitution C07 Marking/labeling LTA B5 PROCUREMENT CONTROL LTA C01 Control of changes to procurement specifications/ purchase order LTA C02 Fabricated item did not meet requirements C03 Incorrect item received C04 Product acceptance requirements LTA B6 DEFECTIVE, FAILED OR CONTAMINATED C01 Defective or failed part C02 Defective or failed material C03 Defective weld, braze or soldering joint C04 End of life failure C05 Electrical or instrument noise C06 Contaminant B1 CALIBRATION FOR INSTRUMENTS LTA C01 Calibration LTA C02 Equipment found outside acceptance criteria B2 PERIODIC/ CORRECTIVE MAINTENANCE LTA C01 Preventive maintenance for equipment LTA C02 Predictive maintenance LTA C03 Corrective maintenance LTA C04 Equipment history LTA B3 INSPECTION/ TESTING LTA C01 Start-up testing LTA C02 Inspection/testing LTA C03 Post-maintenance/Post- modification testing LTA

19. A fire in the capacitor bank at BNL’s Accelerator Test Facility (ATF) resulted in a building evacuation and destruction of the bank. An investigation found no specified inspection interval for the capacitors. Apparent Cause(s) • B5- Procurement Control • B3 - Inspection/Testing LTA • B6 - Defective, Failed, or Contaminated ü • CO1 -Start-up testing LTA • CO3 -Post-maint. testing LTA • CO2 -Inspection/Testing LTA ü

20. A3 – Human Performance LTA

21. A3 – Human Performance LTA B3 KNOWLEDGE BASED ERROR C01 Attention was given to wrong issues C02 LTA conclusion based on sequencing of facts C03 Individual justified action by focusing on biased evidence C04 LTA review based on assumption that process will not change C05 Incorrect assumption that a correlation existed between two or more facts C06 Individual underestimated the problem by using past events as basis B4 WORK PRACTICES LTA C01 Individual’s capability to perform work LTA C02 Deliberate violation B1 SKILL BASED ERROR C01 Check of work was LTA C02 Step was omitted due to distraction C03 Incorrect performance due to mental lapse C04 Infrequently performed steps were performed incorrectly C05 Delay in time caused LTA actions C06 Wrong action selected based on similarity with other actions C07 Omission/ repeating of steps due to assumptions for completion B2 RULE BASED ERROR C01 Strong rule incorrectly chosen over other rules C02 Signs to stop were ignored & step performed incorrectly C03 Too much activity was occurring & error made in problem solving C04 Previous success in use of rule reinforced continued use of rule C05 Situation incorrectly id.ed or represented resulting in wrong rule used

22. Human Performance ModesRob Fisher, PresidentFisher Improvement Technologies, LLC802-233-0760Rob@fisherit.com This PERSON on this TASK at this TIME Key WordsError Rate SKILL-BASED -- stored patternsHABIT of pre-programmed actions.Familiar (> 50 times) 1 per Acting out of habit withoutNot Thinking1000 conscious thoughts. RULE-BASED -- rules accumulatedThere is a rule, via experience and training.andI KNOW there1 per Can be WRITTEN or VERBALis a rule100 KNOWLEDGE-BASED -- usingYou don’t know1 in 2 analytical processes and stored knowledge.what you don’t know to Has NOTHING to do with how smart1 in 10 you are! You are uncertain how to proceed.

23. RB KB SB While trying to pick up visual clues of familiar landmarks along the river bank & preoccupied with the danger of navigating in dense fog, the barge pilot had become disoriented. He had not been monitoring the radar closely & had lost track of his position while attempting to find a docking point. As a result the barge struck a bridge pylon causing damage to the bridge structure. ü • CO6 -Underestimates problem by using past events as basis • CO1 - Attention was given to wrong issues • CO3 -Individual justifies action by focusing on biased evidence ü

24. RB KB SB I completed every step in the procedure, signed it, then gave it to my supervisor. I thought that I had initialed all of the appropriate steps. The supervisor found a step that I failed to initial. ü ü • CO1 - Check of work is LTA • CO4 - Infrequently performed steps were performed incorrectly • CO6 - Wrong action selected based on similarity with other actions

25. RB WP SB The operator made an error while attaching two color-coded wires. He was color blind. There were no standards or requirements concerning color blindness associated with this job - even though discriminating between colors was necessary. A3B4C01

26. RB WP SB I knew that the area was a “safety-glasses required” area. I was only going to read a gauge and I didn’t have my safety glasses with me. I figured that by the time I went to get my glasses and got back I could have already taken the reading and gone to lunch. I didn’t think that it was a big deal but if I had known that the Safety Engineer was watching I would have gone to get them. Violation A3B4CO2

27. Is A3 Cause Coding Underreported? • 411 Causes Cited • 17% are A3 Causes • Only 48% of the Final “R” Reports cited an A3 Cause Code • Only 49% of those Reports cited a Cause Code Couplet

28. A4 – Management Problem

29. A4 – Management Problem The Largest Branch on the Tree • 5 ‘B’ Level Nodes B1 MANAGEMENT METHODS LTA • B2 RESOURCE MANAGEMENT LTA • B3 WORK ORGANIZATION & PLANNING LTA • B4 SUPERVISORY METHODS LTA • B5 CHANGE MANAGEMENT LTA

30. A tank overflowed because the liquid level instrumentation was out of calibration. Corrective measures had been identified for a previous overflow of the tank but had not been implemented when the second overflow occurred. Apparent Cause(s) ü • B1- Management Methods • B2- Resource Management • B4- Supervisory Methods • CO2 – Job performance stds not adequately defined • CO7 – Responsibility of personnel not well defined • CO8 –Corrective action response to a known or • repetitive problem was untimely ü

31. A5 – Communications LTA

32. A5 – Communications LTA B1 WRITTEN COMMUNICATIONS METHOD OF PRESENTATION LTA C01 Format deficiencies C02 Improper referencing or branching C03 Checklist LTA C04 Deficiencies in user aids (charts, etc.) C05 Recent changes not made apparent to user C06 Instruction step / information in wrong sequence C07 Unclear / complex wording or grammar B2 WRITTEN COMMUNICATION CONTENT LTA C01 Limit inaccuracies C02 Difficult to implement C03 Data / computations wrong / incomplete C04 Equipment identification LTA C05 Ambiguous instructions/ requirements C06 Typographical error C07 Facts wrong / requirements not correct C08 Incomplete / situation not covered C09 Wrong revision used B3 WRITTEN COMMUNICATION NOT USED C01 Lack of written communication C02 Not available or inconvenient for use B4 VERBAL COMMUNICATION LTA C01 Communication between work groups LTA C02 Shift communications LTA C03 Correct terminology not used C04 Verification / repeat back not used C05 Information sent but not understood C06 Suspected problems not communicated to supervision C07 No communication method available

33. Apparent Cause(s) A mechanic did not correctly replace a pump. The instructions simply stated “replace the pump.” Numerous actions were required to replace the pump, including an electrical lockout, which were not correctly performed. ü • B2- Written Communication Content LTA • B3- Written Communication Not Used • B4- Verbal Communication LTA • CO2 - Difficult to implement • CO8 - Incomplete / situation not covered • CO9 - Wrong revision used ü

34. A6 – Training Deficiency

35. A6 – Training Deficiency • B1 NO TRAINING PROVIDED • C01 Decision not to train • C02 Training requirements not identified • C03 Work incorrectly considered “skill of the craft” • B2 TRAINING METHODS LTA • C01 Practice or hands-on experience LTA • C02 Testing LTA • C03 Refresher training LTA • C04 Inadequate presentation • B3 TRAINING MATERIAL LTA • C01 Training objectives LTA • C02 Inadequate content • C03 Training on new work methods LTA • C04 Performance standards LTA

36. Apparent Cause(s) An operator made a mistake weighing material because of incorrect use of a scale. He had received instruction on the use of the scale but had not been tested on his ability to actually use the scale. • B1- No Training Provided • B2- Training Methods LTA • B3- Training Content LTA ü • CO1 - Practice or hands-on experience LTA • CO2 - Testing LTA • CO3 - Refresher training LTA ü

37. A7 – Other Problem

38. A7 – Other Problem B1 EXTERNAL PHENOMENA C01 Weather or ambient conditions LTA C02 Power failure or transient C03 External fire or explosion C04 Other natural phenomena LTA B2 RADIOLOGICAL / HAZARDOUS MATERIAL PROBLEM C01 Legacy contamination C02 Source unknown

39. Traces of PCBs were found during routine environmental survey. The location had been previously used as a storage site for transformers. The transformers had leaked. The leakage was unknown / undiscovered at the time the transformers were removed. Apparent Cause(s) • B1- External Phenomena • B2- Radiological / Hazardous Material • Problem ü ü • CO1 - Legacy contamination • CO2 - Source unknown • CO3 - External fire or explosion

41. What are the Preferred Root Cause Analysis Methods in the DOE Complex?

42. Corrective Action Hierarchy

43. In Summary: • Open, honest reporting is the crucial first step • Event/issues do not speak for themselves • Analysis is essential • Organizational response is more significant than the events/issues themselves