Commercial Space Vehicles Lessons Learned Needs Workshop “NASA Implementation of a Formal Lessons Learned Process”

1. Commercial Space Vehicles Lessons Learned Needs Workshop“NASA Implementation of a Formal Lessons Learned Process” David Oberhettinger Office of the Chief Engineer Jet Propulsion Laboratory, California Institute of Technology September 18, 2006

2. Technical Excellence/Mission Success • Why is NASA placing a renewed emphasis on lessons learned? • Repeated mistakes, or violation of known best practices, pose a risk that is potentially avoidable • “Progress, far from consisting of change, depends on retentiveness... Those who cannot remember the past are condemned to repeat it.” -George Santayana • “An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them.” -Werner Karl Heisenberg • "Why - I learnt what one ought not to do, and that is always something." - The Duke of Wellington describing the failed Dutch campaign of 1794 • Diaz Report assessed the agency-wide applicability of the CAIB report • “… require that everyone understand their responsibilities and are given the authority to perform their jobs, with the accountability for their individual and program’s successes and failures, including lessons learned.” (Diaz Rpt, Page 10) • “The CAIB concluded NASA ‘has not demonstrated the characteristics of a learning organization’ after investigators observed mistakes being repeated and lessons from the past apparently being relearned.” (Diaz Rpt, Page 11)

3. A Formal Lessons Learned Process • NASA has maintained a lessons learned system since 1992 • NASA lessons learned repository has 1500 lessons, an advanced search capability, and is accessed 2500 times per month • The new NASA Engineering Network (NEN) replaces the former repository, and links to additional engineering information (project documentation, guidance from “experts”) related to each lesson learned • Public access Lessons Learned Site: http://llis.nasa.gov • JPL has had a formal lessons learned process since 1984 • JPL Lessons Learned Committee that meets weekly, and recent emphasis on lessons learned “infusion” into rules for spacecraft design, test, and mission operations • NASA NPD 7120.6, The NASA Lessons Learned Process, issued in March 2005 • Implements a formal system, based on the JPL model, to ensure important lessons are captured and that they are used

4. LL Identification & Documentation • JPL criteria for a valid lesson learned: • Candidate must (1) effect mission success, (2) be relevant to JPL projects, (3) not duplicate a previously published lesson learned • Sources: MIB reports, FRACAS, project-maintained list, rumor mill • Lessons Learned Committee role • Members report recent events, identify and prioritize potential topics, validate each candidate lesson, review the text, verify the facts, approve the lesson, disseminate to member orgs (JPL tech divisions) • Lesson learned structure and format • Abstract, Event Description, Lesson Learned summary, (implementable) Recommendation(s), references, metadata • Example: “Control Blow-By From Pyrotechnic Devices” Information File Pyro Video

5. Recent JPL Lessons Learned “Managing Mars Rover/Mars Orbiter Relay Link Prediction Variability“ The difference between the predicted versus achieved data volume returned by the Mars Exploration Rover relay link impacted the daily planning of rover driving and science data collection. This problem can be alleviated by refining the operations and science data return planning process. “Mitigating the Risk of ‘Single String’ Spacecraft Architecture” Mars Exploration Rover met and exceeded mission requirements, despite a largely ‘single string’ spacecraft architecture, due to effective risk management, ample fault tolerance, flight system flexibility, access to experienced designers, ample stress testing, use of proven designs, and a rigorous approach to fault protection. “The Pitfalls of ‘Engineering-by-Presentation’” The increased use of e-mails and slides instead of formal engineering documentation may be inhibiting the ability of NASA projects to reference the basis for technical decisions and to validate or verify engineering designs. “Genesis Sample Return Mishap” The Genesis sample return mishap was attributed to a design error in which the gravity switches that activate the parachute deployment sequence were phased (oriented) incorrectly so that their mechanisms could not detect the atmospheric entry. “If You Don’t Understand an Environment, Provide ‘Well-Margined’ Capabilities to Encompass the Worst Case” Mars Exploration Rover designers responded to a high level of uncertainty regarding Martian winds that could damage the lander by providing a set of small, sideways-pointing rockets and adding a capability to directly sense horizontal motion. Mate/Demate, Verify, and Document Connectors One-at-a-Time An integration and test failure was traced to the difficulty of confirming, verifying, and documenting that a flight connector had been left unmated in a crowded and physically constrained assembly.

6. NASA Best Practices for Design & Test • 180 docs that each define a NASA-consensus engineering practice: • Example Environmental best practices (total of 10): - “Micrometeoroid Protection” - “Monitoring Spacecraft Exposure to Magnetic Fields” - “Optical Fiber Cable Terminations and Procedures” • Example Engineering Design best practices/guidelines (total of 88): - “EEE Parts Derating” - “Electrical Grounding Practices for Aerospace Hardware” - “Thermal Design Practices for Electronic Assemblies” - “Contamination Control of Space Optical Systems” - “Space Radiation Effects on Electronic Components” - “Material Selection Practices” - “Selection of Electric Motors for Aerospace Applications” - “Vehicle Integration/Tolerance Build-up Practices” - “Active Redundancy” - “Pre-Ship Review”

7. NASA Best Practices (Continued) • Example Analysis best practices/guidelines (total of 23): - “Problem/Failure Report Independent Review/Approval” - “Part Electrical Stress Analysis” - “Redundancy Switching Analysis” - “Thick Dielectric/Internal Electrostatic Discharge (IESD)” - “Failure Modes, Effects, and Criticality Analysis (FMECA)” • Example Test best practices/guidelines (total of 48): - “Pyrotechnic Shock Testing” - “Radiated Susceptibility System Verification” - “Sine-Burst Load Test” - “Heat Sinks for Parts Operated in Vacuum” - “RF Breakdown Characterization” - “Voltage/Temperature Margin Testing” - “Power System Corona Testing” - “Selection of Spacecraft Materials and Supporting Vacuum Outgassing Data” - “Spacecraft Deployed Appendage Test Guidelines”

8. Lessons Learned “Infusion” • Iterative review of 1500+ lessons learned by projects is difficult, so infuse the recommendations into NASA procedures and training • Use of JPL engineering “process owners” and closed-loop corrective action system to assure recommendations are infused • Focusing on infusion into key rules for design and for project management • Other NASA Centers have primary standards equivalent to JPL’s Design Principles and Flight Project Practices documents that distill decades of mission experience • The NASA Technical Standards Program is the repository for NASA standards and guidelines, plus subscriptions to industry standards used by NASA. http://standards.nasa.gov/ • JPL Standards & Specs Access Site for Contractors and Bidders • Other resources: technical documentation in NASA Center flight project libraries, memos, and procedures repositories such as JPLRules!