Supplier Assessments NASA / GSFC

1. Supplier Assessments NASA / GSFC Goddard Space Flight Center - Greenbelt, MD Charlie Robinson / NCAS Quality Assurance and Risk Management Services, Inc. 28-30 October 2008

2. AGENDA • PROCESS • Preparing for an Assessment • RESULTS • Using Historical Data • LESSONS-LEARNED • Document Control • Contracts • Software • Internal Audits • SUMMARY

3. Preparing for an Assessment General Information AS9100 Assignments Assessment Plan Blank Assessment Card Assessors Input Template E-mails to Team & NASA Rep. Evaluation Form In-Briefing Attendance Sheet Team Conference Call The Assessment Team Takes Pre-Assessment Planning Seriously

4. Using Historical Data NCAS – 55% (18 out of 33) have Software problems (NOV 06 – SEP 08) SAC – 89% (32 out of 36) had Software problems (FEB 02 - JUN 05) 10 Suppliers have been assessed 2 to 4 times since FEB 2002

5. Average Small Medium Large University Using Historical Data (continued) HIGH R I S K Identify Requirements • Items produced & delivered before identifying the Environment & Contamination Requirements • Contractually required Stress Relieve Procedure not developed. • Unaware of DPAS rating • Wiring ‘Crimped” – Contract requires “Welding” LOW

6. Average Small Medium Large University Using Historical Data (continued) HIGH R I S K Software • No Configuration Management Process • Amendment to Instrument Coding Standard made without Peer-Review • Life-cycle metrics not distributed as required by contract • No Software Safety Program • Coding Standards in SDP not used. • A “Draft” Satellite Simulator Software is being used. LOW

7. Lessons-Learned Document Control 47 Non-compliances & 37 Observations • Internal Conflicts between procedures • Matrix of procedure inter-relationships is a way to avoid these conflicts • Industry Specifications not current • A consolidated listing of all industry specifications that are maintained • Procedures not maintained to reflect current practices • Supervisor responsibility & Internal Audits should provide direction and assurance that procedures are current

8. Lessons-Learned Contracts 46 Non-compliances & 45 Observations • There is not a reliable internal flow-down of requirements • Clean-room, DPAS, Calibration, QMS, Record Control, etc. • The review is flawed as suppliers agree to requirements they do not meet. • Suppliers often do not know their basic system and therefore are not able to identify those contractual requirements that they do not satisfy. • CDRL Items are either late or revised without updating the customer. • Assure that a single person is responsible for tracking the status of all CDRLs

9. Lessons-Learned Software 38 Non-compliances & 24 Observations • The software controls are not all encompassing • Control of software includes deliverable, design, test, and operational software. • There is not a compliant Software Safety Program (SSP) • Evaluate the SSP against NASA-Std-2100 • The individual who developed the software integration are developing the integration test. • The need for objective independence is compromised by having the individual who developed the software integration also develop the integration test.

10. Lessons-Learned Internal Audits 18 Non-compliances & 14 Observations • Contractual compliance and procedural support are not always part of the internal audit process. • Without auditing the contractual requirements the management will believe the Quality Management System is appropriate when in fact it does not support contractual requirements. • Internal Audits are behind schedule. • This is a discipline that is necessary to avoid requirement creep or slippage. As Dwight D Eisenhower said “The uninspected inevitably deteriorate.” • The internal auditors do not audit or are not qualified to audit special processes. • Assure that subject mater experts are used when auditing software, soldering, heat treat, plating, etc.

11. Summary • Individual commitment to a group effort - that is what makes a • team work, • company work, • society work, • civilization work. • Vince Lombardi