Jim Schultz 28-October-2008

1. Constellation Program Surveillance Strategy Jim Schultz 28-October-2008

2. CxP Surveillance Strategy • Traditional Surveillance Strategies consist primarily of Inspections and Audits. • Inspections and audits are necessary, however they are “after-the-fact” and should not be the sole surveillance approach.

3. CxP Surveillance Strategy (cont.) • Constellation Projects are required to develop and document a surveillance strategy concurred with by the Constellation Program QA manager. • The level of risk and the impact of failure are major determinants in defining the type of surveillance required. If the impact of failure is minor and the risk low, only a minimal amount of insight driven surveillance is needed. Conversely, if the failure impact could be significant and the risk high, more extensive surveillance (including possible oversight) is necessary. • The focus should be on prevention rather than detection, i.e., emphasizing controlled processes and methods of operation, as opposed to relying solely upon inspection to identify problems.

4. CxP Surveillance Strategy (cont.) • Identify specific risks and surveillance methods used to help track and mitigate these risks. • Examples of Surveillance Methods • Metrics reviews (e.g., process escapes, audits scheduled, audits completed, audit findings, nonconformances, corrective actions taken, cost of quality, delivery performance, quality performance, process performance, cycle time, labor performance (utilization, efficiency, productivity), lost workdays/recordable incidents, employee surveys, etc.) • Document/quality record reviews • Audits

5. CxP Surveillance Strategy (cont.) • Examples of Surveillance Methods (cont.) • Management information (e.g., management review minutes (w/ focus on internal audit and corrective action effectiveness), development projects status reports, board/meeting minutes, Contractor weekly activity reports, NASA/Constellation Project Contractor Management tag-ups, Special Issues meetings, Flight Readiness Review chart, Data Requirements Documents (DRD), etc.) • Performance reviews (e.g., witnessing the performance of the task being surveyed such as flight controller simulation and real-time operations, process audits, Flight Readiness Reviews (FRR), real-time inline product inspection, tests, etc.)

6. CxP Surveillance Strategy (cont.) • Examples of Surveillance Methods (cont.) • Quality tool implementation (e.g., Design of Experiments (DOE), Multiple Environment Overstress Testing (MEOST), Environmental Stress Screening (ESS), Quality Functional Deployment, Total Productivity Maintenance (TPM), Benchmarking, Self-Check Systems, Next Operation as Customer, Supply Management, Total Value Engineering, Cycle Time Reduction, Lean Six Sigma, Fault Tree Analysis, Process Failure Modes and Effects Analysis (PFMEA), Statistical Process Control (SPC), Probability Risk Analysis (PRA), PRACA, FMEA/CIL, etc.) • Customer feedback - formal and informal (e.g., board representatives, Program Offices, all levels of contractmanagement, customer surveys, field data, etc.)

7. CxP Surveillance Strategy (cont.) • Examples of Surveillance Methods (cont.) • Self-Check Systems • Provide sensors (electrical, mechanical, or visual) that warn an operator that a mistake has been or is about to be made. Keep the solutions simple. • In-Process Monitoring – address condition before it becomes a problem (e.g., Statistical Process Control (SPC), Chemical Fingerprinting, cycle time, precursor analysis, etc.)

8. CxP Surveillance Strategy (cont.) • Examples of Surveillance Methods (cont.) • Supplier Performance • Quality performance • Delivery performance • Total recordable incident rate (TRIR) • Lost work incident rate (LWIR) • Earnings Before Interest, Taxes, Depreciation, and Amortization • Mergers or changes in management • Mergers or changes in major subcontractor management

9. CxP Surveillance Strategy (cont.) • The surveillance strategy employed must not drive technical requirements. • The surveillance strategy employed must not drive design; instead design, manufacturing, and assembly must drive surveillance strategy employed. • The surveillance strategy should change over the life cycle of the item (vehicle, system, etc.). As the item ages and more risk information is identified, changes may be necessary or beneficial to reflect either an increase or decrease in risk. • The surveillance strategy should be reviewed and adjusted at least annually.

10. CxP Surveillance Strategy (cont.) • Use a risk management approach and apply Penetration Levels based on level of risk of each area: • High Risk Areas = Higher Penetration • Low Risk Areas = Lower Penetration • Minimum acceptable Penetration Level is contingent on ensuring: • Safety for the public. • Safety for astronauts and pilots. • Safety for NASA workforce. • Safety for high-value equipment and property.

11. CxP Surveillance Strategy (cont.) • Feedback should be used to adjust level of penetration • Changes in risk • NASA and Contractor metrics • Customer evaluations • Quality performance / delivery performance • Cost of Quality (rework, repair, scrap, warranty, concessions, recalls/GIDEPs/NASA Alerts, returns, etc.) • Inspection / test results • Audit / survey / assessment results • Schedule aggressiveness • Number and scope of requirements and design changes

12. Risk H = High M = Medium L = Low Penetration Level 5 Level 4 Level 3 Level 2 Level 1 CxP Surveillance Strategy (cont.) TypicalPenetration Mapped on Risk Chart Review of Implementation Did they do things right? L i k e l i h o o d Increasing SR&QA Penetration Review of Processes Consequence of Failure Did they do the right things?

13. CxP Surveillance Strategy (cont.) • Acceptable risk is the risk that is understood and agreed to by the appropriate parties sufficient to achieve the defined success criteria within the approved level of resources. • Characterization of a primary risk as "acceptable" must be supported by the rationale, with the concurrence of the senior management, that all reasonable mitigation options (within cost, schedule, and technical constraints) have been instituted.

14. Surveillance Activity must Add Value • The surveillance strategy must add value to the output of the process. • Identify and correct problems (or potential problems) as early as possible in the process. • Do not add unnecessary surveillance activities in the process. • The surveillance strategy should be streamlined into the manufacturing/assembly process • Steps performed within process during operation. • Avoid/rethink surveillance that inspects batches of parts. Avoid surveillance bottle necks.

15. Yes Surveillance Activity must Add Value Yes Is it (part of): *adjusting? *approving? *doing work over? *expediting? *inspecting? *moving? *reviewing? *storing? *troubleshooting *waiting Does it convert input to output? (Is it required to produce output?) Are we doing this for customer (vs. for us)? Does the end customer care? (Is the end customer willing to pay for it?) NVA No No No No Activity Yes Yes VA

16. CxP Surveillance Strategy (cont.) • Summary: • The level of surveillance is determined by the level of risk and the impact of failure. • There are many different surveillance methods; not just inspection. • Put an emphasis on surveillance methods that uncover conditions before they become problems (e.g., precursor, self-check, SPC, business/financial changes, etc.) • Characterize the acceptable risk with rationale and concurrence from Senior Management. • Surveillance methods must add value and be streamlined into processes.

17. Questions?

18. Backup

19. Assurance Penetration LevelsProgram / Project Level DRAFT The premise of risk-based surveillance is simply this: The higher the risk in terms of safety, hardware criticality, or contractor’s stability or capability, the deeper the surveillance penetration.

20. Assurance Penetration LevelsProcess / Task Level DRAFT

21. PENETRATION Program / Project Level SR&QA Penetration

22. PENETRATION Process / Task Level SR&QA Penetration

23. Assurance Levels - Quality DRAFT