Six Sigma in Measurement Systems: Evaluating the Hidden Factory

OK First Time Correct Inputs Operation Inspect NOTOK Rework Hidden Factory Scrap Time, cost, people Six Sigma in Measurement Systems: Evaluating the Hidden Factory Bill RodebaughDirector, Six Sigma GRACE

Objectives • The Hidden Factory Concept • What is a Hidden Factory? • What is a Measurement System’s Role in the Hidden Factory? • Review Key Measurement System metrics including %GR&R and P/T ratio • Case Study at W. R. GRACE • Measurement Study Set-up and Minitab Analysis • Linkage to Process • Benefits of an Improved Measurement System • How to Improve Measurement Systems in an Organization

OK First Time Correct Inputs Operation Inspect NOTOK Rework Hidden Factory Scrap Time, cost, people The Hidden Factory -- Process/Production • What Comprises the Hidden Factory in a Process/Production Area? • Reprocessed and Scrap materials -- First time out of spec, not reworkable • Over-processed materials -- Run higher than target with higherthan needed utilities or reagents • Over-analyzed materials -- High Capability, but multiple in-processsamples are run, improper SPC leading to over-control

The Hidden Factory -- Measurement Systems OK Production SampleInputs Lab Work Inspect NOTOK Re-test Hidden Factory Waste Time, cost, people • What Comprises the Hidden Factory in a Laboratory Setting? • Incapable Measurement Systems -- purchased, but are unusabledue to high repeatability variation and poor discrimination • Repetitive Analysis -- Test that runs with repeats to improve knownvariation or to unsuccessfully deal with overwhelming sampling issues • Laboratory “Noise” Issues -- Lab Tech to Lab Tech Variation, Shift toShift Variation, Machine to Machine Variation, Lab to Lab Variation

The Hidden Factory Linkage • Production Environments generally rely upon in-process sampling for adjustment • As Processes attain Six Sigma performance they begin to rely less on sampling and more upon leveraging the few influential X variables • The few influential X variables are determined largely through multi-vari studies and Design of Experimentation (DOE) • Good multi-vari and DOE results are based upon acceptable measurement analysis

Observed Process Variation Actual Process Variation Measurement Variation Long-term Short-term Variation Variation due Variation due Process Variation Process Variation w/i sample to gage to operators Repeatability Calibration Stability Linearity Possible Sources of Process Variation We will look at “repeatability” and “reproducibility” as primary contributors to measurement error

LSL USL LSL USL How Does Measurement Error Appear? Actual process variation - No measurement error Observed process variation - With measurement error

Measurement System Terminology • Discrimination - Smallest detectable increment between two measured values • Accuracy related terms • True value - Theoretically correct value • Bias - Difference between the average value of all measurements of a sample and the true value for that sample • Precision related terms • Repeatability - Variability inherent in the measurement system under constant conditions • Reproducibility - Variability among measurements made under different conditions (e.g. different operators, measuring devices, etc.) • Stability - distribution of measurements that remains constant and predictable over time for both the mean and standard deviation • Linearity - A measure of any change in accuracy or precision over the range of instrument capability

Measurement Capability Index - P/T • Precision to Tolerance Ratio • Addresses what percent of the tolerance is taken up by measurement error • Includes both repeatability and reproducibility • Operator x Unit x Trial experiment • Best case: 10% Acceptable: 30% Usually expressed as percent Note: 5.15 standard deviations accounts for 99% of Measurement System (MS) variation. The use of 5.15 is an industry standard.

Measurement Capability Index - % GR&R • Addresses what percent of the Observed Process Variation is taken up by measurement error • %R&R is the best estimate of the effect of measurement systems on the validity of process improvement studies (DOE) • Includes both repeatability and reproducibility • As a target, look for %R&R < 30% Usually expressed as percent

Case Study Background • Internal Raw Material, A1, is necessary for Final Product production • Expensive Raw Material to produce – produced at 4 locations Worldwide • Cost savings can be derived directly from improved product quality, CpKs • Internal specifications indirectly linked to financial targets for production costs are used to calculate CpKs • If CTQ1 of A1 is too low, then more A1 material is added to achieve overall quality – higher quality means less quantity is needed – this is the project objective • High Impact Six Sigma project was chartered to improve an important quality variable, CTQ1 • The measurement of CTQ1 was originally not questioned, but the team decided to study the effectiveness of this measurement • The %GR&R, P/T ratio, and Bias were studied • Each of the Worldwide locations were involved in the study • Initial project improvements have somewhat equalized performance across sites. Small level improvements are masked by the measurement effectiveness of CTQ1

CTQ1 MSA Study Design (Crossed) Site 2 Lab Site 3 Lab Site 1 Lab Site 4 Lab Site 1 Sample 2 Site 2 Sample 1….. Site 1 Sample 1 6 analyses/site/sample 2 samples taken from each site 2*4 Samples should be representative Each site analyzes other site’s sample. Each plant does 48 analyses 6*8*4=196 analyses Op 1 Op 2 Op 3 T2 T1

CTQ1 MSA Study Results (Minitab Output)

CTQ1 MSA Study Results (Minitab Session) Source DF SS MS F P Sample 7 14221 2031.62 5.0079 0.00010 Operator 11 53474 4861.27 11.9829 0.00000 Operator*Sample 77 31238 405.68 1.4907 0.03177 Repeatability 96 26125 272.14 Total 191 125058 %Contribution Source VarComp (of VarComp) Total Gage R&R 617.39 90.11 Repeatability 272.14 39.72 Reproducibility 345.25 50.39 Operator 278.47 40.65 Operator*Sample 66.77 9.75 Part-To-Part 67.75 9.89 Sample, Operator, & Interaction are Significant

CTQ1 MSA Study Results *Conf Int not calculated with Minitab, Based upon R&R Std Dev

CTQ1 MSA Study Results (Minitab Output) Dotplot of All Samples over All Sites Site 3 Site 1 Site 2 Site 4

CTQ1 MSA Study Results (Minitab Session) Analysis of Variance for Site Source DF SS MS F P Site 3 37514 12505 26.86 0.000 Error 188 87518 466 Total 191 125032 Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev -+---------+---------+---------+----- Site 1 48 824.57 15.38 (---*---) Site 2 48 819.42 22.11 (---*---) Site 3 48 800.98 20.75 (---*---) Site 4 48 840.13 26.58 (---*---) -+---------+---------+---------+----- Pooled StDev = 21.58 795 810 825 840 Site and Operator are closely related

CTQ1 MSA Study Results (Minitab Output) X-bar R of All Samples for All Sites Discrimination Index is “0”, however can probably see differences of 5 Most of the samples are seen as “noise”

X-bar R of All Samples for Site 4 CTQ1 MSA Study Results (Minitab Output) Mean differences are seen in X-bar area Most of the samples are seen as “noise”

CTQ1 MSA Study Results – Process LinkageSite 2 Example MSA Study Results with Mean = 819.4 2002 Historical Process Results with Mean = 832.5 Selected Samples are Representative

CTQ1 MSA Study Results – Process LinkageSite 2 Example MSA Study Results with Range = 17.92, Calc for Subgroup 2002 Historical Process Results with Range = 25.08 Calc for pt to pt When comparing the MSA with process operation, a large percentage of pt-to-pt variation is MS error (70%) --- a back check of proper test sample selection

CTQ1 MSA Study Results – Process LinkageSite 2 Example • Key issue for Process Improvement Efforts is “When will we see change?” • Initial Improvements to A1 process were made • Control Plan Improvements to A1 process were initiated • Site 2 Baseline Values were higher than other sites • Small step changes in mean and reduction in variation will achieve goal • How can Site 2 see small, real change with a Measurement System with 70+% GR&R? Use Power and Sample Size Calculator with and without impact of MS variation. Lack of clarity in process improvement work, results in missed opportunity for improvement and continued use of non-optimal parameters

CTQ1 MSA Study Results – Process LinkageSite 2 Example 2-Sample t Test Alpha = 0.05 Sigma = 22.23 Sample Target Actual Difference Size Power Power 2 2117 0.9000 0.9000 4 530 0.9000 0.9002 6 236 0.9000 0.9002 8 133 0.9000 0.9001 10 86 0.9000 0.9020 12 60 0.9000 0.9023 14 44 0.9000 0.9007 16 34 0.9000 0.9018 18 27 0.9000 0.9017 20 22 0.9000 0.9016 2-Sample t Test Alpha = 0.05 Sigma = 6.67 Sample Target Actual Difference Size Power Power 2 192 0.9000 0.9011 4 49 0.9000 0.9036 6 22 0.9000 0.9015 8 13 0.9000 0.9074 10 9 0.9000 0.9188 12 7 0.9000 0.9361 14 5 0.9000 0.9156 16 4 0.9000 0.9091 18 4 0.9000 0.9555 20 3 0.9000 0.9095 Simulated Reduction of Pt to Pt variation by 70% decreases time to observe savings by over 9X.

CTQ1 MSA Study Results – Process LinkageSite 2 Example Benefits of An Improved MS • Realized Savings for a Process Improvement Effort • For A1, an increase of 1 number of CTQ1 is approximately $1 per ton • Change of 10 numbers, 1000 Tons produced in 1 month (832  842) • $1 * 10 * 1000 = $10,000 • More trust in all laboratory numbers for CTQ1 • Ability to make process changes earlier with R-bar at 6.67 • Previously, it would be pointless to make any process changes within the 22 point range. Would you really see the change? • As the Six Sigma team pushes the CTQ1 value higher, DOEs and other tools will have greater benefit

Measurement Improvement in the Organization • Initial efforts for MS improvement are driven on a BB/GB project basis • Six Sigma Black Belts and Green Belts Perform MSAs during Project Work • Lab Managers and Technicians are Part of Six Sigma Teams • Measurement Systems are Improved as Six Sigma Projects are Completed • Intermediate efforts have general Operations training for lab personnel, mostly laboratory management • Lab efficiency and machine set-up projects are started • The %GR&R concept has not reached the technician level • Current efforts enhance technician level knowledge and dramatically increase the number of MS projects • MS Task Force initiated (3 BBs lead effort) • Develop Six Sigma Analytical GB training • All MS projects are chartered and reviewed; All students have a project • Division-wide database of all MS results is implemented

Measurement Improvement in the Organization • Develop common methodology for Analytical GB training

Final Thoughts • The Hidden Factory is explored throughout all Six Sigma programs • One area of the Hidden Factory in Production Environments is Measurement Systems • Simply utilizing Operations Black Belts and Green Belts to improve Measurement Systems on a project by project basis is not the long term answer • The GRACE Six Sigma organization is driving Measurement System Improvement through: • Tailored training to Analytical Resources • Similar Six Sigma review and project protocol • Communication to the entire organization regarding Measurement System performance • As in the case study, attaching business/cost implications to poorly performing measurement systems

Six Sigma in Measurement Systems: Evaluating the Hidden Factory