Reference Value Estimators in Key Comparisons

1. Reference Value Estimators in Key Comparisons Margaret Polinkovsky Advisor: Nell Sedransk NIST May 2004

2. Statistics, Metrology and Trade • Statistics • Estimation for measurements • (1st moment) • Attached Uncertainty • (2nd moment) • Incredible precision in National Metrology Institute (NMI) • Superb science • Exquisite engineering • Statistical analysis

3. What are Key Comparisons? • Each comparability experiment • Selected critical and indicative settings – “Key” • Tightly defined and uniform experimental procedures • Purpose • Establish degree of equivalence between national measurement standards • Mutual Recognition Arrangement (MRA) • 83 nations • Experiments for over 200 different criteria

4. Equivalence Comparability NMIa NMIb Measurements Measurements Specifications Requirements products Sellera Buyerb money NMI: National Metrology Institute

5. Elements of Key Comparisons • Key points for comparisons • Experimental design for testing • Participating NMIs • Measurement and procedure for testing • Statistical design of experiments • Analysis of target data • Statistical analysis of target data • Scientific review of measurement procedure

6. Issues for Key Comparisons Pilot NMI NMIs • Goals: • To estimate NMI-NMI differences • To attach uncertainty to NMI-NMI differences • To estimate Key Comparison Reference Value (KCRV) • To establish individual NMI conformance to the group of NMIs • To estimate associated uncertainty • Complexity • Artifact stability; Artifact compatibility; Other factors

7. Statistical Steps • Step 1 • Design Experiment (statistical) • Step 2 • Data collected and statistically analyzed • Full statistical analysis • Step 3 • Reference value and degree of equivalence determined • Corresponding uncertainties estimated

8. Present State of Key Comparisons • No consensus among NMIs on best choice of procedures at each step • Need for a statistical roadmap • Clarify choices • Optimize process

9. Outcomes of Key Comparisons • Idea • “True value” • Near complete adjustment for other factors • Model based, physical law based • Non-measurement factors • Below threshold for measurement • Precision methodology assumptions • Highly precise equipment used to minimize variation • Repetition to reduce measurement error

10. Outcomes of Key Comparisons Each NMI: Observation= “True Value”+ measurement +non-measurableerrorerror same for all NMIsvaries for NMI varies for NMI (after adjustment,if any)data based estimate different expert for each NMI common artifact“statistical uncertainty“ “non-statisticalor physical event uncertainty” • Goal • Estimate “True Value”: KCRV • Estimated combined uncertainty and degrees of equivalence Combined uncertainty

11. Problems to Solve • Define Best Estimator for KCRV • Data from all NMIs combined • Many competing estimators • Unweighted estimators • Median ( for all NMIs) • Simple mean • Weighted by Type A • Weighted by 1/Type A (Graybill-Deal) • Weighted by both Type A and Type B • Weighted by 1/(Type A + Type B) (weighted sum) • DerSimonian-Laird • Mandel-Paule

12. Role of KCRV • Used as reference value • “95% confidence interval” • Equivalence condition for NMI

13. Research Objectives • Objectives • Characterize behavior of 6 estimators • Examine differences among 6 estimators • Identify conditions governing estimator performance • Method • Define the structure of inputs to data • Simulate • Analyze results of simulation • Estimator performance • Comparison of estimators

14. Model for NMIi • Reference value: KCRV • (Laboratory/method) bias : Type B • (Laboratory/method) deviation : Type B • Measurement error:Type A

15. NMI Process NMI Process NMI Process NMI Process NMI Process Expertise b2 Data y s2 y = m + d* + rb* + e s2 = data-based variance estimate (Type A) d* = experiment-specific bias rb* = experiment-specific deviation e = random variation b2 = d2 + sb2 (Type B) d = systematic bias sb2= extra-variation Sources of Uncertainty

16. Scientist: unobservable Systematic Bias ~N( , ) Simulate random Extra-variation Simulate random Data: observable Observed “Best Value” Variance estimate s2~ 1/df( ) Simulate random Experimental Bias Simulate random Experimental Deviation Simulate random Translation to Simulation • Uncertainty • Type A: s2 • Type B:

20. Conclusions and Future Work • Conclusions • Uncertainty affects MSE more than Bias • Estimators performance • Graybill-Deal estimator is least robust • Dersimonian-Laird and Mandel-Paule perform well • When 1 NMI is not exchangeable the coverage is effected • Number of labs changes parameters • Future work • Use on real data of Key Comparisons • Examine other possible scenarios • Further study degrees of equivalence • Pair-wise differences

21. Looking Ahead • Use on real data of Key Comparisons • Examine other possible scenarios • Further study degrees of equivalence • Pair-wise differences

22. References • R. DerSimonian and N. Laird. Meta-analysis in clinical trials. Controlled Clinical Trials, 75:789-795, 1980 • F. A. Graybill and R. B. Deal. Combining unbiased estimators. Biometrics, 15:543-550, 1959 • R. C. Paule and J. Mandel. Consensus values and weighting factors. Journal of Research of the National Bureau of Standards, 87:377-385 • P.S.R.S. Rao. Cochran’s contributions to the variance component models for combining estimators. In P. Rao and J. Sedransk, editors. W.G. Cochran’s Impact on Statistics, Volume II. J. Wiley, New York, 1981 • A. L. Rukhin. Key Comparisons and Interlaboratory Studies (work in progress) • A. L. Rukhin and M.G. Vangel. Estimation of common mean and weighted mean statistics. Jour. Amer. Statist. Assoc., 73:194-196, 1998 • J.S. Maritz and R.G. Jarrett. A note on estimating the variance of the sample median. Jour. Amer. Statist. Assoc., 93:303-308, 1998 • SED Key Comparisons Group (work in progress)