User’s Guide to the ‘QDE Toolkit Pro’

1. Ch 9: 111 User’s Guide to the ‘QDE Toolkit Pro’ Sept 5, 2003 National Research Conseil national Council Canada de recherches Excel Tools for Presenting Metrological Comparisons byB.M. Wood, R.J. Douglas & A.G. Steele Chapter 9. Graphing and Pooling Measurement Distributions (ii) In this chapter, we present the use of the QDE Toolkit Pro’s facilities for graphing distributions and pooled distributions. We also present some summary statistics, calculated by the QDE Toolkit Pro, that are useful even before a KCRV is chosen.

2. Ch 9: 112 QDE Toolkit Pro Graphs -working with these Pooled Distributions After running the macrotk_pool_PlotBuilder, the graphs can be adjusted with Excel. Here’s a quick review of some things you will want to do: Point at a corner or edge of a selected graph and drag (left mouse button) to resize. Point at a graph, left mouse click to select it and pop it to the top of the stacked graphs. Tip: We findExcel’s undo capability is very useful in reversing unwanted formatting of graphs. Select Edit|Undo with the mouse, or ‘control Z’ from the keyboard. Double-click on the selected chart’s y axis (the vertical line at x=0) to get a dialog box, select the scale tab, enter the new y maximum... Similarly, the x-axis range, etc. can be adjusted. There is a very wide range in x of data points available (~1000 uniformly spaced points across the default graph range and ~1000 more on an expanding mesh to cover ~200 x the breadth (for integrating Student distribution tails).

3. Ch 9: 113 QDE Toolkit Pro Graphs -working with these Pooled Distributions After running the macrotk_pool_PlotBuilder, the graphs’ data can be edited or recalculated with Excel (for example to convolute with either a uniform or a “goalpost” distribution). The data (~2000 rows x ~10-20 columns) are extensive enough so that some Excel tricks (shift Edit|Paste Picture, or using F9 to convert a graph series into an explicit formula array) for taking a frozen picture of a graph no longer always work well. It is often very helpful to take a frozen snapshot of a graph, andgoing beyond screen resolution can be a challenge! After getting the Excel graph about right, one trick seems to give the best for-printing conversion to Windows metafile form… (continued on next page) Prior to Excel 97 SR2, Excel had a memory leak when creating lots of charts that could give a “not enough memory” error and required restarting Windows occasionally.

4. Ch 9: 114 QDE Toolkit Pro Graphs –freezing a graph (as a picture) For Office XP, we copy the selected graph to the clipboard – Control-C Paste it into Microsoft Word – Control-V, then click on the clipboard icon beside the pasted graph to select the “Paste Options” select “Picture of Chart (smaller file size)” Press the Esc key to get rid of the icon Select and copy the graph in Word Paste it back into Excelas a “frozen picture”. For Office 97, or for Office 2000, we copy the selected graph to the clipboard, and do a “Edit |Paste Special” into Microsoft Word as a “Picture (Enhanced Metafile)”. Then, copying from this graph in Word, it can be pasted (not Paste Special this time) back into Excel as a “frozen picture”.

5. Ch 9: 115 QDE Toolkit Pro Graphs- working with Reference-Value Distributions The macrotk_pool_PlotBuildercan also graph the PDFs of the candidate KCRVs. The RVs are graphed with thick lines. In column A of the input table, the block of reference values starts beneath the block of Lab comparison data, with the first name containing the character pair RV (case insensitive). It ends with the first blank in column A. Here, two RVs are graphed. One is labeled the KCRV, with no uncertainty given. (This is sometimes done to avoid profound difficulties, for example by the CCT.) This first RV (its value happens to be the simple mean of the 8 pooled Labs) is plotted as a delta-function (really .001 of the initial graph width). The second RV is an inverse-variance weighted mean of the 8 pooled labs, with its formal standard uncertainty (the same as the product of PDFs, since they are all normal) and correlation coefficients with the contributor labs (which are not used in plotting the RV’s PDF).

6. Ch 9: 116 QDE Toolkit Pro Graphs -working with the “supplementary information” After running the macrotk_pool_PlotBuilder,there is a block of supplementary information in the 16 columns to the right of the correlation coefficient matrix. We will discuss them in turn, but here’s the overview: Tip: the Toolkit Pro’s output italic numbers are unitless, and regular font numbers have units. Col after rij 1Lab names: comments hold reduced chi-square of differences with all other pooled Labs. 2-4Input Comparison Data for Labs, similar statistics for pools 5-6Du/u for degrees of freedom: variance-based and tail-based. 7-8Coverage Factors for 68.0% and 95.0% confidence (from trapezoidal integration of the distributions). Mostly of interest for the pooled distributions. 9-12Error in the symmetric vs rigorous confidence intervals. Mostly of interest for quantifying the un-importance of this effect for the asymmetric pooled distributions. 13-15Mean, Median and (first) Mode of each distribution. Mostly of interest for the pooled distributions, 16Lab Names, just for convenience

7. Ch 9: 117 QDE Toolkit Pro Graphs -col 1 of the “supplementary information” 1 Col after rij 1The comment in the top row of the column contains the date and time of creation of the block of supplementary information. The column is of Lab names for each row, but some really useful information is in the comments of n “in-pool” labs, which give the reduced chi-square statistical information about their rms En with all other (n-1) “in-pool” Labs: (n-1) terms, between 1 and (n-1) degrees of freedom. The bottom (“Pair Difference”) name has a comment that gives the all-pairs chi-squared for the pool: there are n(n-1) terms to sum, with an obvious 2:1 redundancy: Despite having n(n-1) terms, the degrees of freedom is still (n-1). Note that this chi-square is independent of any choice of KCRV. Its use is illustrated in: Hill, Steele and Douglas, Metrologia39, 269 (2002). It is also discussed in a bit more detail on the following page. It uses the Excel statistical function ChiDist to convert from an all-pairs-variance to a probability that it could be exceeded by chance.

8. Ch 9: 118 QDE Toolkit Pro Graphs- VERY USEFUL INFORMATION about Lab j The chi-squared of each of N in-pool Labs, with respect to the N-1 other Labs, is calculated and put into this column as comment boxes. This is the Lab’s “rms En”:j2 = (N-1)-1 i=1Ni≠j(xi – xj)2 / (ui2 + uj2 - 2rijuiuj)If the differences are independent and normally distributed about zero, then this is EXACTLY a reduced chi-squared statistic with N-1 degrees of freedom. It is perhaps the best test as to whether Lab j agrees with the “rest of the world”, and is independent of any choice of KCRV. In practice, the differences are only independent when the other Labs’ uncertainties dominate, and the degrees of freedom for j2 may be as small as 1 when uj dominates. If the differences were independent and there was perfect agreement, within the stated uncertainties, for Lab j, this value of j2 is expected to be exceeded by chance with only this probability: a minimum probability that may be a substantial underestimate if the differences’ lack of independence is considered. In the Toolkit Version 2.07, the probability, between 0 and 1, is given in scientific number format.

9. Ch 9: 119 QDE Toolkit Pro Graphs- VERY USEFUL INFORMATION about comparison The reduced chi-squared Labs j with respect to the N-1 other Labs, is:j2 = (N-1)-1 i=1Ni≠j(xi – xj)2 / (ui2 + uj2 - 2rijuiuj)By averaging the Nj2’s, 2 = (N)-1 i=1Nj2the all-pairs variance. It is perhaps the best test as to whether these Labs agree with each other within their uncertainties, and is independent of any choice of KCRV. The degrees of freedom of this reduced chi-squared is N-1. If there were perfect agreement, within the stated uncertainties, for all Labs, this value of 2 is expected to be exceeded by chance with only this probability. If this probability is very small, then the measured differences are not described very well by the stated uncertainties. In the Toolkit Version 2.07, the probability, between 0 and 1, is given in scientific number format.

10. Ch 9: 120 QDE Toolkit Pro Graphs- VERY USEFUL INFORMATION – a TIP There are so many comment boxes that the workbook needs to have its comments hidden unless mouse is pointing to a commented cell. (Tools | Options |View | and select “Comment indicator only”. ) To keep one or more comment boxes on display, Right-mouse-click on the cell Select “Show comment” To hide one or more comment boxes that is on display, Right-mouse-click on the cell Select “Hide comment”

11. Ch 9: 121 QDE Toolkit Pro - Using Pair Differences The bilateral pair differences of laboratories can incorporate all knowledge about the pairs, including correlations, so the reduced chi-square of all pair differences is an appropriate tool to consider for characterizing a comparison model without having to choose a specific reference value. The “all-pairs variance” or APV of a comparison with N laboratories can be calculated APV = [i =1Nj =1N(xj - xi)2 / (ui2 + uj2 - 2 rijui uj) ] / (N(N-1)) for normalPDF’s for the xi’s , the APV is distributed as a reduced chi-square with N-1 degrees of freedom. Aside: Here we use the name APV here since it sometimes may be appropriate to interpret the APV in terms of its model distribution, which is not necessarily a reduced chi-squared if we include the effects of the stated effective degrees of freedom for each laboratory. In the QDE Toolkit Pro Versions 2.04…2.07, the analysis is all in terms of the normal distribution limit and the reduced chi-squared.

12. Ch 9: 122 QDE Toolkit Pro - Using Pair Differences The “all-pairs variance” or APV of a comparison with N laboratories is APV = [i =1N j =1N (xj - xi)2 / (ui2 + uj2 - 2 rijui uj) ] / (N(N-1)) for normalPDF’s for the xi’s , the APV is distributed as a reduced chi-square with N-1 degrees of freedom. At the left, the Monte Carlo simulation of the distribution of the APV of a 12-Lab comparison is compared with the reduced chi-squared curves having 10, 11 and 12 degrees of freedom. Although the APV sum is over 132 non-zero terms, with 66 terms that look distinct, the distribution of APVs is just the reduced chi-squared distribution with 11 degrees of freedom. The APV has been constructed to be independent of any change of reference value. Although we may have not yet chosen a specific KCRV, one degree of freedom is used by this possibility.

13. Ch 9: 123 QDE Toolkit Pro - Using Pair Differences The “all-pairs variance” or APV of a comparison with N laboratories is the master chi-squared for the comparison. This is true in the sense that … if a comparison has failed the APV chi-squared test, then NO choice of reference value can, by itself, rescue the comparison… This can be demonstrated by considering what happens to the pair differences when the reference value is changed: the pair differences are invariant, and so the APV chi squared statistic is also invariant.

14. Ch 9: 124 QDE Toolkit Pro Graphs -col 2-4 of the “supplementary information” 2 3 4 Col after rij 2-4 Input Comparison Data for Labs: value, uncertainty, degrees of freedom. In columns 2 and 3, similar stastistical information is calculated for the pools, and explained in comments. Degrees of freedom is just as input, with a default to “normal. 3 4

15. Ch 9: 125 QDE Toolkit Pro Graphs -col 5-6 of the “supplementary information” 5 6 Col after rij 5Du/u for degrees of freedom: estimate based on the chi distribution’s variance-about-the-mean based and tail-based. The overly broad tails of the Student distribution at low degrees of freedom arise from the small arguments (and large value) of the chi distribution.Approximate fractional uncertainty in the standard uncertainty from variance of the chi-square family of distributions that are highly asymmetric for degrees of freedom less than 10. See ISO Guide Eq.E-7 & Table E-1. 6Du/u for degrees of freedom Improved estimate Du for the fractional uncertainty in the standard uncertainty, derived from the correct inverse-chi distribution's interval [0,u+Du] with 84% confidence (the same Du as Col 5 in the limit of large degrees of freedom). See discussion on pages 49-52 of this Guide, in the context of discussing Tables of Equivalence. Two definitions for degrees of freedom?No. The variance-based method is simply a bad approximation for degrees of freedom < 10, if the degrees of freedom is to be used for evaluating coverage factors from Student distributions. Nonetheless, we suggest using the symbol nSfor a degrees of freedom aimed at describing the tails of the Student distribution. Fortunately, in precision metrology, usually n > 10 and there is no difficulty.

16. Ch 9: 126 QDE Toolkit Pro Graphs -col 7-8 of the “supplementary information” 7 8 Col after rij 7-8 Coverage Factors for 68.0% and 95.0% confidence (from trapezoidal integration of the distributions). Mostly of interest for the pooled distributions. Because the product PDF is narrow, there may be only a few tens of samples within ±s. An easy way of monitoring this effect is to include a normal RV of about the same value and width. Any variation of the coverage factor from the expected norm (here +0.1%) is an indication of the accuracy if the product were over Student distributions instead of normal distributions. If the accuracy is not sufficient, in the Visual Basic code in module QDE_Toolkit_PlotBuilder, in subroutine tk_pool_PlotBuilder_With_Anchor, near comment line ‘C140, change poolpoints = 1002 to a larger number, such as poolpoints = 10002 (Excel will limit you to ~60000)... near comment line ‘C110 in the same subprogram Confidence_k1 = 0.68 Confidence_k2 = 0.95 can be edited to the values of your choice. These do not affect the MRA value of 95% confidence.

17. Ch 9: 127 QDE Toolkit Pro Graphs- col 9-12 of the “supplementary information” 9 10 11 12 Col after rij 9-12 Error in the symmetric vs rigorous confidence intervals. For the symmetric input data, you see the effects of round-off in the trapezoidal integration. These columns are mostly of interest for quantifying the un-importance of asymmetry for the asymmetric pooled distributions: “How close are the intervals {mean-u, mean+u] and [mean-U, mean+U] to the true asymmetric confidence intervals, starting for example at X where CDF(X)=0.025 and running to X’ where CDF(X’)=0.975.

18. Ch 9: 128 QDE Toolkit Pro Graphs - col 13-15 of the “supplementary information” 13 14 15 Col after rij 13-15 Mean, Median and (rightmost) Mode (the zero-slope peak) determined from the table of each distribution. Now this is mostly of interest for the pooled distributions, So as you see, there’s lots of information included in the “supplementary information” columns that is eminently ignore-able, most of the time. If and when it is wanted, it will be waiting!