Embedding equivalence t-test results in Bland Altman Plots visualising rater reliability

1. PhUSE 2011 Embedding equivalence t-test results in Bland Altman Plots visualising rater reliability Jim Groeneveld, OCS Consulting, ‘s Hertogenbosch, Netherlands. PhUSE 2011

2. Equivalence t-test & Bland Altman • AGENDA / CONTENTS • Rater reliability (inter- / intra-) • Methods, variable type dependent • Equivalence t-test (quantitative) • Bland Altman Plots (qualitative) • Integration of both, visualising equivalence t-test results in Bland Altman Plots, showing quantitative (in)significant equivalence in the plots • Advantages of integration

3. Equivalence t-test & Bland Altman • Rater reliability • Determine reliability of measuring instrument (device and/or human) • Repeated measurements (judgments by raters) on same objects • by same instrument: intra-rater or within-rater reliability (2 or more repetitions) • by similar, but other instrument: inter-rater or between-rater reliability (2 or more) • Application (before and after study): • Certification on representative data (before) • QC (on sample) of existing study data (after)

4. Equivalence t-test & Bland Altman • B. Methods, variable type dependent • Categorial data (nominal or ordered) • Cohen’s Kappa analysis (>2 cats: Fleiss) • McNemar’s test (>2 cats: McNemar-Bowker) Application: non-missing vs missing (binary) • Continuous data (interval or ratio) • Mean Absolute Difference (MAD) of pairs • Intraclass Correlation Coefficient (ICC), pairs • Equivalence t-test (quantitative interpretation) • Bland Altman Plots (qualitative interpretation) Application: ordered multi-level categorical data

5. Equivalence t-test & Bland Altman • C. Equivalence t-test (range limits) • on differences between paired measurements • two one-sided non-inferiority t-tests • user specification of equivalence range limits ((a)symmetrical) • Result for each combination of pairs of matching, repeated measurements: • significant equivalence or not • depending on range limits

6. Equivalence t-test & Bland Altman • D. Bland Altman Plots • Scattergram of pairwise points of: • Mean of pairs: X=(v1+v2)/2 versus • Difference of pairs: Y= v1-v2 including • Horizontal line of mean difference and • Confidence Interval (CI) of points,upper and lower horizontal lines • Qualitative interpretation of reliability

7. Equivalence t-test & Bland Altman • D. Bland Altman Plots (example)

8. Equivalence t-test & Bland Altman • E. Integration of equivalence t-test and Bland Altman Plots • Scattergram of pairwise points of: • Mean of pairs: X=(v1+v2)/2 versus • Difference of pairs: Y= v1-v2 including • Horizontal line of mean difference and • Confidence Interval (CI) of the mean,upper and lower horizontal lines • T-test range limits, horizontal lines • Quantitative interpretation of reliability

9. Equivalence t-test & Bland Altman • E. Integration of equivalence t-test and Bland Altman Plots (example with significant equivalence)

10. Equivalence t-test & Bland Altman • E. Integration of equivalence t-test and Bland Altman Plots • visualising equivalence t-test results in Bland Altman Plots • showing quantitative significant equivalence in the plots • if the Confidence Interval of the mean lies fully within the T-test range limits there is significant equivalence

11. Equivalence t-test & Bland Altman • E. Integration of equivalence t-test and Bland Altman Plots (example with non-significant equivalence)

12. Equivalence t-test & Bland Altman • F. Advantages of integration • Extension of (value of) Bland Altman Plots with quantitative interpretation on equivalence (in)significance • Equivalence (in)significance clearly visualised, depending on range limits • Results of two reliability analysis methods in one plot • showing a quantitative result and a qualitatively interpretable scatterplot

13. Equivalence t-test & Bland Altman • QUESTIONS • & • ANSWERS • SASquestions@ocs-consulting.com • Jim.Groeneveld@ocs-consulting.com • http://jim.groeneveld.eu.tf

14. Equivalence t-test & Bland Altman • More than 2 matching measurements • Pairwise analysis of repetitions(may yield many pairs of more than 3) • If more than 3 reduce number of analyses to “pairs” consisting of: • each individual measurement versus • the mean of all other matching measurements • This reduces the amount of “pairs” and analyses and facilitates an overall interpretation of the results.

15. Equivalence t-test & Bland Altman • A SAS macro (Concord) is currently under development in which these techniques already are supported and applied. • Additional features: relative differences • difference between both values:Y = v1- v2 • proportional difference with mean of both: Y = (v1- v2) / mean[v1,v2] = 2 * (v1- v2) / (v1+ v2) • (relative) proportion of both values, minus 1: Y = (v1 / v2) - 1 = (v1 - v2) / v2 • proportion of 1 value of mean of both, minus 1: Y = (v1 / mean[v1,v2]) -1 = (v1-v2) / (v1+v2)

16. Equivalence t-test & Bland Altman SAS Macro TickMark (version 0.0.1) Neat automatic ticmarks for graphs based on minimum and maximum of an existing value range (tickmarks 1 to 2 significant digits). Optional specification: desired minimum and maximum number of tick marks and minimum percentage of coverage of existing data range by generated value range (default values: minimum=7, maximum=12, pct coverage=80). Return of From, To and By values via macro variables or as a single return value. 16