A Unified Approach for Assessing Agreement

1. A Unified Approach for Assessing Agreement Lawrence Lin, Baxter Healthcare A. S. Hedayat, University of Illinois at Chicago Wenting Wu, Mayo Clinic

2. Outline • Introduction • Existing approaches • A unified approach • Simulation studies • Examples

3. Introduction • Different situations for agreement • Two raters, each with single reading • More than two raters, each with single reading • More than two raters, each with multiple readings • Agreement within a rater • Agreement among raters based on means • Agreement among raters based on individual readings

4. Existing Approaches (1) • Agreement between two raters, each with single reading • Categorical data: • Kappa and weighted kappa • Continuous data: • Concordance Correlation Coefficient (CCC) • Intraclass Correlation Coefficient (ICC)

5. Existing Approaches (2) • Agreement among more than two raters, each with single reading • Lin (1989): no inference • Barnhart, Haber and Song (2001, 2002): GEE • King and Chinchilli (2001, 2001): U-statistics • Carrasco and Jover (2003): variance components

6. Existing Approaches (3) • Agreement among more than two raters, each with multiple readings • Barnhart (2005) • Intra-rater/ inter-rater (based on means) /total (based on individual observations) agreement • GEE method to model the first and second moments

7. Unified Approach • Agreement among k (k≥2) raters, with each rater measures each of the n subjects multiple (m) times. • Separate intra-rater agreement and inter-rater agreement • Measure relative agreement, precision, accuracy, and absolute agreement, Total Deviation Index (TDI) and Coverage Probability (CP)

8. Unified Approach - summary • Using GEE method to estimate all agreement indices and their inferences • All agreement indices are expressed as functions of variance components • Data: continuous/binary/ordinary • Most current popular methods become special cases of this approach

9. Unified Approach - model • Set up • subject effect • subject by rater effect • error effect • rater effect

10. Unified Approach - targets • Intra-rater agreement: • overall, are k raters consistent with themselves? • Inter-rater agreement: • Inter-rater agreement (agreement based on mean): overall, are k raters agree with each other based on the average of m readings? • Total agreement (agreement based on individual reading): overall, are k raters agree with each other based on individual of the m readings?

11. Unified Approach – agreement(intra) • : for over all k raters, how well is each rater in reproducing his readings?

12. Unified Approach – precision(intra) and MSD • : for any rater j, the proportion of the variance that is attributable to the subjects (same as ) • Examine the absolute agreement independent of the total data range:

13. Unified Approach – TDI(intra) • : for each rater j, % of observations are within unit of their replicated readings from the same rater. is the cumulative normal distribution is the absolute value

14. Unified Approach – CP(intra) • : for each rater j, of observations are within unit of their replicated readings from the same rater

15. Unified Approach – agreement(inter) • : for over all k raters, how well are raters in reproducing each others based on the average of the multiple readings?

16. Unified Approach – precision(inter) • : for any two raters, the proportion of the variance that is attributable to the subjects based on the average of the m readings

17. Unified Approach – accuracy(inter) • : how close are the means of different raters:

18. Unified Approach – TDI(inter) • : for overall k raters, % of the average readings are within unit of the replicated averaged readings from the other rater.

19. Unified Approach – CP(inter) • : for each rater j, of averaged readings are within unit of replicated averaged readings from the other rater

20. Unified Approach – agreement(total) • : for over all k raters, how well are raters in reproducing each others based on the individual readings?

21. Unified Approach – precision(total) • : for any two raters, the proportion of the variance that is attributable to the subjects based on the individual readings

22. Unified Approach – accuracy(total) • : how close are the means of different raters (accuracy)

23. Unified Approach – TDI(total) • : for overall k raters, % of the readings are within unit of the replicated readings from the other rater.

24. Unified Approach – CP(total) • : for each rater j, of readings are within unit of replicated readings from the other rater

25. Unified Approach is the inverse cumulative normal distribution is a central Chi-squre distribution with df=1

26. Estimation and Inference • Estimate all means, variance components, and their variances and covariances by GEE method • Estimate all indices using above estimates • Estimate variances of all indices using above estimates and delta method

27. Estimation and Inference (2) : the covariance of two replications, and ,with coming from rater and coming from rater

28. Estimation and Inference (3) : the variance from each combination of (i, j), i.e., each cell. Thus is the average of all cells’ variances.

29. Estimation and Inference (4) : the variance of replication of rater : the covariance of two replications, and , both of them coming from rater .

30. Estimation and Inference (5) • Using GEE method to estimate all indices through estimating the means and all variance components:

31. Estimation and Inference (6)

32. Estimation and Inference (7)

33. Estimation and Inference (8) • is the working variance-covariance structure of , “working” means assume following normal distribution • is the derivative matrix of expectation of with respective to all the parameters

34. Estimation and Inference (9) • GEE method provides: • estimates of all means • estimates of all variance components • estimates of variances for all variance components • Estimates of covariances between any two variance components

35. Estimation and Inference (10) • Delta method is used to estimate the variances for all indices

36. Estimation and Inference (12)

37. Estimation and Inference (13)

38. Estimation and Inference (14)

39. Estimation and Inference (15)

40. Estimation and Inference (16)

41. Estimation and Inference (17)

42. Estimation and Inference (18) • Transformations for variances • Z-transformation: CCC-indices and precision indices • Logit-transformation: accuracy and CP indices • Log-transformation: TDI indices

43. Simulation Study • three types of data: binary/ordinary/normal • three cases for each type of data • k=2, m=1 / k=4, m=1 / k=2, m=3 • for each case: 1000 random samples with sample size n=20 • for binary and ordinary data: inferences obtained through transformation vs. no-transformation • For normal data: transformation

44. Simulation Study (2) • Conclusions: • Algorithm works well for three types of data, both in estimates and in inferences • For binary and ordinary data: no need for transformation • For normal data, Carrasco’s method is superior than us, but for categorical data, our is superior. • For ordinal data, both Carrasco’s method and ours are similar.

45. Example One • Sigma method vs. HemoCue method in measuring the DCHLb level in patients’ serum • 299 samples: each sample collected twice by each method • Range: 50-2000 mg/dL

46. Example One – HemoCue method HemoCue method first readings vs. second readings

47. Example One – Sigma method Sigma method first readings vs. second readings

48. Example One – HemoCue vs. Sigma HemoCue’s averages vs. Sigma’s averages

49. Example One – analysis result (1)

50. Example One – analysis result (2) *: for all CCC, precision, accuracy and CP indices, the 95% lower limits are reported. For all TDI indices, the 95% upper limit are reported.