Practical Closed Stepwise Procedures to Test Homogeneity of All Subsets of Means

1. Practical Closed Stepwise Procedures to Test Homogeneity of All Subsets of Means James Schmeidler Departments of Psychiatry and Biomathematical Sciences Mount Sinai School Of Medicine New York, NY

2. Stepdown Procedures • Assumptions as in one-way ANOVA • k groups, ni per group • When testing the null hypothesis that a set of p means is homogeneous, a step-down procedure requires rejection for every set including all p means

3. Homogeneity Test for a Subset • Use pooled within-group mean square, s2 • Two procedures have been considered • ANOVA (test F times numerator df) • Studentized ranges: take maximum of (xa - xb) /{s[(1/na + 1/nb)/2]1/2} for all pairs of means in the subset

4. Significance Level for a Subset • Experimentwise α level • i means in the subset • (i/k)α level (Bonferroni inequality) • Better than Scheffe or Tukey procedures • They test every subset at α level critical value for a sample of k means

5. Closed Procedure (Peritz and Gabriel) • Consider partitions that include the set of p • For each partition, there is a composite null hypothesis that all its sets are homogeneous • The partition of the set of p and singleton sets tests only the set of p, at α level • Reject the null hypothesis for the set of p if the composite hypothesis is rejected for every partition

6. Begun and Gabriel Algorithm • Reject null hypothesis for the set of p if • The set of p is significant at α level • Every complementary set of i > 2 means significant at (i/k)α level • Only necessary to test each set once, not all sets in every partition

7. Application of the Algorithm • m = k - p complementary means have 2m - m - 1 complementary subsets of i > 2 • Many fewer complementary sets to test than number of partitions • General strategy: test least likely sets first • Specific strategy: test pairs of means adjacent in the complementary set

8. Conjectures About Pairs of Adjacent Complementary Means • If adjacent pairs of complementary means differ significantly: xa << xb and xb << xc • xa << xc • The set of three means is heterogeneous • Do these results generalize to more means? • Do these results apply to both ANOVA and Studentized ranges? • Do they apply in unbalanced designs, too?

9. Theorem 1. Testing Adjacent Pairs of Means • For Studentized ranges and ANOVA • If xa < xb< xcwith both {xa, xb} and {xb, xc} significant at the (2/k)α level • {xa, xc}is significant at the (2/k)α level

10. Theorem 2. Testing Adjacent Sets of Means • For Studentized ranges in balanced designs and all ANOVAs • If sets of D means with ends xa < xb and of E means with ends xb< xc are significant at the (D/k)α and (E/k)α levels, respectively • Their union is significant at the [(D + E – 1)/k]α level

11. Theorem 3. Test Only Pairs of Adjacent Complementary Means • For Studentized ranges in balanced designs and all ANOVAs • If all adjacent pairs of mean in a set are significant at the (2/k)α level • The set and all its subsets are significant at the respective (i/k)α levels

12. Counterexample for Theorem 2 • Studentized ranges; α = .05; k = 5; s = 1.00 • n1 = n2 = n3 = 10; n4 = n5 = 3500 • Means: 0.00, 1.00, 2.00, 2.74, 2.80 • Test of first two means using closure • Each complementary pair is heterogeneous • All three complementary means are not

13. Implications of the Counterexample • For Theorem 2 to fail for unbalanced Studentized ranges, the sample sizes must be ridiculously discrepant • Therefore Theorems 2 and 3 apply to Studentized ranges in almost all practical applications

14. Seaman, Levin, and Serlin (SLS) Improved Significance Levels • If r means left after excluding singleton sets from the partition • Test set of i means at the (i/r)α level • If r < k, (i/r)α > (i/k)α, increasing power • A set has different significance levels in different partitions • Cannot use Begun and Gabriel algorithm

15. Problematic Sets and Partitions • A “problematic” complementary set is one not known to be significant at the (i/k)α level by a direct test, Theorem 1, or Theorem 2 • A “problematic” partition consists of the set of p means, at least one problematic set, and perhaps singletons

16. Theorem 4. Using SLS, Test Only Problematic Partitions • If any problematic partition has r = k, set of p not rejected using SLS • Let u = largest integer for which the set of p means is rejected at the [(p/(u + p)]α level • All problematic sets with i > u means must be rejected at the [(i/(i + p)]α level • If r < u + p, no need to test a partition’s sets

17. Situations For Examining Only Problematic Sets, Not Partitions • If every problematic partition has only one problematic set • If every problematic set of i means is rejected at the [i/(v + p)]α level, for v means in the union of all problematic sets

18. Example: Fail to Reject Only One Complementary Pair • Any set that includes the pair is problematic • Every problematic partition has only one problematic set, so only examine the sets • Only sets with i > u means need to be examined • If a set of i means is rejected at (i/k)α level, Theorem 2 applies to all sets including it

19. Conclusions • Closed stepdown procedures improve power, but at the cost of additional tests • Can use Tukey or Scheffe, stepdown, closure, and SLS in succession • Theorem 3 makes the Begun and Gabriel algorithm much more practical • Theorem 4 simplifies the SLS procedure