Valério De Patta Pillar Departamento de Ecologia Universidade Federal do Rio Grande do Sul

1. Accuracy and power of randomization tests in multivariate analysis of variance with vegetation data Valério De Patta Pillar Departamento de Ecologia Universidade Federal do Rio Grande do Sul Porto Alegre, Brazil vpillar@ecologia.ufrgs.br http://ecoqua.ecologia.ufrgs.br

2. Randomization testing: • Became practical with fast microcomputers. • Applicable to most cases analyzed by classical methods. • Applicable to cases not covered by classical methods.

3. How good is randomization testing? • Is it accurate? • Is it powerful enough?

4. Group comparison by randomization testing Choose a test criterion () to compare the groups Permute the data according to the conditions stated by the null hypothesis (Ho) that the groups do not differ Calculate the test criterionin the random data and compare it to the value found in the observed data. After many iterations, the probability P(o ≥ ) will be the number of iterations with o ≥  divided by the total number of iterations. Reject Ho if P(o ≥ ) is smaller than a threshold () Manly, B. F. J. 1997. Randomization, Bootstrap and Monte Carlo Methods in Biology. 2 ed. Chapman and Hall.

5. Randomization test criteria for multivariate comparisons of any number of groups *Pillar, V. D. & Orlóci, L. 1996. J. Veg. Sci. 7:585-592.

6. Observed squared distance matrix Total sum of squares (Qt)= (2.1 + 1.3 + ... + 10.0)/10 = 60.088 SQ within groups (Qw) = 0/1 + 0.5/2 + (0.1+2+2.6+10.6+10.8+5.8)/4 + 3.6/2 + 0/1 = 10.02 An example Is there a significant effect of N on vegetation composition as defined by these two PFTs? SQ between groups(Qb) = 60.088 - 10.02 = 50.068 How common is a Qb ≥ 50.068 if Ho were true (that the composition is unrelated to group)?

7. Reference set under Ho If Ho true, the observation vector in a given sampling unit is independent from the group to which the unit belongs.

8. Observed Permuted squared distance matrix Permuted Total sum of squares (Qt)= (6.8 + 10.6 + ... + 6.1)/10 = 60.088 SQ within groups (Qwo) = 0/1 + 0.7/2 + (13+12.8+44.4+0.2+10+15.6)/4 + 8/2 + 0/1 = 28.35 A random permutation and corresponding statistics SQ between groups(Qbo) = 60.088 - 28.35 = 31.738 Since, 31.738 < 50.068 (Qbo < Qb), this iteration adds zero to the frequency of cases in which Qbo ≥ Qb.

9. After 10000 random permutations…

10. Two-factor designs Test criterion: Qb = Qt - Qw is based on the groups defined by the joint states of the factors. Qb is partitioned as Qb = Qb|A + Qb|B + Qb|AB where Qb|A: sum of squares between la groups according to factor A disregarding factor B Qb|B: sum of squares between lb groups according to factor B disregarding factor A Qb|AB: sum of squares of the interaction AB, obtained by difference. F-ratio = Qb/Qw

11. Unrestricted permutation in two-factor design

12. Two-factor Multivariate Analysis of Variance One random permutation Observed Data: Species (57) composition in 8 vegetation units surveyed in two landscape positions (factor A) and two grazing levels (factor B).

13. After 10000 random permutations… Data: Species (57) composition in 8 vegetation units surveyed in two landscape positions (factor A) and two grazing levels (factor B). Unrestricted random permutations. Test criterion F-ratio = Qb/Qw.

14. Restricted permutations • In two-factor (not nested) designs, for testing one factor, permutations may be restricted to occur within the levels of the other factor (Edgington 1987). • Restricted permutation within the levels of factor A (for testing factor B): Edgington, E. S. 1987. Randomization Tests. Marcel Dekker, New York.

15. Permutations of residuals instead of raw data

16. Two-factor multivariate analysis of variance by randomization testing for the effects of landscape position and grazing level in natural grassland, southern Brazil (data from Pillar 1986). The data set contains 16 polled community stands by 60 species. Restricted random permutations for testing factors landscape and grazing. Permutation ofresiduals removing both factors for testing the interaction.

17. How good is randomization testing in two-factor multivariate analysis of variance?

18. Simulation of interaction • For each case, 1000 data sets were generated, with distribution properties of real vegetation data and subject to multivariate analysis of variance with randomization testing. • When factor or interaction effect is set to zero, the proportion of Ho rejection under a given a threshold estimates Type I Error, the probability of wrongly rejecting Ho when it is true. • If Type I Error is equal to a, the test is exact. • When factor or interaction effect > 0, the proportion of Ho rejection estimates the power of the test, which is the one-complement of Type II Error, the probability of not rejecting Ho when it is false.

19. Simulated data generated with distributional properties of real data Data set: 16 grassland units described by cover of 60 species. Two factors: landscape position (top-convex, concave-lowland) and grazing levels (grazed, ungrazed). Procedure described by Peres-Neto & Olden (2000): • Calculate the mean () and the standard deviation (ij) for each species vector i within each group j defined by the four factor level combinations; • Standardize these vectors for mean equal 0 and standard deviation equal 1, thij=(xhij-)/ ; • Randomly permute whole stand vectors across groups; • Restore the original dispersion within each group by computing new observations shij= thij, defining in this way a data set with the conditions specified by Ho; • Apply to the species vectors the corresponding group differences for factor and interaction effects; • Perform the randomization tests using 1000 random permutations; • Repeat the steps (3) to (6) 1000 times, recording the proportion of Ho rejection. Peres-Neto, P.R. & Olden, J.D. 2000. Animal Behaviour 61: 79-86.

20. With no factor and interaction effects, type I error is not different from 0.05, as expected by using  = 0.05. As the effect of factor 1 increases, type I error for factor 2 and interaction are underestimated with unrestricted permutations with Qb and -ratio, but not with restricted permutations and residuals. Results of power evaluation by data simulation in two-factor MANOVA. The proportion of Ho rejection at a = 0.05 was obtained for 1000 simulated data sets generated on the basis of plant community data with 16 units and 60 species, with increasing difference between the two groups for factor 1, with no interaction. Each factor combination had equal number of units. For each data set a randomization test was run with 1000 iterations.

21. As the effects of both factors increase, type I error for the interaction is underestimated with unrestricted permutations with Qb and -ratio, but not with residuals.

22. As the effect of interaction increases, type I error for both factors is underestimated with Qb and -ratio, un- and restricted permutations. But, main factors should not be considered at all when interaction is present!

23. As the effects of both factors increase, the power of the test with permutations of raw data is decreased for detecting the interaction when using Qb and -ratio, but not when permuting residuals.