Statistical tests for replicated experiments

1. Statistical tests for replicated experiments • Normal probability plots are a less formal diagnostic tool for detecting effects • F-tests and t-tests provide a statistical test of factor effects

2. Statistical tests for replicated experiments • Statistical tests are possible for unreplicated designs (unreplicated pilot studies are essential tools in sample size calculations) • We will first focus on statistical tests for replicated designs

3. Statistical tests for replicated experiments--Example • Response--Pulse rate of subject • Factors • Treatment (Energy Drink, Placebo) • Setting (Moderate, Difficult) • Machine (Stair climber, Recumbent bike)

4. Statistical tests for replicated experiments--Example

5. Statistical tests for replicated experiments--Example

6. Statistical tests for replicated experiments--Example

7. Statistical tests for replicated experiments • Effect sizes depend on the measurement scale • Statistical tests are based on standardized effects • To compute standardized effects, start with an estimate of experimental error

8. Statistical tests for replicated experiments • Experimental error can be summarized by the square root of the variance of the background noise (the standard deviation) • The experimental error measures variation in a single observation

9. Statistical tests for replicated experiments • The variance is best estimated by the Mean Square for Pure Error (MSPE)

10. Statistical tests for replicated experiments--Example • The standard deviation for each run is ~3 beats per minute

11. Statistical tests for replicated experiments • While the standard deviation for a single response is the square root of MSPE, the standard deviation of an effect (its standard error) is:

12. Statistical tests for replicated experiments • We divide an effect in a k-factor experiment with n replications (e.g., A) by its standard error to compute a t-test statistic :

13. Statistical tests for replicated experiments • Test statistics for other effects are computed similarly • U-do-it: Calculate the T-statistics of all effects for the Exercise data

14. Statistical tests for replicated experiments • When an effect is negligible, T has a t-distribution • The shape of the t-distribution curve depends on the number of replicates (“degrees of freedom”=2k(n-1)) • The t-distribution curves have slightly more spread than the bell-shaped (“normal”) curve

15. Statistical tests for replicated experiments--t curve for 3-factor design

16. Statistics tests for replicated experiments • If |T| is larger than the 99.5th or 97.5th percentile of the t distribution, an effect is significant • These percentiles are commonly found in textbooks (but please use a computer package instead)

17. Statistical tests for replicated experiments--t critical value for 3-factor design (n=4)

18. Statistical tests for replicated experiments • Sometimes, twice the area to the right of |T| is reported as a p-value. Small p-values suggest that a standardized effect is distinguishable from background noise • You definitely need a computer to compute p-values--in the following example, the p-value for the M effect is 2*.122=.244

19. Statistical test for replicated experiments--Example

20. Statistical tests for replicated experiments--Example • U-do-it: Compute p-values for the remaining effects. Which effects are significant? Are these the same effects that the probability plot detected?

21. Statistical tests for replicated experiments • F tests for individual effects are equivalent to t-tests • F tests allow several comparisons to be tested simultaneously • The t-test can be used to help in computing the number of replications needed in a factorial experiment

22. Statistical tests for replicated experiments • Hypothesis testing can be extended to combine estimates of error from both pure error and negligible effects • Negligible effects can be selected a priori or from effects plots • Degrees of freedom for t-tests and F-tests should be adjusted accordingly

23. Statistical tests for replicated experiments Error estimated from negiglible terms (Lack of Fit) is similar to MSPE Source DF SS MS Residual Error 28 300.38 10.73 Lack of Fit 4 45.38 11.34 Pure Error 24 255.00 10.63