1 / 36

Power Analysis

Power Analysis. How confident are we that we can detect an important difference, if it exists? Power = 1 -  , where ES = Effect Size Used to design an experiment ( a priori ) The size of an experiment is often limited by factors other than statistics

abel-black
Download Presentation

Power Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Power Analysis • How confident are we that we can detect an important difference, if it exists? • Power = 1 -  , where ES = Effect Size • Used to design an experiment (a priori) • The size of an experiment is often limited by factors other than statistics • Nonetheless, it is good experimental technique to try to estimate the degree of precision that will be attained and to present this information as part of the proposal for the experiment • Evaluate a completed experiment (a posteriori ) • May provide justification for publishing nonsignificant results • Guide for improving experimental technique in the future

  2. Power Analysis • Sources of input for power analysis • educated guesses derived from theory • results of previous studies reported in literature • pilot data • Effect Size – several options • minimal practical significance • educated guess of the true underlying effect • Questions of interest • number of replications • number of subsamples • plot size • detectable difference

  3. Replications needed to estimate a mean • For a single population, you may want to determine the sample size needed to obtain a given level of precision • Recall that • Rearrange the formula r is the number of replications t is the critical t with r-1 df d is the limit of the confidence interval

  4. Example • In a preliminary trial, a sample of 5 replicates has a standard deviation of 3.4 units • You would like to conduct an experiment to estimate the mean within two units of the true mean with a confidence level of 95% But we have a problem here... • We need to know how many reps in order to calculate degrees of freedom • So it is: • pick • plug • adjust

  5. r = 14 Calculations Given s = 3.4, d = 2,  = 0.05 Try starting with r = 5 Know your t table! In Excel, =T.INV.2T(0.05,4)  2.78 (2-tailed distribution) In Kuehl, use =0.025  = 4  2.78 (Prob. of t  t,) r  (2.782 * 3.42)/22 = 22.3 Additional iterations… r df t calculated r 23 22 2.1 12.4 13 12 2.2 13.7 14 13 2.2 13.5

  6. Detecting differences between means may be expressed as percentage of mean or on actual scale • Where r = the number of replications • t1 = t at the significance level for the test • t2 = t at 2(1-P), where P is the selected probability of obtaining a significant result (power) • (note that 1-P is , the probability of a type II error) • s = standard deviation (or CV%) • d = the difference that it is desired to detect

  7. What is meaningful? • If it can be established that the new is superior to the old by at least some stated amount, say 20%, then we will have discovered a useful result • If the experiment shows no significant difference, we will be discouraged from further investigation • The experiment should be large enough to ensure a meaningful difference.

  8. r > 2(2.145+1.076)2(11)2/(20)2 r > 6.28 ~ 7 r > 2(2.447+1.134)2(11)2/(20)2 r > 7.76 ~ 8 then test to see if 7 reps (12 df) is correctr > 6.44 ~ 7 For example: r > 2(t1 + t2)2CV2 / d2 • We have two varieties to compare. A previous experiment with these treatments found a CV of 11% • How many replications would be needed to detect a difference of 20% with a probability of .85 using a 5% significance level test? then test to see if 8 reps (14 df) is correct first pick 4 reps (6 df) t1=T.INV.2T(0.05,6)=2.447 t2=T.INV.2T (0.30,6)=1.134

  9. Number of reps for the ANOVA • The ANOVA may have more than two treatments, so we must consider differences among multiple means • Power curves are commonly used • see Kuehl pg 63 for more information • For this class, we will often get approximate estimates of the number of reps needed using the formula for two means • use the error term from ANOVA (MSE) to estimate s2 • use the appropriate degrees of freedom for the MSE • df = #treatments*(r-1) for a CRD • df = (#treatments-1)(r-1) for an RBD

  10. Power curve for ANOVA - example

  11. SAS Power Calculations • PROC POWER • PROC GLMPOWER • SAS power and sample size application • Stand alone desktop application that utilizes PROC POWER and PROC GLMPOWER through a user friendly interface

  12. Determining Plot Size Factors that affect plot size • Type of crop • Type of experiment • Phase of the research program • Variability of the experimental site • Presence and nature of border effects • Type of machinery to be used • Number and type of treatments • Land area available • Cost

  13. Factors Affecting Plot Size Increasing Plot Size FactorSmall plotsLarge Plots Soil variabilityUniform Heterogeneous CropTurf--Cereals -- Row crops -- Trees -- Pasture Research phaseEarly Late Experiment typeBreeding -- Fertilizer -- Tillage -- Irrigation MachineryNoneResearch Farm Scale

  14. Effect on Variability • Variability per plot decreases as plot size increases • But large plots may yield higher experimental error because of larger more variable area for the experiment • Very small plots are highly variable because: • Losses at harvest and measurement errors have a greater effect • Reduced plant numbers • Competition and border effects are greater

  15. Variability Size How Plot Size Affects Variability

  16. Plot Size ‘Rule of Thumb’ • There are some lower limits: • Should be large enough to permit removal of borders with enough left over to harvest and measure adequately • Should be large enough to handle the machinery needed • Once the plots are large enough to be handled conveniently, precision is increased faster by increasing the number of replications

  17. V xb Vx= Smith’s Soil Variability Index Where: V = variance of a unit plot Vx = variance, on a per unit basis, of plots formed from x adjacent units x = plot size in multiples of adjacent unit plots b = index of soil variability

  18. V xb Vx= V x When b=1 then Vx = Smith’s Soil Variability Index The index can vary from 0 to 1 When b=0 then Vx = V This means there is no relation between variance and plot size. Adjacent plots are completely correlated. Nothing can be gained from larger plots. This is the case when the soil is highly uniform. This means that the units that make up the plots are independent of each other. Increasing plot size will reduce variance. This is the case when the soil is highly heterogeneous.

  19. Effect of plant sample size • However, can only obtain b=0 when plant sample size is above a minimal level • For large plants, sample size may be more critical than soil variability in determining optimal plot size

  20. V xb Vx= Variability Index Calculation log Vx = log V - b log x y = a + bx a (intercept) Log of variance per plot y x b (slope) Log of plot size

  21. Raw data from Uniformity Trial 488 406 491 479 438 367 488 392 457 438 296 510 457 359 436 396 503 337 440 403 388 450 420 384 325 365 369 343 361 376 347 462 393 390 368 437 424 387 408 354 386 402 426 384 346 389 375 375 448 454 347 374 321 403 476 430 490 428 435 452 416 445 440 415 449 442 387 382 319 393 438 455 470 418 353 402 470 410 446 426 352 394 427 435 410 393 385 411 446 441 388 420 389 419 414 348 303 333 389 396 473 371 315 328 365 426 455 375 976 798 948 917 734 877 897 762 824 846 923 721 672 827 762 733 729 813 850 771 754 743 761 777 924 884 837 802 756 855 803 827 759 808 887 897 916 844 705 796 897 845 798 813 774 830 860 789 618 661 754 822 928 746 Combine yields of plots from adjacent rows (2x1)

  22. Combination x Log(x) Vx Log(Vx) 1x1 1 0.0000 2177.18 3.3379 1x2 2 0.3010 1244.46 3.0950 1x3 3 0.4771 959.05 2.9818 1x6 6 0.7782 631.50 2.8004 2x1 2 0.3010 1447.50 3.1606 2x2 4 0.6021 920.00 2.9638 2x3 6 0.7782 706.10 2.8489 2x6 12 1.0792 428.81 2.6323 3x1 3 0.4771 1090.53 3.0376 3x2 6 0.7782 767.66 2.8852 3x3 9 0.9542 592.02 2.7723 3x6 18 1.2553 432.91 2.6364 4x1 4 0.6021 684.23 2.8352 4x2 8 0.9031 472.11 2.6740 4x3 12 1.0792 335.64 2.5259 4x6 24 1.3802 351.01 2.5453 6x1 6 0.7782 339.54 2.5309 6x2 12 1.0792 194.84 2.2897 6x3 18 1.2553 159.58 2.2030

  23. 4 3.5 3 2.5 2 0.0000 0.4771 0.7782 0.9542 1.2553 Log of plot size Smith’s Index of Variability log Vx = 3.3261- 0.7026 log x log Vx = log V - b log x y = a + bx

  24. Is there an easier way? • Examination of a large number of data sets indicated that a value of b=0.5 may serve as a reasonable approximation. “Finagles” constant b=0.5

  25. Optimum Plot Size • Optimum Size will either minimize cost for a fixed variance or minimize variance for a fixed cost • You must know certain costs: T = K1 + K2x Where: T = total cost per plot ($ or time) K1= cost per plot ($ or time) that is independent of plot size K2= cost per plot ($ or time) that depends on plot size x = number of unit plots

  26. bK1 (1-b)K2 xopt = Optimum Plot Size • Knowing those costs, Smith found the optimum plot size to be: Where: T = total cost per plot ($ or time) K1= cost per plot ($ or time) that is independent of plot size K2= cost per plot ($ or time) that depends on plot size x = number of unit plots b = Smith’s index of soil variability

  27. bK1 (1-b)K2 xopt = 2 m 7 m Optimum Plot Size • So in our example, we found b to be 0.70 and our unit plot area was 5m x 2m or 10m2 • Assume K1 = $3.00 and K2 = $5.00 for 10 m2 = (0.70)(3.00) = 1.4 (1-0.70)(5.00) Area = (1.4)(10) = 14.0 m2

  28. xb = 2(t1 + t2)2 2 rd2 Convenient Plot Size • Smith’s optimum plot size was based on soil variability and cost. • Hathaway developed a formula based on soil variability, the size of difference to be detected, and significance level of the test.

  29. xb = 2(t1 + t2)2 2 rd2 Convenient Plot Size Where: x= number of units of plots of size x (plot size) b= Smith’s coefficient of soil variability t1= t at the significance level for the test t2= t in the t table at 2(1-P), where P is the selected probability of obtaining a significant result 2= variance from a previous experiment (may also use the CV2) r= number of replications d= difference to be detected (may be an absolute amount or expressed as percentage of the mean)

  30. For Example . . . • A variety trial of 50 selections in a randomized block design with three blocks • What size plot do we need to detect a difference of 25% of the mean with an 80% probability of obtaining a significant result using a 5% significance level test? • Previous experiment had a CV of 11%

  31. xb = 2(t1 + t2)2 CV2 rd2 Then: xb = 2(1.984 + 0.845)2(11)2 = 1936.78 (3)(25)2 1875 xb = 1.0330 or x.70 = 1.0330 x = 1.03301/.70 = 1.0475 basic units error df = (3-1)(50-1) = 98 t1 = t0.05(98)) = 1.984 t2 = t0.40(98) = 0.845 {t 2(1-P) = t 2(1-0.80) = t0.40} b = 0.70 CV = 11%

  32. 2 m 5.24 m Convenient Plot Size • Since the basic plot has an area of 10 m2, the required plot size would be: (10.00)(1.0475) = 10.48 m2

  33. d2 = 2(t1 + t2)2 2 rxb The Detectable Difference • Using Hathaway’s formula for convenient plot size and solving for d2, it is possible to compute the detectable difference possible when you know the number of reps and the plot size.

  34. d2 = 2(t1 + t2)2 CV2 rxb An Alternative Example • We want to find the difference that can be detected 80% of the time at a 5% significance level using a plot 2 m wide and 7 m long in 4 replications • As before, the CV=11%, and b=0.70

  35. d2 = 2(t1 + t2)2 CV2 rxb d2 = 2(1.976 + 0.844)2(11)2 (4)(1.40.70) because 2m*7m/10m2=1.4 = (2)(7.954)(121) = 1924.90 (4)(1.2656) 5.0623 = 380.24 d = 19.50 Therefore, with a plot size of 2m x 7m using 4 replications, a difference of 19.50% of the mean could be detected

  36. Using these tools 35 30 25 20 15 10 5 Detectable difference (% of Mean) 2 Replication 4 6 8 10 20 30 Plot Area (m2)

More Related