Understanding Fractional Factorial Designs for Performance Analysis

1. Fractional Factorial Designs Andy Wang CIS 5930 Computer Systems Performance Analysis

2. 2k-p FractionalFactorial Designs • Introductory example of a 2k-p design • Preparing the sign table for a 2k-p design • Confounding • Algebra of confounding • Design resolution

3. Introductory Exampleof a 2k-p Design • Exploring 7 factors in only 8 experiments:

4. Intuition • 27 design involves solving 128 unknown coefficients (q0, qA, qB, … qABCDEFG) • with 128 equations • Suppose we have just 8 unknowns • q0’ = q0 + (15 high order terms) • q1’ = qA + (15 high order terms) • q2’ = qB + (15 high order terms) • Only need 8 equations

5. Assumptions • High-order interactions contribute less to the net performance • E.g., caching on/off vs. multicore on/off • Might be not be true • E.g., two layers of caching

6. Analysis of 27-4 Design • Column sums are zero: • Sum of 2-column product is zero: • Sum of column squares is 27-4 = 8 • Orthogonality allows easy calculation of effects:

7. Effects and Confidence Intervals for 2k-p Designs • Effects are as in 2k designs: • % variation proportional to squared effects • For standard deviations, confidence intervals: • Use formulas from full factorial designs • Replace 2k with 2k-p

8. Preparing the Sign Table for a 2k-p Design • Prepare sign table for k-p factors • Assign remaining factors

9. Sign Table for k-p Factors • Same as table for experiment with k-p factors • I.e., 2(k-p) table • 2k-p rows and 2k-p columns • First column is I, contains all 1’s • Next k-p columns get k-p chosen factors • Rest (if any) are products of factors

10. Assigning Remaining Factors • 2k-p-(k-p)-1 product columns remain • Choose any p columns • Assign remaining p factors to them • Any others stay as-is, measuring interactions

11. Confounding • The confounding problem • An example of confounding • Confounding notation • Choices in fractional factorial design

12. The Confounding Problem • Fundamental to fractional factorial designs • Some effects produce combined influences • Limited experiments means only combination can be counted • Problem of combined influence is confounding • Inseparable effects called confounded

13. An Example of Confounding • Consider this 23-1 table: • Extend it with an AB column:

14. Analyzing theConfounding Example • Effect of C is same as that of AB: qC = (y1-y2-y3+y4)/4 qAB = (y1-y2-y3+y4)/4 • Formula for qC really gives combined effect: qC+qAB = (y1-y2-y3+y4)/4 • No way to separate qC from qAB • Not problem if qAB is known to be small

15. Confounding Notation • Previous confounding is denoted by equating confounded effects:C = AB • Other effects are also confounded in this design:A = BC, B = AC, C = AB, I = ABC • Last entry indicates ABC is confounded with overall mean, or q0

16. Choices in Fractional Factorial Design • Many fractional factorial designs possible • Chosen when assigning remaining p signs • 2p different designs exist for 2k-p experiments • Some designs better than others • Desirable to confound significant effects with insignificant ones • Usually means low-order with high-order

17. Algebra of Confounding • Rules of the algebra • Generator polynomials

18. Rules ofConfounding Algebra • Particular design can be characterized by single confounding • Traditionally, use I = wxyz... confounding • Others can be found by multiplying by various terms • I acts as unity (e.g., I times A is A) • Squared terms disappear (AB2C becomes AC)

19. Example:23-1 Confoundings • Design is characterized by I = ABC • Multiplying by A gives A = A2BC = BC • Multiplying by B, C, AB, AC, BC, and ABC:B = AB2C = AC, C = ABC2 = AB,AB = A2B2C = C, AC = A2BC2 = B,BC = AB2C2 = A, ABC = A2B2C2 = I • Note that only first line is unique in this case

20. Generator Polynomials • Polynomial I = wxyz... is called generator polynomialfor the confounding • A 2k-p design confounds 2p effects together • So generator polynomial has 2p terms • Can be found by considering interactions replaced in sign table

21. Example of FindingGenerator Polynomial • Consider 27-4 design • Sign table has 23 = 8 rows and columns • First 3 columns represent A, B, and C • Columns for D, E, F, and G replace AB, AC, BC, and ABC columns respectively • So confoundings are necessarily:D = AB, E = AC, F = BC, and G = ABC

22. Turning Basic Terms into Generator Polynomial • Basic confoundings are D = AB, E = AC, F = BC, and G = ABC • Multiply each equation by left side:I = ABD, I = ACE, I = BCF, and I = ABCGorI = ABD = ACE = BCF = ABCG

23. Finishing Generator Polynomial • Any subset of above terms also multiplies out to I • E.g., ABD times ACE = A2BCDE = BCDE • Expanding all possible combinations gives 16-term generator (book is wrong):I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG= ABCDEFG

24. Design Resolution • Definitions leading to resolution • Definition of resolution • Finding resolution • Choosing a resolution

25. Definitions Leadingto Resolution • Design is characterized by its resolution • Resolution measured by order of confounded effects • Order of effect is number of factors in it • E.g., I is order 0, ABCD is order 4 • Order of confounding is sum of effect orders • E.g., AB = CDE would be of order 5

26. Definition of Resolution • Resolution is minimum order of any confounding in design • Denoted by uppercase Roman numerals • E.g, 25-1 with resolution of 3 is called RIII • Or more compactly,

27. Finding Resolution • Find minimum order of effects confounded with mean • I.e., search generator polynomial • Consider earlier example:I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = ABDG= CEFG = ABCDEFG • So it’s an RIII design

28. Choosing a Resolution • Generally, higher resolution is better • Because usually higher-order interactions are smaller • Exception: when low-order interactions are known to be small • Then choose design that confounds those with important interactions • Even if resolution is lower

29. White Slide