1 / 59

Part 4

Design and Execution. Part 4. Course Outline. Day 1 Day 2. Part 0: Student Introduction Paper Helicopter - Pt 0 Use what you know Part 1: DOE Introduction What is a Designed Experiment? Part 2: Planning Understand the test item’s process from start to finish

brita
Download Presentation

Part 4

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Design and Execution Part 4

  2. Course Outline Day 1 Day 2 • Part 0: Student Introduction • Paper Helicopter - Pt 0 Use what you know • Part 1: DOE Introduction • What is a Designed Experiment? • Part 2: Planning • Understand the test item’s process from start to finish • Identify Test Objective – screen, characterize, optimize, compare • Response variables • Identify key factors affecting performance • Paper Helicopter - Pt 1 Planning • Part 3: Hypothesis Testing • Random variables • Understanding hypothesis testing • Demonstrate a hypothesis test • Sample size, risk, and constraints • Seatwork Exercise 1 – Hang time Measurements • Part 4: Design and Execution • Part 4: Design and Execution • Understanding a test matrix • Choose the test space – set levels • Factorials and fractional factorials • Execution – randomization and blocking • In-Class F-18 LEX Study - Design Build • Paper Helicopter - Pt 2 Design for Power • Part 5: Analysis • Regression model building • ANOVA • Interpreting results – assess results, redesign and plan further tests • Optimization • In-Class F-18 LEX Study - Analysis • Seatwork Exercise 2 – NASCAR • Paper Helicopter – Pt 3 Execute and Analyze • Paper Helicopter – Pt 4 Multiple Response Optimization • Part 6: Case Studies

  3. Science of Test IV Metrics of Note Plan Sequentially for Discovery Factors, Responses and Levels Design with Confidence and Power to Span the Battlespace N, a, Power, Test Matrices Analyze Statistically to Model Performance Model, Predictions, Bounds DOE Execute to Control Uncertainty Randomize, Block, Replicate

  4. Understanding a Test Matrix • Confounding:Two factors are confounded when they vary in the same pattern in the test matrix such that their effects cannot be separated • Example: Throwing Darts (situation) Two friends Matt and Aaron are at a bar and want to see who’s better at throwing darts. Matt and Aaron can throw 4 darts before they leave.(Sample size is established and limited by time.) Matt stands 20 ft away and throws 2 darts. Aaron starts to throw his first dart and then loses his balance and takes a few steps forward (10 ft) and throws both of his darts. Both are asked to leave. (Test is executed.) Both Matt’s darts hit about 10 inches from the bull's-eye and Aaron’s about 5 inches away. (Measurements completed.)

  5. Understanding a Test Matrix • Example: Throwing Darts (Analysis) • Aaron Concludes that he’s the better thrower because his shots landed closer • Matt Concludes that the test was unfair since Aaron stumbled forward and threw from a closer distance

  6. Confounding • Example: Throwing Darts (confounding) • True answer is you don’t know who’s the better dart player because the potential effect of dart thrower is confounded with the factor of distance thrown • Each Player should have thrown one dart from each of the two distances away

  7. Factor Levels - Factorial • Example: Throwing Darts (terminology) • Factors • Player--two levels (Matt and Aaron) • Distance--two levels (10ft and 20ft) • All possible combinations = product of all the levels of each factor • 2 x 2 = 4 • The resulting combinations are called a factorial matrix

  8. Effects – Linear only • Main Effect: A statistically significant change in the response (points) occurs when changing the levels of a factor • Aaron scores better than Matt; Scores decrease as distance increases

  9. Effects – Interaction • Interaction: The failure of one factor (player) to have a similar effect on the response (points) at different levels of another factor (distance) • One factor level, Matt, scores better when distance changes, but Aaron is unaffected by distance

  10. Factorial Basics • What is a factorial design? • Factors varied together (not one at a time) • Factors varied to cover all possible combinations • Factorial designs eliminate confounding, include interactions • Typically generalized with letter identifiers for factors (A,B,C…) and signs for highs (+) and lows (-) • Building the factorial follows a simple pattern which we call standard order - - - - - + - + - + - + + - - + + - - + + + + +

  11. (-1,1,1) (1,1,1) (-1,1,-1) (1,1,-1) B + (-1,-1,1) (1,-1,1) C + B - C - (-1,-1,-1) (1,-1,-1) A - A + Factorial Basics • Factorial geometry: Sometimes useful to visualize factorial designs as mapping all corners of a geometric shape

  12. Coding Main Effects Interaction Terms • Factorial designs are often coded with -1 and +1 to represent low and high • Enables simple analysis • Allows us to see setting of interaction terms and recognize confounding • Interaction levels are determined by the products of their main effects • If any two columns are the same or exactly opposite, they are confounded

  13. Maverick AGM Tests Constant: F16C, LAU-88, Force correlate track Cloud cover Sun orientation INPUTS (Factors) OUTPUTS (Responses) PROCESS: Miss Distance (ft) Missile Variant Ground Range Impact Angle Error (deg) Maverick AGM Launch Altitude Impact Velocity Error (deg) Attack Airspeed Noise

  14. Understanding a Test Matrix • 4 Factors each at 2 levels is 24 combinations = 16 runs required Mav Mav Mav • Maverick AGM key factors: • Missile Type • Launch Ground Range • Launch Altitude • Launch Airspeed

  15. Centers to check for curvature Understanding a Test Matrix • 2-level factorial design structure: efficient, symmetric, powerful + Altitude + _ Range _ _ + Airspeed  + Variant

  16. Checking for Curvature • Using the center point response • Compare to the model predicted response • Significant differences mean linear model is inadequate y Test to evaluate this distance 0 +1 -1

  17. Test Matrix Randomized - Run Order Standardized

  18. Maverick AGM (3 Factors Weapon, Range, Altitude) • Real World Matrix is Coded: • All High Values of xi’s = +1 • All Low Values = -1 Coding Example R E S P O N S E R E S P O N S E Mav

  19. Design and Model Matrix Model = Linear effects + interactions Design

  20. Confidence and Power: Factorials

  21. Understanding a Test Matrix • Example: Maverick AGM • Why only 2 levels to each factor? • What if there is no effect across the factor? • Could have used those runs to look at something else • Need at least 2 levels to a factor to see an effect • Why not 3 or 4 for each? • More than 2 levels allows tests for curvature • More than 2 levels causes expensive growth of matrix (2X2X2=8, 2X2X3=12, 2X2X4=24) • More difficult to separate effects • Setting matrix up with all combinations of all levels give orthogonal design • Orthogonalityallows us to easily see the effect of each factor and interaction and collapse across factors that have no significant effect

  22. Choose a Test Space Maverick AGM vB Mav Maverick AGM vA & vB Test Points Maverick AGM vA&B Orthogonal design

  23. At this point we have introduced: • Sample size and appropriate risk • Test matrix design • The factorial points • The addition of center points • Now it’s time to execute our design • The order we execute the runs in is random • Protects against: • “Background” Things that we know are happening in the background but can’t control • “Unknowns” Things that we don’t know are happening in the background and can’t control – often called Lurking Variables Execution

  24. Example: By randomizing run order, we average the effects of a transducer error that is growing with time during a wind tunnel airfoil test Randomization Example Lift Coefficient vs. Angle of Attack 1.6 1.4 1.2 1 CL 0.8 Sequential Random 0.6 TRUE 0.4 0.2 0 0 2 4 6 8 10 12 14 16 18 A-o-A

  25. Blocking Example • By blocking on days and randomizing within blocks, facility related nuisance factors still effect results but now we can quantify the effect and remove it

  26. Execution (Blocking) • Use the highest order interaction • How do you decide which runs go in which blocks? • If we had to divide the matrix below into two blocks…what do we do? • Code the matrix with -1’s and 1’s and determine interaction levels • Put all -1’s of highest order interaction in first block and all 1’s in second • Confounds the block with the interaction, but because 3-ways are rare we ignore it Randomize run order within blocks!

  27. Treating Factors Outside the Matrix: • What happens to all the other Factors we can’t fit into our “Factorial Matrix”? 1. Treat effects as noise during the test, i.e. ignore them (N) careful • All effects that aren’t in the matrix are interpreted by the mathematics as random noise by default • If something not in the matrix is effecting the response variable we won’t know 2. Hold them constant during the test (C) 3. Block them and randomize within blocks • Protects against background effects • Protects against nuisance factors unknown unknowns 4. Fractionate your factorial matrix and make it a “fractional factorial matrix”

  28. Fractionating • When constraints prevent us from doing all the runs in a factorial, how do we determine which runs to do? • Fractionate • Method of selecting a portion of the runs that minimizes confounding issues • Determined similar to determining which runs to put into blocks • Confound the highest order interactions possible • Fraction name corresponds to number of runs accomplished (half fraction, do ½ of the runs, quarter-fraction, do ¼ of the runs)

  29. All Possible Combinations or Fraction • Consider a four factor full factorial design (16 runs ) • Suppose we can’t afford 16 but only 8 • Choose blocks confounded with ABCD (choose – signs here as the ½ fraction)

  30. All Possible Combinations or Fraction D – + + C + B - - - + A • ½ fraction of full factorial • Design geometry

  31. So what do we give up ? • Fractionated designs • Have reduced power compared to full factorials • Confounded effects • For instance, the previous example is known as a resolution IV design and has confounded pairs of two factor interactions (also called alias chains)

  32. Fractions and Sequential Build:Recipe for Success • If the fraction you run doesn't allow you to sort out the effects and model, run another fraction • It is always advisable to run only a subset (say 25-50%) of the total number of tests in the first experiment • Learning takes place after each set of tests • You may need to change the factors or factor levels • You may need additional runs to sort out the influential factors • You might need to validate your findings • You might be essentially done, and only need a few more tests!

  33. C B A 26 Full Factorial Geometry F – + – + D – E +

  34. 26-2 Fractional Geometry and a Foldover – Sequential Design C B A original augment* F – + – + D – center E + *complete foldover on A

  35. Confidence and Power: Factorials

  36. Session 4: Test design and Execution • Understanding a test matrix • Avoiding confounding • Include tests for interactions • Choose the test space – set levels: 2 is often efficient • Factorials • Execution – randomization and blocking • Fractional factorials Session 4: Summary

  37. Example 1 Design:Maverick AGM

  38. Example: Maverick AGM vB • The Air Force wants to field a new air-to-ground missile. Early testing has shown that the missile often lands either well short or long of the target. • The new Maverick AGM vB variant must perform as well as or better than the legacy Maverick AGM vA model. The Maverick AGM vB supply is limited.

  39. Identify Test Objective • What are we testing? Are we… • screening, characterizing, optimizing, comparing? • Once we decide what we’re testing, we have to pick our MOPs • Measure of Performance (MOP): a.k.a Response Variable • Variables which indicate the quality of the item or process under test.

  40. Response Variables (MOPs) Types of Measures of Performance (MOPs): • Categorical: • Discrete, category but no order (e.g. missile types, ECM types) • Ordinal: • Discrete, category but order determines value (e.g. survey data) • Interval: • Continuous, but zero and ratios have no real world meaning (e.g. temperature, dates) • Ratio: • Continuous, zero has a real meaning, ratios are relevant (e.g. miss distance) Continuous data are better!

  41. Response Variable (MOPs) Maverick AGM Measures

  42. Identify Key Factors Cause and Effect Diagram: Maverick AGM

  43. Treating Other Factors • Allow some factors to vary as noise (N) during test • All effects that aren’t in the matrix are interpreted by the mathematics as random noise by default • Factors outside the matrix potentially affecting the response • Examples: Cloud cover, Pod performance, Experience, Tail number • Hold constant (C) during test • Understand the implication of such a decision • Inference only for specific settings of that factor • Examples: Platform, Rail, Missile mode

  44. Maverick AGM Tests Constant: F16C, LAU-88, Force correlate track Cloud cover Sun orientation INPUTS (Factors) OUTPUTS (Responses) PROCESS: Miss Distance (ft) Missile Variant Ground Range Impact Angle Error (deg) Maverick AGM Launch Altitude Impact Velocity Error (deg) Attack Airspeed Noise

  45. Understanding a Test Matrix • 4 Factors each at 2 levels is 24 combinations = 16 runs required Maverick Maverick Maverick • Maverick AGM key factors: • Missile Type • Launch Ground Range • Launch Altitude • Launch Airspeed

  46. Centers to check for curvature Understanding a Test Matrix • 2-level factorial design structure: efficient, symmetric, powerful + Altitude + _ Range _ _ + Airspeed  + Variant

  47. Test Matrix Randomized - Run Order Standardized

  48. Maverick AGM (3 Factors Weapon, Range, Altitude) • Real World Matrix is Coded: • All High Values of xi’s = +1 • All Low Values = -1 Coding Example R E S P O N S E R E S P O N S E Maverick

  49. Design and Model Matrix Model = Linear effects + interactions Design

  50. Design Build using Software:Maverick example

More Related