1 / 28

Test Planning and Analysis

Test Planning and Analysis. Landing distance of an F-18. Test Planning and Analysis. Test Planning and Analysis. Test Planning and Analysis. What if 60% isn’t sufficient confidence? Typically 95% is the desired confidence. Need more data points! How many? Can’t say for sure.

aldan
Download Presentation

Test Planning and Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EMIS 7307 T&E Part 5 Test Planning and Analysis Landing distance of an F-18

  2. EMIS 7307 T&E Part 5 Test Planning and Analysis

  3. EMIS 7307 T&E Part 5 Test Planning and Analysis

  4. EMIS 7307 T&E Part 5 Test Planning and Analysis • What if 60% isn’t sufficient confidence? • Typically 95% is the desired confidence. • Need more data points! • How many? • Can’t say for sure. • Depends on s and x with the new data.

  5. EMIS 7307 T&E Part 5 Test Planning and Analysis Let’s assume that if we collect more data the standard deviation and (x -V*) don’t change. This not a very good assumption, but possibly the best we can do. Then we can rewrite tsas ts= k *√(n-1). With this assumption k = .351. For: n =2, ts= .351, α=.39, (1-α) = .61 n =5, ts= .702, α=.27, (1-α) = .73 n =10, ts= 1.053, α=.16 (1-α) = .84 n = 17, ts = 1.404, α=.09 (1-α) = .91 n = 26, ts = 1.755, α=.04 (1-α) = .96 n = 37, ts = 2.106, α=.015 (1-α) = .985

  6. EMIS 7307 T&E Part 5 Test Planning and Analysis • From the previous chart we get a sense of the “cost” of confidence. • For an increase of only three tests (from 2 to 5) we can boost our confidence by 12%. • But the change from 17 to 26 only further increased our confidence by 5%! • This illustrates why 90% confidence is often chosen as good enough. It’s a compromise of increased confidence versus cost.

  7. EMIS 7307 T&E Part 5 Test Planning and Analysis ANOVA

  8. EMIS 7307 T&E Part 5 Test Planning and Analysis

  9. EMIS 7307 T&E Part 5 Test Planning and Analysis ANOVA

  10. EMIS 7307 T&E Part 5 Test Planning and Analysis

  11. EMIS 7307 T&E Part 5 Test Planning and Analysis

  12. EMIS 7307 T&E Part 5 Test Planning and Analysis

  13. EMIS 7307 T&E Part 5 Test Planning and Analysis

  14. EMIS 7307 T&E Part 5 Test Planning and Analysis

  15. EMIS 7307 T&E Part 5 Test Planning and Analysis • In this example the table f value is 2.78. Use 4 and 25 as the degrees of freedom. • The calculated f is 4.3. • Therefore, reject the null hypothesis.

  16. EMIS 7307 T&E Part 5 Test Planning and Analysis

  17. EMIS 7307 T&E Part 5 Test Planning and Analysis • ANOVA is a powerful data analysis tool • Using a few samples answers the question, does an effect on a subgroup of a hypothetical population cause that subgroup to not really be from the hypothetical population? • We hypothesized that the five different aggregates were from a population that absorbed moisture similarly. • The ANOVA showed that they were not!

  18. EMIS 7307 T&E Part 5 Test Planning and Analysis • ANOVA is readily available in many spreadsheet packages. • Excel has it under tools then click on data analysis and select ANOVA.

  19. EMIS 7307 T&E Part 5 Test Planning and Analysis • Bottom line. • Use student t or normal distribution (n>30) when comparing to a spec. • Use ANOVA when looking to see if two groups are statistically the same to some confidence level.

  20. EMIS 7307 T&E Part 5 Test Planning and Analysis • Bottom line (cont’d) • For both test planning and data analysis at least one member of the team needs to be statistics savvy. • From a management perspective. • This facilitates planning for adequate data collection to meet the desired confidence. • Assures data collection ends when sufficient data exist.

  21. EMIS 7307 T&E Part 5 Special TopicSoftware testing • In an earlier lecture I presented the results of a study showing the economic impact of poor or limited software testing. • I’ve also made the case that earlier problem finding and fixing is exponentially cheaper. • Here I want to introduce several software testing concepts.

  22. EMIS 7307 T&E Part 5 Percent of Specification Requirements Involving Software F-22 B-2 F-16 F-15 F-111 A-7 F-4 Ref: Lockheed - Martin Corp.

  23. EMIS 7307 T&E Part 5 Special TopicSoftware testing • Human testing is defined as informal, noncomputer-based methods of evaluating architectures, designs and interfaces. It can consist of: • • Inspections – The programmer explains his/her work to a small group of peers with discussion and direct feedback on errors, inconsistencies and omissions. • • Walk-through – A group of peers develop test cases to evaluate work to date and give direct feedback to the programmer. • • Desk Checking – A self evaluation is made by the programmer of his/her own work. There is a low probability of identifying his/her errors of logic or coding.

  24. EMIS 7307 T&E Part 5 Special TopicSoftware testing • • Peer Ratings – Mutually supportive, anonymous reviews are performed by groups of peers with collaborative evaluations and feedback. • • Design Reviews – Preliminary design reviews (PDRs) and critical design reviews (CDRs) provide milestones in the development efforts that review development and evaluations to date.

  25. EMIS 7307 T&E Part 5 Special TopicSoftware testing • Computerized software testing • Black box • Functional testing of a software unit without knowledge of how the internal structure or logic will process the input to obtain the specified output. Within-boundary and out-of-boundary stimulants test the software’s ability to handle abnormal events. Most likely cases are tested to provide a reasonable assurance that the software will demonstrate specified performance. Even the simplest software designs rapidly exceed capacity to test all alternatives.

  26. EMIS 7307 T&E Part 5 Special TopicSoftware testing • Computerized software testing • White box • Structural testing of the internal logic and software structure provides an opportunity for more extensive identification and testing of critical paths. The process and objectives are otherwise very similar to black box testing.

  27. EMIS 7307 T&E Part 5 Special TopicSoftware testing • IV&V • Independent verification and validation are risk-reducing techniques that are applied to major software development efforts. The primary purpose of IV&V is to ensure that software meets requirements and is reliable and maintainable. The IV&V is effective only if implemented early in the software development schedule. Requirements analysis and risk assessment are the most critical activities performed by IV&V organizations.

  28. EMIS 7307 T&E Part 5 Special TopicSoftware testing • IV&V • The desire to boast of a high SEI rating in proposals and to generally improve the corporations efficiency has moved many companies to significantly improved software development capabilities. • This in turn has reduced the perceived need by customers to require and pay for IV&V.

More Related