Trend Quality Assurance

1. Trend Quality Assurance Edward Tsai　蔡木本 趨勢科技全球軟體研發部協理 April 22, 2003

2. Contents • Software quality assurance overview • Testing skill and debug tools • Testing concepts. • Testing methodology. • Testing tools. • The importance of QA Automation • Software testing in enterprise environment

3. Overview • What is the quality? • What is QA role in Trend? • How many different type of testing strategy? • What is the typical software life-cycle? • What is the QA challenge and future? • What is Trend QA organization?

4. What is the quality? • 2 level of quality: • Level 1: product defect rate and reliability. • Level 2: product quality, process quality, customer satisfaction. • Quality from professional views: • Conformance to requirement. • Fitness for use. • For example: • Cadillac and Chevrolet car, Which has good quality?

5. What is QA role in Trend? • The purpose of QA engineers: • To discover all possible errors in development cycle. • To discover every conceivable fault or weakness in product. • To simulate customer’s focus and need. • To measure the quality level of product. • To delivers the quality report to service team.

6. How many different type testing strategy? • Black-Box testing: • Derived from the functional design spec, without touching the internal program structure. • White-box (or Glass-box) testing: • Require knowledge of the internal program structure and derived from the internal design spec or code. • Trendmicro: • Black-box testing and will migrate to Gray-box testing.

7. What is the typical software life-cycle? • Concept. • Requirement. • Design. • Implementation or coding. • Test. • Operation and maintenance.

8. DOT-U model Code and spec modification 1 Requirements specification 1 11 System and Acceptance validation Requirements verification Products simulation Functional design specification Code and spec modification 3 9 2 10 Usability test Function validation Functional Design verification 6 Internal design specification Code Code and spec modification Code and spec modification 4 5 7 8 Internal design verification Code verification Unit validation Integration validation

9. What is Trendmicro software life-cycle? • Requirement. • High-Level design/Planning. • Detailed design. • Construction. • Pre-release. • Closure. • Service.

10. What is QA challenge/future? • Professional discipline requiring trained, skilled people. • Advances in technology/tools to create complex software in GUI, Client-Server, Intranet/Internet architecture. • Multi-platform, multi-language. • Many/complex compatibility software testing. • Keep on process improvement. • Test automation is not mature yet.

11. Defect Distribution

12. The cost of defect Cost Alpha Beta Release Time

13. Tester-to-Developer Ratios • The lesson: • The importance of testing is now recognized. • Vendors are willing to pay for proper testing. • A dramatic shift: • Historically, for Mainframes : 1: 5-10. • At Microsoft, 1992 : 2:3. • At Microsoft, 1995 : 1:1. • At Microsoft, 2000 : 2:1. • At Microsoft, 2005 : ? • TrendMicro, 2000 : 1 : 2 • TrendMicro, 2003 : 1 : 1

14. What is the Trend QA organization?

15. Testing skill and debug tools • Testing methodology. • Test case design. • Testing tools.

16. Testing methodology • Verification testing: • Reference to Dotted-U Model. • Verification methods : Inspections, walkthroughs, Technical reviews, others. • The verification tool : Checklists. • Include: Requirement, Architecture, Plans, Design Spec, code, Documentation. • Validation testing: • A process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirement. • Include: Unit testing, integration testing, Usability testing, Function testing, Regression testing, System testing, Acceptance testing (Alpha, Beta testing).

17. Testing case design • Go through all testing synopsis at every checking point. • Task-Oriented Functional Test (TOFT) • Forced-Error Test (FET) • Boundary test • Volume Test • Stress Test • Performance Test • Security Test. • Configuration Compatibility Test • Documentation Test • Trend Standards in Product

18. Test Coverage • Test coverage has three components: • Requirements coverage. • Function coverage. • Logic coverage.

19. Example

20. Example • Input a Directory name to install a product in NT system. • Task-Oriented Functional Test (TOFT) • Input a legal directory name with length between 1 to 255 character. • Forced-Error Test (FET) • Read privilege only directory, Non-existed drive, more than 255 characters, Not enough disk space. • Boundary test • Input a space or null character, Input one character, Input 255 characters, Input 266 characters. Memory leak after install complete. • Volume Test • Install it with 30 MB free disk space. • Stress Test • Install it with CPU utilization more than 70%, Free visual Memory left less than 1 MB, ... • Performance Test • Memory usage, CPU utilization, Any network traffic generated,...

21. Example • Security Test. • Does it can be installed from every users or Administrator only. • Configuration Compatibility Test • N/A • Documentation Test • The description are properly and correctly • Trend Standards in Product • Does it install under c:\programs files\TrendMicro\PC-cillin 98?

22. Testing tools • Testing documents database: • Test manager, TCMT. • Bugs tracker system: • PVCS Tracker 7.0. • Automation tools: • Visual test 4.0, Robot J, ATP. Winrunner. • Source code Coverage: • PureCoverage. • System tools: • Loadrunner, Load test, Winbench. • Debug tools: • Boundchecker, Quantity, Purify, SoftIce.

23. Q & A