Testing Tools

1. Testing Tools Software Testing and Verification Lecture 14 Prepared by Stephen M. Thebaut, Ph.D. University of Florida

2. Overview of Topics • Why is test automation desirable? • Types of test automation • test design tools • test implementation (including “testing frameworks”) tools • test execution and other run-time tools • Sources for testing tools • One other “Testing Tool:” Test Plans and Planning

3. Why is test automation desirable? • A recent survey result reported in SoftwareTech News† (a DoD publication) confirms that software testing activity consumes a significant fraction of the overall development lifecycle. †October 2007 Vol. 10, Number 3

4. Why is test automation desirable? (cont’d) • In the same article, a study undertaken by the US Navy showed that substantial time savings can be achieved via automated testing.

5. Other benefits of automated testing • Some types of testing are not feasible without automation, e.g.: • load (“concurrency”) testing • long term “soak” testing to detect memory leaks/performance degradation • More efficient build verification (“smoke”) testing • Stand-alone “lights out” testing • Increased repeatability • Testers can focus on more creative tasks • Much higher levels of coverage

6. Test Design Tools

7. Test Design Tools • Parameter value generators • Boundary value, equivalence class, and special value generators • Random data generation based on probabilistic CFGs (Context-Free Grammars) • Fault tree coverage • Cause-Effect analysis • Database test data generators (for performance, load, and usability test) (cont’d)

8. Test Design Tools (cont’d) • Statistical test generators (for reliability, availability, and performance testing) • Structural test case design tools: • utilize code instrumentation (see Run-Time Tools) • source code specific • may incorporate symbolic execution capabilities to aid in path sensitization

9. Test Implementation Tools

10. Test Implementation Tools • Test Case Scripting Languages: • test scripts automate the steps involved in carrying out tests • suitable for tests that will be repeated (i.e., in support of regression testing) • scripts are maintained in a change-controlled Test Database (cont’d)

11. Test Implementation Tools (cont’d) • Test Harness (scaffolding) Generators: • automates development of test drivers, stubs, and oracles (“assertions”) • often coupled with a test scripting capability (= “testing frameworks”) • useful for module and integration test

12. Run-Time Tools

13. Run-Time Tools • Code Instrumentation: • inserts additional code to • monitor cumulative logic/dataflow coverage, • support trace and debug functions, and • detect run-time errors. • aids in designing test cases to meet white-box coverage goals or to debug code. • alters execution performance and may mask or introduce timing errors. • many commercial (source code specific) tools available. • among the oldest of testing tool types. (cont’d)

14. Let’s Pause for a Moment… Imagine that it’s summertime and that a 3-day weekend is just starting… Wouldn’t it be great to just grab a fishin’ pole and head on out to the lake!

15. Let’s Pause for a Moment… (Take 2) There’s NOTHING like Paris in the the spring!

16. Run-Time Tools (cont’d) • Data / Output Comparators • used to automatically compare output or data files for discrepancies • effective in dealing with the “Paris in the the spring” problem† • most are not “smart enough” to ignore “don’t care” data fields. • useful for regression testing when combined with a keystroke recorder and playback tool… †Also know as inattentional or perceptual blindness. (cont’d)

17. Run-Time Tools (cont’d) • Keystroke Recorder and Playback Tools • records and plays-back keystrokes, pointer movement, and mouse clicks • useful for regression testing when combined with a data/output comparator

18. Sources for Tools • An excellent source of information about testing tools is the WWW. Use your favorite search engine to find sites associated with “software testing tools” (about 94 million Google hits at last count). • Several QA and testing related magazines, newsletters, and trade journals are available which publish tool reviews and comparisons. (cont’d)

19. Sources for Tools (cont’d) • Relevant links can also be found at the websites of several QA and testing related organizations and societies, e.g., • SEI • IEEE • ACM • American Society for Quality • Society for Software Quality • Quality Assurance Institute • European Organization for Quality (cont’d)

20. Sources for Tools (cont’d) • Some useful testing tools-related links: • www.opensourcetesting.org • www.testingfaqs.org • www.qaforums.com

21. Test Plans and Planning Pezze and Young, Software Testing and Analysis, Wiley, 2008

22. Motivation • Planning – just as important in testing as in any other engineering activity. • Allows for the effective use of limited resources and management of activities. • In a nutshell, it involves a disciplined, timely consideration of: • What needs to be done, • When and How it will be done, and • What resources (including time and effort) will be required.

23. The Master Test Plan • Also called a Comprehensive Test Plan. • Must be compatible with overall project plan. • Developed early– during project planning and requirements stages. • Defines requirements for lower level plans: • Unit Test Plan • Component Test Plan • Product Test Plan • System Test Plan

24. Master Test Plan Check-List • Defines test stages(e.g., unit, integration, system)schedules, and responsibilities. • Establishes objectives, strategies, and procedures. • Identifies requiredtools, facilities, and test libraries. • Sets criteria for test completion and success. • Defines the integration teststrategy and iden-tifies all scaffolding required with respon-sibilities and schedules for its production. (cont’d)

25. Master Test Plan Check-List (cont’d) • Defines test coverage goals and methods for tracking progress. • Establishes a regression test policy. • Specifies required hardware and software, particularly non-standard or special purpose elements. • Identifies estimated resource requirementsincluding machine time, disk storage, physical / logical device access, etc. • Identifies training required. • Specifies test entry and exit criteria.

26. Typical Test Entry Criteria • Test plan is complete and has been reviewed • Design and implementation of test cases reviewed and all problems fixed • Drivers, stubs, oracles in place • Test environment (hardware and software) is ready

27. Typical Test Exit Criteria • All problems discovered have been recorded and fixed • All fixes have been re-tested • Regression test(s) complete • All test cases and associated documentation updated as necessary

28. IEEE Test Plan Template • Introduction and references • Test items (versions, media, references, etc.) • Features to be tested • Features NOT to be tested (and reasons) • Testing strategies and approach • Dependencies • Test case success/failure criteria • Pass/fail criteria for the complete test • Test entry/exit criteria (cont’d)

29. IEEE Test Plan Template (cont’d) • Test deliverables/status communication vehicles • Testing tasks • Hardware and software requirements • Problem determination and correction responsibilities • Staffing and training needs/assignments • Test schedules • Risks and contingencies • Approvals

30. Test Planning Summary • Test planning allows for the effective use of limited resources and the effective manage-ment of test activities. • A plan is only as good as the planning that goes into it!

31. Testing Tools Software Testing and Verification Lecture 14 Prepared by Stephen M. Thebaut, Ph.D. University of Florida