Introduction to Testability Techniques

1. CSSE 377 – Intro to Testability Steve Chenoweth Tuesday, 10/04/11 Week 5, Day 2 Right – Typical tool for reading out error codes logged by your car’s computer, to help analyze its problems. From http://niceparts.blogspot.com/2007_12_01_archive.html.

2. Today • How to do Project 4… • We’ll get your “implementers” a bit more involved this time • Software testability – this  • Bass’s Ch 4, pp 88-90 (Testability scenarios) & Ch 5, pp. 118-121 (Testability tactics) • Lots of discussion also in CSSE 376! Q 1

3. Coming up… • Thursday – (Steve’s at Purdue) • Guest speakers – Matt Ellis, Justin Hutchings of Microsoft • Remaining time – if any – Work on first turnin for Testability • Friday – More on this & time to work on Project 4 – including… • Getting your “implementers” involved – first talk with them about “what should be more testable” on your system, then have them try to run tests themselves. • Monday: Analyzing archs & the ATAM / CBAM

4. We next pick testability from Bass’s QA list… • Bass’s list of six, from the inside back cover of his book: • Availability • Modifiability • Performance • Security • Testability • Usability

5. And here’s the project about it: Overall - On the same project, • Use an arch tactic to make it more testable: • Determine what is not very testable in your current project system. • Implement a tactic to improve this by a predicted amount! • Have your implementers observe the “before” and “after” testing, and confirm that there’s an improvement • We’re also going to continue our study of how to document the arch, so: • An added feature of this project – add one more section to your draft of that, for your sys.

6. And a first step to take: • Try to think of a problem area in testing your current system. • Run through this area of tests, of your system as it exists now. Time how long this takes, and find a way to measure how “effective” (thorough) it is. • See if you can get your “implementers” to repeat the same tests. • Make a “prediction” about how fast or effective this testing will be with the testing change you plan to make. • Turn in your ideas, those “existing times,” and other measures, with your prediction, in your “team journal” by Friday at 11:55 PM. Next step will then be – Make the changes to testing that will speed this up or make it more effective.

7. What’s Bass say about this QA? • Problem – Testing needs to be more efficient & effective • It consumes a high percentage of software development time & effort • The heuristic is “half” • Goal – Allow easier testing when an increment of software development is completed. • Motivation – The overall arch of the system, and of its testing methods, can help make testing easier • Scenarios – What’s in “The Notes” at the end of the supplementary spec template? • What is Testability “about” – Ch 4? • What are some good tactics – Ch 5? Q 2

8. Bass’s testability scenarios • Source: Unit developer • Increment integrator • System verifier • Client acceptance tester • System user • Stimulus: Analysis, architecture, design, class, subsystem integration completed; system delivered • Artifact: Piece of design, piece of code, complete application • Environment: At design time, at development time, at compile time, at deployment time • Response: Provides access to state values; provides computed values; prepares test environment • Response Measures: (examples) • Percent executable statements executed • Probability of failure if fault exists • Time to perform tests • Length of longest dependency chain in a test • Length of time to prepare test environment Q 3

9. Example scenario • Source: Unit tester • Stimulus: Performs unit test • Artifact: Component of the system • Environment: At the completion of the component • Response: Component has interface for controlling behavior, and output of the component is observable • Response Measure: Path coverage of 85% is achieved within 3 hours

10. Testability situations It’s a development-time attribute: • Probability of fault discovery • Need to control components • Need to observe component failure Right: Ren & Stimpy tell why people ignore system testability. From web site sctest.cse.ucsc.edu/ . Ch 4

11. Tactics to achieve testability • Manage Input/Output • Record/playback • Separate interface from implementation • Specialize access • Internal monitoring • Built-in monitors • Generally, the goal is to automate as much testing as you can – why? Ch 5 Q 4

12. Examples - 1 • Test harness – like the stimulator you might have done for performance • Automates some aspect of testing • But, for finding errors, the test cases need to be more complete (unless performance / reliability are what you’re testing!) • Can be for internal classes, etc., as well as external – See next slide  • Give this itself an “architecture” with options, different modes, etc. Q 5

13. Examples – 1, cntd • Typical test harness: • In MVC, replace the “View” with a stimulator that fires test cases at the rest of the system: Replace with stimulator, for testing

14. Examples – 1, cntd • How about stimulating the GUI? • A slightly harder trick, replacing the human with the mouse and keyboard. How do we test the GUI itself? (Or the whole System!?) See list of open source tools at http://en.wikipedia.org/wiki/List_of_GUI_testing_tools

15. Examples - 2 • Built-in monitors – the system itself is “instrumented” to provide test results. • Usually, this can be turned on/off so it doesn’t interfere with normal operations. • May have multiple special modes. • The “heart” of white-box testing!

16. Industrial strength tools • Good example – IBM’s Rational PurifyPlus • Shows code coverage • Analyzes memory problems (e.g., buffer overflow, leaks) • Identifies performance issues like bottlenecks • Runs on Linux, Unix, Windows See http://www-01.ibm.com/software/awdtools/purify/?cn=agus_rtnlrp-20100514&cm=k&csr=google&cr=rational_purify&ct=AGRAK605&ck=rational_purify&mkwid=s28L9ro6h_5281300654_432dt32930&cmp=109HG

17. Project 4 – Testability (overall) Overall theme (of the testability part of the project) – • Pick an area where you want to improve testing. • Pick a strategy to improve it, using one of the above tactics, and verify (Friday) with your implementers. The improvement can be in either of these two areas: • Effectiveness (thoroughness) of the testing, or • Speed of the testing, with the same effectiveness. • Try it “as-is” to verify how long it takes and the coverage. • Includes letting novice testers, your “implementers,” try it. • Guess how much you can improve that with your change. • Make the change to improve testability. • Run the testing again, with the change. Measure how fast and how effective the new testing is. Let your implementers verify the differences. • Report on the results.