Software Testing

Software Testing Test Early, Test Often CS673 Fazle Khan Nov 1, 2011

What Is Software Testing? • Book: A validation process, the purpose of which is to detect as many defects of as high a level of seriousness as possible. • Wikipedia:An investigation conducted to provide stakeholders with information about the quality of the product or service under test

Planning • What type of testing is needed for your application? • When are the tests a going to be executed • Where are the tests going to be executed • How to structure an application to perform each of the selected test types? Here is where testing directly impacts software design • Metrics • Test Success/Failure Count . A zero failure policy is the best choice • Code Coverage • Defects Per Hour

Scope • Important to define upfront how broad testing effort will be. • Defined from the Requirements Document • Web applications should define their supported browsers with versions • Multi-platform application need to have their platform specified • Failure to properly scope the testing effort will result in a mix of a missed deadlines, buggy code, and cost overruns

Who Tests What?An example

Black Box Testing • Wikipedia • A method of software testing that tests the functionality of an application as opposed to its internal structures or workings • Specific knowledge of the application's code/internal structure and programming knowledge in general is not required. • Test cases are built around specifications and requirements, i.e., what the application is supposed to do. • It uses external descriptions of the software, including specifications, requirements, and designs to derive test cases. • These tests can be functional or non-functional, though usually functional. • The test designer selects valid and invalid inputs and determines the correct output.

White Box Testing • Wikipedia • A method of testing software that tests internal structures or workings of an application, as opposed to its functionality • In white-box testing an internal perspective of the system, as well as programming skills, are required and used to design test cases. • The tester chooses inputs to exercise paths through the code and determine the appropriate outputs.

Unit Testing • Wikipedia: • a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application • In procedural programming a unit may be an individual function or procedure. • In object-oriented programming a unit is usually an interface, such as a class (method) • Ideally, each test case is independent from the others: substitutes like method stubs, mock objects, fakes and test harnesses can be used to assist testing a module in isolation. • Java Tools and techniques • JUnit • EasyMock • Dependency Injection / Inversion of Control (IoC) design pattern • Critical for any non-compiled/typed languages to maintain API through iterations

Sample JUnit codeException testing @Test public void testMatchNullInstrumentTest() {MatcherThread x = new MatcherThread("hi", 1L);TransactionMakerHoldertmh = new TransactionMakerHolder();tmh.setTransactionMaker(new TransactionMaker() { @Override public Object transact(TransactionMakerWorkertransactionMakerWorker) { return transactionMakerWorker.work(); } });x.setTransactionMakerHolder(tmh);x.setInstrumentDAO(new InstrumentDAO() { @Override @Nullable public Instrument exclusiveLockAndLoad(Long instrumentId) { return null; } }); try {x.match();Assert.fail(); } catch (RuntimeException r) {Assert.assertNotNull(r); } }

Module Testing • Exercise methods in combination • 2-5, typically • Test most common sequences first • Include sequences likely to cause defects • Focus test on each attribute • Initialize, then execute method sequences that affect it • Verify that class invariants are unchanged • Verify invariant true with initial values • Execute a method sequence • Verify invariant still true • Verify that objects transition among expected states • Plan the state/transition event • Set up the object in the initial state by setting variables • Execute event and check that transition occurred

Sample EasyMock Code public void testAddAndChangeDocument() { Collaborator mock = EasyMock.createMock(Collaborator.class); ClassUnderTestclassUnderTest = new ClassUnderTest(); classUnderTest.addListener(mock); //Mock Dependency injected! mock.documentAdded("Document"); mock.documentChanged("Document"); EasyMock.expectLastCall().times(3); EasyMock.replay(mock); classUnderTest.addDocument("Document", new byte[0]); classUnderTest.addDocument("Document", new byte[0]); classUnderTest.addDocument("Document", new byte[0]); classUnderTest.addDocument("Document", new byte[0]); verify(mock); }

Integration Testing • Big Bang Integration (so y2k) • Bring up all modules up at once for testing • Promotes systems with circular dependencies among modules • Only option for legacy systems with existing circular dependencies without a rewrite • Incremental Integration • Modules are layered on top of other modules, dependencies run in one direction • Bottom-Up Integration • lowest level module (for example DAO class) is written and tested first. • then modules dependant on the previous module are coded and tested with the previous module still in context • Top-Down Integration • Highest level module with the most valuable business logic is written first • The dependencies of the first module are stubbed out initially and are implemented later in the development cycle. • Usually done with GUIs • Continuous Integration • Small amounts of code are added to the baseline application on a very frequent basis • As long as code is unit-tested and does not introduce errors to the base line it is integrated into the base-line • Java Tools • Any continuous integration build server, for example Teamcityor CruiseControl

System Testing • When possible performed on the required environment • Functional Tests focus on how the customer will use the product in real life • Acceptance Tests • Non-Functional Testing

Non-Functional System Testing • Performance • The system’s through put • Load/Stress Testing • Under what load does the system start to break • Reliability and Availability • How long between failures • Recoverability • Can the system be restarted after an unexpected failure • Usability • Can users actually work with the system • Security • Can the system be hacked • Compatibility • Does the system work well with other systems • Installation • How easy is it to install the system • Serviceability • How easy is it to fix and/or enhance the system

Regression Testing • Wikipedia • any type of software testing that seeks to uncover new errors, or regressions, in existing functionality after changes have been made to a system, such as functional enhancements, patches or configuration changes. • The intent of regression testing is to ensure that a change, such as a bugfix, did not introduce new faults. • One of the main reasons for regression testing is that it's often extremely difficult for a programmer to figure out how a change in one part of the software will echo in other parts of the software."

Luddite vs. Machinedeath to the trained money • Manual testing is a relic of the past • Does not intrinsically provide a repeatable regression suite • Very expensive to repeat tests • Not reliable or reproducible in the long run • Automation makes all test suites regression tests allowing the code base to expand with minimal team effort. • Today all testing should be coded/programmed in repeatable tests suites that are automatically run by regression test server

Software Testing