Acceptance Testing vs. Unit Testing: A Developer’s Perspective

Acceptance Testing vs. Unit Testing:A Developer’s Perspective Owen Rogers orogers@thoughtworks.com Rob Styles rob@objectagile.com

What are Acceptance Tests? • Tests owned and defined by the customer to verify that a story is complete and correct.

Why we use Acceptance Tests: • Increase team confidence that the system is correct. • Structures communication between customer and developers. • Help customer think through requirements. • Informs developer estimates. • Indicates how to split a story. • Concrete definition of story completion (“knowing when we’re done”). • Early involvement of QA. • Automated regression testing for free.

Why other people don’t: • Customer participation. • Management buy-in. • QA support. • More tests for developers to implement. • No common framework.

ATs and UTs are the same though, right? • Both can and should be written first, before implementation. • Both should be automated by containing assertions for validating expected results. • Both typically use fixture setup and teardown code to run before and after the execution of the tests. • Both can be implemented using an xUnit framework.

Difference: ATs are written by the customer. • UTs only need to be read and understood by other developers. • ATs must be defined in a language familiar to the customer. • ATs require a customer meta-language: • Generic enough to capture requirements • Abstract enough to be maintainable • Simple enough to be easily understood

Example Acceptance Tests. FIT Acceptance Test: FAT Acceptance Test: • Request: “http://google.com” • Check title: “Google” • Click link: “Jobs” • Check title: “About Google” • Click link: “Press” • Click link: “Review” • Check title: “Google Press Room”

Difference: Customers don’t use an IDE. • UTs are implemented in the language of development using an IDE. • ATs should be defined using a tool accessible to the customer (Excel, web page, wiki). • ATs must be retrieved from the tool and parsed. • Ideally, the customer should be able to run the ATs from this environment.

Interacting with customer tool. Extract Test (.csv) Excel AT Framework Write Test Execute Test Wiki Extract Test (.html)

Difference: ATs are not directly executable. • UTs invoke code-under-test directly. • AT definition is abstracted from its implementation. • ATs specify what needs to be done, but not how to do it. • ATs require an implementation layer to invoke the application. • AT definitions must be interpreted – parsed and mapped onto corresponding implementation. • Tests must be definable and executable prior to implementation.

public void Request(string url) { } public void CheckTitle(string title) { } public void ClickLink(string link) { } Test Definition / Test Implementation Test: Verify Google Links AT Framework Class: WebPageFixture Interpreter • Request: “http://google.com” • Check title: “Google” • Click link: “Jobs” • Check title: “About Google” • Click link: “Press” • Click link: “Review” • Check title: “Google Press Room”

Difference: ATs test through the same interface. • UTs test at level of a single unit (class). • ATs test at level of user/client interaction. • All tests go through a common gateway (Http, Soap, UI controls or application façade) • Considerable opportunity for reuse across test implementations. • ATs need a framework in the technology of the interface it drives: Web, GUI, Web Service, Messaging etc.

ATimpl ATimpl ATimpl Testing through the application gateway. Definition Acceptance Test Interpreter Implementation Application Gateway Test Client Application Application Gateway ATD ATD ATD

Difference: ATs test the Users’ Experience. • UTs should test all application logic in detail. • ATs encourage the customer to consider the all aspects of user experience. • ATs should test: • Interaction & Flow • Performance • Error Handling • Security • ATs don’t test: • Usability • Look & Feel

Difference: ATs are end-to-end. • UTs test single unit in isolation. • ATs test interaction of all layers in the system. • ATs require external systems to be operating correctly.

Difference: ATs are process-driven. • Each UT typically tests a single public method. • ATs consist of a sequence of steps simulating a client interacting with the system. • ATs may test the same steps repeatedly in different variations with different data. • Considerable overlap across different ATs. • Single point of failure will break multiple ATs.

Difference: ATs have serious side-effects. • For UTs, setup and teardown code should be common only to a single fixture. • ATs interact extensively with external systems and the environment. • ATs typically require separate setup and teardown per test as different tests will have different side-effects. • Setup and teardown routines are commonly shared across tests and even across stories.

Difference: ATs suffer concurrency issues. • UTs should be largely independent of their environment. • ATs require that the environment is setup as expected by the tests. • Problems can result from different users running ATs concurrently. • Ensure sufficient variation in the data used for each test to prevent collisions.

Difference: Failing ATs don’t break the build. • UTs must pass when checked in – failure breaks the build. • ATs may be written at the start of a story and may not pass until the very end of the iteration. • We mark ATs as [Ignore] until they have been fully implemented.

Difference: ATs don’t go quietly • UTs are designed to run silently, with all verification encoded in the assertions. • Customer requires visual feedback to give reassurance that test does what is expected. • ATs need to provide step-by-step visibility of the contents and results of running the tests.

Difference: ATs are useful outside of dev environment. • UTs are primarily useful to developers for ensuring the code does what was intended of it. • ATs on the other hand can provide a good assessment of the overall health of the system. • This makes them useful for assuring deployments to testing environments are sound. • To do this usually requires environmental parameters to be extracted to a configurable location.

Difference: ATs mock out systems, not units. • Introducing mocks for dependencies in UTs helps to decouple the code. • We often have the same needs in ATs to mock third party interfaces that are not always available. • Mocking is very hard because the AT is outside of the application context so a specific interface for the system being mocked has to be created. • Mocking is still important to maintain customer’s trust in the solution. • Introduce mocks to help simulate failure conditions.

Setup setup ATD ATimpl impl UT impl UT teardown setup ATD ATimpl impl UT impl UT teardown setup impl UT ATD ATimpl impl UT teardown Teardown AcceptanceTest Definition Implementation Unit Implementation Unit TestFixture Acceptance Test Client Application Interface Mock

Acceptance Testing vs. Unit Testing: A Developer’s Perspective