Helping Customers Write Acceptance Tests - Making It Easy

1. Helping Customers Write Acceptance Tests - Making It Easy Jeremy Stell-Smith jeremy@thoughtworks.com

2. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

3. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

4. Why Test? • What’s the point? • To make change SAFE • To lead to a flat cost of change curve • As such, testing is the basis of XP

5. eXtreme Programming The Cost of Change Traditional Development Cost Time

6. “Test-Time” checking • Why do we like type safety? • As code changes, the compiler catches errors, and these errors are easy to fix • But, compile-time checking can’t check everything • There are runtime errors, and worse, bugs • Tests CAN guard against these • If it compiles, there are no compile-time errors • If the tests pass, there are no errors period

7. Types of Tests • Fine-grained vs. coarse-grained tests • Developer vs. customer tests • Fine-grained, developer tests (unit tests) are the ideal • Fast, come the closest to 100% coverage • Low duplication -> Low cost of change, low coupling • Use coarse-grained tests to protect yourself against holes in your bricks • Use customer tests to protect yourself from building the wrong thing right

8. An Acceptance Test is a Customer Test • Does not have to be coarse • Get your customers to write tests and take ownership • Must be in the language of the customer • Your “customer” may be an analyst or QA • Acceptance tests are not specific to XP

9. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

10. Web Service Project • XML in / XML out • Stories are in terms of XML API • …so tests should be in terms of XML

11. Example Story • Gate In Container Story • I would like record a container as Gated In to a facility by submitting this message. • The syntax should be: <gateInContainer time='2001/04/04 17:00’ container=‘123’> <facility code='us.oak.cfs12.testgap.com'/> <transportVehicle companyCode='bvl.com' code='A1234'/> </gateInContainer>

12. Our Framework • Simplest thing that could possibly work… • Built on top of Junit • Got UI for free • In the “language” of our analysts & customers • Setup / Teardown of Data

13. Example Test <testCase> <name>#3: Container Number was not provided</name> <input> <gateInContainer time='2001/04/04 17:00'> <facility code='us.oak.cfs12.testgap.com'/> <transportVehicle companyCode='bvl.com' code='A1234'/> </gateInContainer> </input> <expectedOutput> <failure> <exception>Required element container was not found</exception> </failure> </expectedOutput> </testCase>

14. Things We Added To It • Includes, to remove some duplication • dbValidation section to check against the db to be a bit more thorough • Let’s take a look

15. Things We Did Right • Analysts wrote the stories & the tests • At the beginning of every iteration, we got both • We kept it simple • Error handling • Analyst owned / developer maintained

16. Things We Did Wrong • Too much duplication – lot of copy / paste • Occasionally EVERY test broke – not good • Partly because XML isn’t type-safe

17. J2EE Project with Swing Client • Stories were at the client level • Client was untested • Client had lots of functionality • Big Project – QA Department

18. First Pass • One weekend • Simple, simple, simple • Used XML to script tests against value objects. • Not general, very specific to our app. • Gave us coarse-grained developer tests • Wrong language • Didn’t solicit input

19. Example Test <test name=‘mytest1’> <createNewDeal type=‘OTC’ subtype=‘Asian’/> <setValue name=‘dealHeader.name’ value=‘MyDeal 1’/> <setValue name=‘dealHeader.date’ value=‘jan 1, 2001’/> … <save/> <assert name=‘dealHeader.name’ value=‘MyDeal 1’/> <assert name=‘dealHeader.date’ value=‘2001/01/01’/> </test>

20. Second Pass • Now testing against swing components • Single threaded • Better language, but still difficult • Generic, but didn’t handle everything

21. Third Pass • Significant effort • JFCUnit • Added a recorder • Added an editor • modify, run, modify, run, modify, run • Had been too hard to debug • Added data population • Scaling became impossible because it wasn’t tested… • Language – Good • Ease of use

22. Example Test <test name=“foobar”> <window name=“Calculator Example”> <click name=“5”/> <click name=“1”/> <click name=“+”/> <click name=“3”/> <click name=“=“/> <click name=“JTextField”/> <assert name=“JTextField” text=“54”/> <click name=“+”/> <click name=“5”/> <click name=“=“/> <assert name=“JTextField” text=“59”/> </window> </test>

23. Forth Pass • Marathon • Uses Jython instead of XML • Access to all java code • Functions, classes, imports, external libraries, etc. • Interpreted • Test-First • Open source • JEdit

24. Example Test from defaultFixture import * def test(): window('Calculator Example') click('5') click('1') click('+') click('3') click('=') click('JTextField') assertText('JTextField', '54') click('+') click('5') click('=') assertText('JTextField', '59') close()

25. Testing Strategies • Tests that ran with the build • Unit tests • Smoke tests : coarse-grained tests that provided safety net for our unit tests - 1/3 of our acceptance tests • Cruise Control • Nightly build of other tests • Developer maintained

26. Things We Did Right • Once and Only Once • Encouraged feedback • Bought where we could

27. Things We Did Wrong • Analysts didn’t write the tests • We didn’t integrate close enough w/ QA • Used them as a substitute for unit tests • Disconnect with stories

28. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

29. Lessons Learned • In terms of stories • Acceptance tests are NOT a substitute for unit tests • Buy don’t build • Junit, Jython, etc • Iterative • Make it modular – once and only once • Language of Analysts • Focus on readability == maintainability • Don’t EVER want to re-record tests

30. Lessons Learned (cont’d) • Yes you have to test your testing framework… • Use a scripting language to write scripts • User friendly • Pair with your customers to write tests • Developers maintained • Error handling • Needs to be Customizable -> Open Source?

31. Things To Watch Out For • How to deploy • When to run • What to do when they break • Are tests independent? • Data? • Acceptance tests are not all passing? • Long running tests

32. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

33. FIT • http://fit.c2.com/ • Open source general purpose functional testing tool • Totally generic, and customizable • Allows fine-grained testing • Remember, customer tests are not necessarily end to end tests • If you want to GUI level tests, I wouldn’t use FIT • Ideal for test first stuff

34. Marathon – GUI Test Runner • http://marathonman.sourceforge.net/ • More suited for testing GUIs • Recorder & editor • Can’t use a recorder for Test First tests • But you can use it help maintain them • Modular without leaving script

35. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…