AAA Automated Testing… For AAA Games

1. AAA Automated Testing…For AAA Games Francesco CarucciTechnical Director

3. How much does a game cost? • Development costs spiraling up • EA Exec: “Cost of games is crazy!” • Some examples: • Killzone 2: > 40 M$ • Halo 3: > 30 M$ • GTA 4: > 100 M$ (rumored) • Clearly, this is not sustainable in the long term

4. Development cost partitioning • To keep it simple: • Producing assets • Writing code (game code, technology, tools) • Debugging and maintaining code • Code dictates cost of producing assets • Better tools means less time to create assets • More efficient code means less time spent optimizing assets

5. Solutions? • Better production frameworks • SCRUM or some variation of it • More experienced teams • Higher senior/junior ratio actually costs less • Better tools (3rd party engines, off the shelf packages) • Out sourcing (yay for assets, nay for code) • Higher quality code

6. Code is not the King • But it’s a first class citizen and we want to keep its cost down • Higher code quality reduces Cost Of Ownership • Less bugs • Less over engineering • Higher code quality means code easier to change and iterate upon • Higher code quality means more time to be spent on polishing the game • Higher code quality means less money spent to make a game! • And a bigger chance to ship on time and hit that market sweet spot

7. Change Prevention Process • Many teams follow a change management process whose main goal is to prevent “feature creep” • It’s effectively a Change Prevention Process • But you can’t design “fun” up front • Games must be developed iteratively until “fun” is found… • So? • Embrace Change 

8. Cost of Change • Game Design requirements change • “The aim system is good, but I would like it more peachy, you know a bit more of this, but slightly less of that, more rocking, less oompfh. Can you change it?” • Traditionally Cost Of Change rises sharply over time • “No, I can’t change it, It would break X and Y and probably Z too, I don’t really know” • We want to keep Cost Of Change as flat as possible • “Yes we can!”

9. Enter Automated Testing… • Automated Testing has been a broadly accepted tool for several decades in the Software Industry aimed at reducing cost of developing working code • In fact… (warning: Boring Slides Ahead)

10. Assessing test-driven development at IBM • “In a software development group of IBM Retail Store Solutions, we built a non-trivial software system based on a stable standard specification using a disciplined, rigorous unit testing and build approach based on the test- driven development (TDD) practice. Using this practice, we reduced our defect rate by about 50 percent compared to a similar system that was built using an ad-hoc unit testing approach. The project completed on time with minimal development productivity impact. Additionally, the suite of automated unit test cases created via TDD is a reusable and extendable asset that will continue to improve quality over the lifetime of the software system. The test suite will be the basis for quality checks and will serve as a quality contract between all members of the team.”http://collaboration.csc.ncsu.edu/laurie/Papers/MAXIMILIEN_WILLIAMS.PDFhttp://portal.acm.org/citation.cfm?id=776892

11. An initial investigation of test driven development in industry • Simposium on Applied Computing • “Test Driven Development (TDD) is a software development practice in which unit test cases are incrementally written prior to code implementation. In our research, we ran a set of structured experiments with 24 professional pair programmers. One group developed code using TDD while the other a waterfall-like approach. Both groups developed a small Java program. We found that the TDD developers produced higher quality code, which passed 18% more functional black box test cases. However, TDD developer pairs took 16% more time for development. A moderate correlation between time spent and the resulting quality was established upon analysis. It is conjectured that the resulting high quality of code written using the TDD practice may be due to the granularity of TDD, which may encourage more frequent and tighter verification and validation. Lastly, the programmers which followed a waterfall-like process often did not write the required automated test cases after completing their code, which might be indicative of the tendency among practitioners toward inadequate testing. This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code development”http://portal.acm.org/citation.cfm?id=952753&dl=GUIDE&coll=GUIDE&CFID=46811546&CFTOKEN=92849791

12. Google • “Standard Google practices include unit tests and functional tests, continuous builds, last known good build, release branches with authorized bug fix check-ins, focused teams for release engineering, production engineering and QA, bug tracking and logging of production runs.”http://agile2007.agilealliance.org/index.php%3Fpage=sub%252F&id=713.html

13. Professionalism and Test-Driven Development • “Test-driven development is a discipline that helps professional software developers ship clean, flexible code that works, on time. In this article, the author discusses how test-driven development can help software developers achieve a higher degree of professionalism”http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F52%2F4163008%2F04163026.pdf&authDecision=-203

14. Unit Testing in Games • “In our experience, the introduction of automated tests and continuous integration makes development teams more efficient and results in more reliable, and often quite simply, better software. Additionally, it reduces the pressure and workload on development teams by reducing the effort for manual testing, and allows bugs to be found earlier in the development process. Certainly, automated tests alone won't make your game a hit. But almost as certainly, they will make life easier for developers, artists, project leaders, producers and even players.”http://www.gamasutra.com/features/20050329/roken_pfv.htm

15. Unit Testing in Games (contd…) • “Test-driven development can be a very effective development technique. We have successfully applied it to game development in a variety of situations, and we’re convinced of the many benefits it has provided us. Right now, the idea of writing code without writing tests first feels quite alien to most of us, and we treat TDD like the scaffolding in building construction: a necessary tool that will not be shipped to the customer but that helps tremendously during development”http://gamesfromwithin.com/?p=50

16. Unit Testing in Games (contd…) • “Wir haben in den letzten fünf Jahren ausschließlich positive Erfahrungen mit automatisierten Tests und Continuous Integration gesammelt. Die automatisierten Tests sichern Stabilität auf hohem Niveau, während Continuous Integration zusätzlich eine deutliche Arbeitserleichterung darstellt. Der Aufwand für die Einrichtung und Wartung der Verfahren ist überschaubar und hat sich schnell amortisiert. Wenn man die anfängliche Skepsis überwindet, möchte man unserer Erfahrung nach diese Art der automatisierten Qualitätssicherung sehr schnell nie mehr missen.”http://www.igda.dimajix.net/fileadmin/ressources/report-03/07_roeken_summary.pdf

17. What’s an Automated Unit Test • A snippet of code that exercise a Unit Under Test (method of a class), assert its correctness and is executed automatically every time the system is built • Running unit test is effectively a compilation step • Compiler checks if the code is syntactically correct • Unit tests check if the code is semantically correct

18. Test First vs Test Last • Writing automated tests is accepted to produce higher quality code at lower cost • More tests == less time spent debugging • Should tests be written after production code? (Test-Last) • Should tests be written before production code? (Test-First) • “This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code development” • Test Driven Development == Test-First

19. Test-Driven Development • Write a failing test before the code is written • Write the simplest code that could possibly make the test pass • Refactor the code to eliminate all possible duplications and code smells • RedGreen Refactor • That’s the coding rhythm

20. A Failing Test • First write a failing test: • CRY_TEST (FireButtonPressAndRelease){ StartFire(); ASSERT_IS_TRUE(IsFiring());} • The test is dictating the interface and the contract of the unit being written: • When StartFire method is invoked, the weapon is firing • … then implement the minimal code to make it pass • bool IsFiring() const{ return true;}

21. Another Failing Test • Write another failing test to dictate more behavior: • CRY_TEST (FireButtonPressAndRelease){StartFire();StopFire(); ASSERT_IS_FALSE(IsFiring());} • When StartFire method is invoked, and then StopFire method is invoked, the weapon is not firing. • The actual word explanation is redundant, the whole information needed is in the code itself in readable form!

22. Another Failing Test (contd.) • Minimal implementation: • bool StartFiring(){ m_isFiring = true;}bool StopFiring(){ m_isFiring = false;}bool IsFiring() const {return m_isFiring;}

23. How does it work • Tests are small, simple, straightforward and test only one condition • Long and complicated tests must be avoided • Tests are self documenting • Tests describe the design of the code to be written • Contract • Invariants • Post-conditions • Calling conventions • A TDD cycle should take few minutes (often less than a minute) • 30 minutes to write a test means a bad test • Break bigger tests in smaller steps (help test coverage) • Begin with baby steps, increase step size when becoming more comfortable

24. Why TDD? • Small and clearly defined steps lead to the final implementation • No over engineering: write only the code that makes failing test pass • Minimal maintenance • Less code for the same needed functionality (including tests!) • Less time needed to change the code when requirements change • Less coupling between units • Less cost to maintain units • Tests are a by product of design and code construction • They keep contracts tested automatically through the life of the project • Code is easier to modify, broken contracts easier to discover • Defects and design inconsistencies are found early on • It costs less to fix fewer bugs!

25. When TDD is not appropriate • Exploratory coding, throw away prototype code to quickly explore different solutions for a problem • But the proper solution must be implemented test-driven to reduce cost • Code wrapping around existing libraries • GUI layer • Very low level code dealing with the hardware

26. Excuses for not Testing

27. Excuses for not Testing (contd.) • … writing tests takes time and this time is stolen from writing code • People forget about the time spent debugging and maintaining code which is much larger than the time spent writing it • Tests written test-first are part of the design process, that has to be done anyway. TDD is just a formalized way of designing code. Overall it takes less time to produce working code test-driven • When the code base is covered, modifying code is safer and takes less time cause of regression provided by the automated tests • By writing tests first, fewer bugs are experienced by working with generally tested code (no huge amount of time spent debugging)

28. Excuses for not Testing (contd.) • … tests can’t cover everything • Some test coverage is still better than no test coverage at all • Legacy code shouldn’t be tested unless it’s being refactored • Rome wasn’t built in a day, but it was built eventually (and conquered the known world )

29. Excuses for not Testing (contd.) • … but this code is temporary and I will change it later, I don’t need to test it now • All code will be modified soon or later: making code simpler to change is the main reason to write it test-driven: reduce over engineering to the minimum, keep it simple • Requirements change: modify the affected tests and production code. If unit tested, the code base is more decoupled and easier to modify with less side effects

30. Excuses for not Testing (contd.) • … game development is different: automated testing might not work • Game development as a whole is different… • … but writing code is the same if not simpler • Game code is mostly algorithmic, which is easier to test than GUI code, for example • An A* algorithm is mostly the same, be it in a game or in a web crawler • Game code changes often: more changes means more pressure for automated tests to protect against side effects

31. Excuses for not Testing (contd.) • … games are about fun: you can’t capture fun with automated tests • But you can capture bugs, and less time spent debugging means more time looking for the “fun” part • Writing a test for a bug fix ensures that bug will not resurface later on • 30% of bug fixes create a new bug caused by side effects

32. Excuses for not Testing (contd.) • … this is AAA Game, it’s too risky to do automated testing on it • AAA game means highly polished, highly technologic, great game that delivers great fun • In order to achieve AAA, the most efficient production tools are needed, Automated Testing is an efficient tool • It’s too risky to NOT do automated testing on a AAA game • Be among the first to gain the competitive advantage and improve the Industry

33. A Slice of a bigger Pie • Automated tests (even test-driven) don’t solve all problems • Chatting up a girl will still be a problem for a programmer  • Automated tests must be part of a wider agile framework including: • Continuous integration • Team code ownership • Daily deliverables • Simple code that works • Functional and Integration testing • These practices work together and strengthen each other with the goal of producing more functionalities in less time.

34. What’s next? • More automated tests beyond Unit testing • Functional testing • Test more units functioning together at higher level • Requires more time to build a Functional Testing Framework • Can capture bugs generated by the interaction of different units • Smoke testing • Load different levels after every check-in looking for crashes • Run scripted sequences looking for crashes • Generate random inputs looking for crashes

35. What’s next? (contd.) • Asset testing • Load different assets from the engine to check for crashes and correctness • Animations, Meshes, Textures • Example: “a Mesh must be smaller than 5000 polygons if it’s a character, 20000 polygons if it’s a building and must have a physics mesh smaller than 100 polygons associated” • It requires metadata • Run tests after every asset check-in by content creators! • Validating assets before they get into the build can save an enormous amount of debugging time

36. Adopting Automated Testing • It doesn’t matter how much you read about it, it’s like riding a bike: you need to try it to learn it and understand it • It needs strong leadership, clear vision, courage and a company willing to invest in Agile Practices • Lots of money to be saved and huge competitive advantage are at stake here • Don’t take decisions without understanding the issues • Automated Testing often goes against perceived common practices in the Game Industry • It needs a lot of mentoring and training (writing good tests is difficult) • It needs strong commitment by management • The whole team must be bought into it • The rogue primadonnas unwilling to improve their skills must be removed from the team

37. Questions? • Shoot, I can take it  • Email: • francesco@crytek.com • francesco@carucci.org