Quality of Life through applying Software Quality Management

1. Quality of Life through applying Software Quality Management A Case Study of the use of the Team Software Process in Games Vicarious Visions Tobi Saulnier, VP Product Development tobi@vvisions.com

2. Why so much overtime? And stress? • Over-ambitious scope versus budget • Inability to determine what is possible • Inability to convince customer or team what is possible • Inability to give hard numbers on the real cost • Underestimation of time needed for • Development • Testing • Fixing • When is it really done? • Inability to manage change • In schedule • In scope • In resources • Tend to just absorb more and more

3. Case Study: Apply Software Quality Techniques • Games are not just software • BUT they share a lot of the problems of software • Carnegie Mellon Software Engineering Institute (SEI) • Funded by the US for tech transfer to the US software industry • Some “products” of SEI • Capability Maturity Model – Levels 1-5 • Personal Software Process • Team Software Process • A bunch of other cool stuff

4. Preparation – Personal Software Process • Teaches software engineers to understand and improve their own process of coding • Better individual estimates, code, problem solving • Involves data collection by the individual • Estimates versus actuals – time, lines of code, … • “Defects” introduced and found • Seeding historic data for future planning • Training is a big time commitment! • Two one-week training sessions • Lots of homework - 10 programming assignments • Expensive! (for game industry) • All VV engineers attended training

5. Next – A TSP Pilot Project • TSP = Team Software Process • Everyone on team uses principles from PSP • Not just software folks – principles can be applied to art and design too • Additional TSP project management framework • Launch meetings • Defined roles • Change process • In-project Meeting scripts • We chose Spidey DS • Short launch project – all the hallmarks of no QoL • Team receptive • Lots of risk and change expected

6. Launching a TSP Project Day 1 Day 2 Day 3 Day 4 1. Establish Product and Business Goals 4. Build Top- down and Next-Phase Plans 7. Conduct Risk Assessment 9. Hold Management Review PM. Launch Postmortem 2. Assign Roles and Define Team Goals 5. Develop the Quality Plan 8. Prepare Management Briefing and Launch Report New Teams: TSP Process Review 3. Produce Development Strategy 6. Build Bottom- up and Consolidated Plans

7. More about the Launch • Allowed ALL team members to understand the game they were going to create and feel equally responsible for delivery • Granularity of tasks elucidated the quantity of work and scope • Scope reduction scenarios were considered and taken to provide a more reasonable schedule • Task granularity was refined: included each process step, not just a broad task (e.g. “Implement Spidey Sense” was broken into multiple PSP steps: DLD, DLD Inspection, Code, Code Review, Code Inspection, Unit Test) • Assumes 4 task hours per day (not 8!) • We did this all before, didn’t we? • Maybe, sometimes, … the structure makes a difference!

8. More About Time Tracking • Team members log time charged to tasks, task completion, and defects (aka bugs) • Provided visibility into tasks that were taking significantly more time than planned • Allowed for course correction such as adding resources or cutting scope • Data also used as a learning tool for planning the next project • This requires trust of management • No sharing of individual data • Fear of productivity monitoring will prevent accurate data • For instance, would be hard to implement if there are recent or impending layoffs

9. Task Completion Tracking • Earned Value (EV) and target EV reported at weekly group meeting by each team member • To achieve EV, task needs to be marked as completed in workbooks • Allows tracking of task completion

10. Defect Logging • Bugs encountered are logged • Bug logging should take note of when the bug was injected • This data should be used to help reduce # of bugs in the future and reduce rework (aka increase quality) • Want to remove bugs as early as possible in the pipeline (e.g. in design rather than implementation or test) • What is a defect? • For art? • For design?

11. Results: What went well • Team Morale • Launch led to team cohesiveness • Individual ownership of tasks • Lessened burden on leads and PM for scheduling by distributing work across all team members • Team responsible for scope reduction • Planning • Planning was detailed • Planning accounted for time spent besides implementation (i.e. design, testing, reviews, etc.) • Helped reinforce pipleines, etc as each task was broken down into process steps which earned value • We were able to justify the need for additional resources by showing specifically what those people would work on

12. More of what went well • Process • Design Inspections and Code Inspections led to solid, reusable code • Team management duties divided among the team left more time for Leads to contribute • Adding additional resources was feasible as the workbooks allowed us to “easily” balance and move tasks between people • Reviews scheduled for Art and Design earlier, prevented rework • Gathered data useful for future planning • Lines of code per hour, animations per day, etc • Review time • Task hours per week • Basic metrics for components

13. Of course, all was not perfect • Individuals could add forgotten or new tasks to their workbooks which could lead to hidden schedule slippage • At Launch did not schedule all tasks needed to complete the project, only a subset • The TSP Excel workbook tools • Clunky for transferring tasks between individuals • Overwhelming and have many acronyms • Team members failed to log defects • Lack of buy-in from some team members (-> bad data) • Process self-discipline slipped in final couple of weeks

14. Bottom Line • A huge difference, esp at the end of the project • Team could identify needs early • Missing resources • Estimations that weren’t working out • Work that hadn’t been planned • Resources were added relatively early • Didn’t have to fight with team about whether they were needed • Clearly identified tasks already identified that new people could be applied to • Crunch time still happened … but shorter, and higher morale We made launch!

15. Lead Artist “In the end I believe TSP saved the day. It allowed us to see the big picture rather early and gave us a focus.”

16. Lead Designer “It was good. I had a sense of what I had to do at most times.”

17. Lead Engineer “I consider the use of TSP on Spidey DS to be a complete and utter success. It forced us to scope back our game early and set out a schedule that could be divided among a growing team. Constant updates on our progress allowed us to keep focused on key features and resist feature creep. I would feel very uncomfortable running any software project without the TSP process in place.”

18. Suggestions for future use • Critical that inspections are occurring • Process must be tailored so that team members are comfortable with the process and feel compelled to use it • TSP needs to be customized further for design, art, and animation • TSP still needs a lot of tuning to be used more efficiently

19. Thanks! Questions? Also See the Related Talk: Wed 9-10 am “Spider-man 2 DS Launch: Double the Screens, Half the Dev Cycle” Also much more info on TSP at www.sei.cmu.edu