What was the problem?

1. What was the problem? • We had a defect tracking tool -- several! • No support for multi-site development. • Maintenance overhead of multiple databases. • No web enabled version. • So naturally, the solution is … • To buy a new tool!

2. Original Plan • Identify products / vendors • Develop short list • Write evaluation criteria • Evaluate & select product • Negotiate price • Install tool in Development • Install new server and migrate to production

3. Execution • Things go okay through installing tool in a tool testing environment. • Usual and predicted hiccups. • Getting server configured with Oracle • Installing license server • Calculating total space requirements • Matching field values to old product

4. Another Track Starts • PIT (Process Improvement Team) starts • Look at collecting “Defect Root Cause” metrics • How can we improve defect management during testing? • No overlapping membership and little awareness of each other • PIT approach is to look only at metrics • Until… • The “process guy” enters the picture

5. Process Mapping • How does this process work today? • Who needs what information? • How is the process controlled? • What are the current breakdowns? • Rework like bad fixes, rebuilds • Stalled problems like “hot potato” or “not my problem” • Missed communications like “that fix was completed 6 weeks ago” • Definition: MR=change request or defect

6. Original Process • Very simple process model, but rework wasn’t understood very well • Multiple (24) States Fixed in STG, Fixed in ITG, Cannot duplicate, Waiting external verification …….

7. PIT Progress • Process Guy:“Defect management is basically a change management process. I know what change management looks like.This shouldn’t be too hard.” • Process guy still is unaware of tool effort. • PIT meets only 1 hour per week and attendance is irregular

8. Second Event: An “IAF” • Internal Audit Finding uncovers a mis-match between the process documentation and the way the current tool works. • Tool guy calls process guy for review of changes. • MR process document is 22 pages of text!

9. Regroup • No longer just a metrics problem • Tool implementation project is stuck • Metrics team is stuck too • MR process has no clear governance • Rules are very complex

10. Plan • Seems like there are three distinct things to do. • Model • Design a robust process and metrics. • Pilot • Find a willing partner, and • Make sure you have time to fix the things you forgot. • Deploy • Conversion will be a big deal so it will need a template and some estimation for the various teams.

11. Plan -- Model The Process • Stake holders • Developers, testers, support, tech writers, customers • Current activity • MR meeting, correcting, builds, testing, postponing,... • Transfer of control, inputs, outputs • Moving to another group, completed fixes, ... • Management and escalation • Postponing, 1 problem that needs 2 groups • Write • Review and fix

12. Model the Process • Use states because MR’s change hands • Different people do different kinds of work. • Rules are applied to exit states. • Management of states • Danger of getting stuck. Resources not balanced. • State transitions: What enforcement? What audits? • Remember Test and Fix usually works fast • Can’t let management get in the way. • So your change control board needs to have rules to focus the activity on higher problems.

13. Process Model State Diagram • New • Working • Fixed • Testing • Postponed • Watch • Closed

15. State: Describe the Activity • New • Purpose: Determine benefit of change and assign work. • What products are affected? • Who gets the assignment? • Remember: mostly it’s about test and fix, but not always. • Working • Someone is assigned to change some specific items for a specific purpose.

16. State: Describe the Activity • Fixed • These changed items solve the problem • Track what was changed and problem number • But it waits for the build because several changes will be submitted together for the testers. • Testing • Sometimes this is an inspection or editorial process.

17. State: Describe the Activity • Postponed • We will not assign resources to this request at this time. • Product Management decision (part of CCB). • Takes Product Management to put back onto the queue. • Watch • “We think this is fixed bu we cannot tell for certain.” • Sits for 3 weeks or 2 test cycles. • Closed

18. Roles Developer & Developer Manager Release Engineer (build) Tester and Tester Manager Program Manager (project manager) Product Manager (postponing) Product Support Representative (field defects) Tool support

19. Map Roles To state activities New: Assign work: Tester and Development Mgr. Fixed: Build: Developer, Release Engineer, Test manager Add process management activities Balance resources & Prioritize: Dev. Mgr, Prog. Mgr., Prd. Mgr., Prd Support Tool Support New projects, new users, performance, new reports

20. Current Activities MR Meeting already exists Testers and developers meet in small groups. Make assignments. Review build contents and test plans. Improvements Agenda is not standard and sometimes revisits the same problem. Escalation and transfer of control between development groups is awkward.

21. MR Meeting Agenda New Defects (test failures) for assignment Prioritize test blockers Testing progress on fixed items Move to watch. Items for escalation to management (postpone, conflicts, etc.)

22. Activities -- CCB EMT: Engineering Management Team This group does intergroup coordination, not CCB. Very large group, inc. CPS, PLM, Ops, etc. It might be ok for smaller projects, but not for big ones. EMT should see MR metrics for risk assessment. CCB Missing! Prioritize, postpone, balance resources, resolve conflicts

23. Creating the CCB • Purpose • CCB is a management function. It’s primary responsibility is to prioritize work and balance the resources. • It will also resolve disputes if a problem seems to be treated like a hot potato. • The CCB assumes a natural priority on defects so they can be handled by MR Meeting. • The CCB assumes full responsibility for prioritizing feature requests, and process changes. • All items for postponement go through CCB.

24. CCB Agenda • Current metrics • Transition activity, Current State #s • Stuck State (things that have been sitting) • Other change requests • Postpone, engineering features, etc. • Prioritize and Rebalance

25. Process Change Plan • Identify Pilot Team • Use a team doing new work to minimize impact • Converting a lot of data will take time • Representative • Must cover software and hardware • Enough resources to absorb pilot effort • Enough time to fix problems with new MR process • Amenable to change • Good managers

26. Process Change Plan • Pilot Work • Set-up database • Convert existing data • Training using scenarios • Support plan two people covering 6am-6pm • Change control for tool! • Continued development plan • Have to work on reports and graphs

27. The Pilot • Team • Large project of 100+ developers • 2-year effort • Hardware engineers have not used defect tracking • Training • 8 scenarios prepared but class only covers 3 • Lots of questions make course take over 3 hours • 3 scenarios is enough for first groups • Pilot is successful. CCB works right away.

28. Pilot Problems • Multi-site does not work as expected • Performance makes remote use problematic • Web interface works, but looks really different • but that’s what we use. • Support Team • Scripting, tool foibles, Crystal Reports • Multiple printers affect reports • Tool developers must do code reviews too!

29. Substates, rules, and reports. • “Invalid defects” and rejects • Someone has a problem! It’s just not yours. • Test case, documentation, and many others • “New” has lots of substates • Under investigation, failed test, re-submit from watch • Processing of Duplicates • Tool provided “Duplicate” function, but most should just close with “Reason=Duplicate” • External ones needed to be tracked to communicate to customer.

30. Great Inventions • Team started to track their process changes • Had to add a Change Type of “process”(addition to software, hardware and documentation) • Needed simplified Testing state • Parent-Child relationship • Problem that requires effort from multiple groups • Used to track feature development mid-stream • and a few complex defects

31. Management Queries & Reports • CCB has Lots of them • Lots of different combinations of filters • Current State • Transactions during current period • “Stuck State” • Candidates for postponement and bringing them back. • “What are the blockers?” & priority questions • MRRB uses “New” • Leads audit transactions for missing data

32. ProcessDocumentation • StateDiagrams

33. Responsibility Matrix

34. Process Change Plans • Technology Development Process • How do you build new capacity and capability? • New process is always a project of this type. • The following, Technology Development Process, type plan would have made it go faster.

35. TD Process

36. TD Phase 1 • Statement of Work (or Project Plan) • Detailed scope and responsibility • Requirements & Specification • Context diagram, quality goals, environment • Vendor & Tool Selection • May make this phase take a while

37. TD Phase 2 • Process Model & Documentation • Tool Implementation • Pilot training material • Pilot validation plan

38. TD Phase 3 • Pilot • Validate and measure process • Assist new users • Fix problems • Develop standard migration procedure • Advertise

39. TD Phase 4 • Publish process documents • Publish training • Migrate • Train and record • Advertise