Dynamic Software Tracking

1. J. Erik Hemdal Robert L. Galen BOSCON 2004 – Track C4 Dynamic Software Tracking

2. Agenda • Introduction • Project Goals • Measures from Defect Data • Triage and Workflow • Conclusions • Questions and Answers

3. Why are we here? • Discuss the Big Four elements you need to control for a successful project • Examine a variety of defect-data measures • For what they tell us • For how they can be defeated • Show how you can manage defects so that you can stay in control of your project and workload – especially in the endgame close to release points.

4. The Big Four Elements • Have stable, agreed-on goals • Know the quality level and quality trends of your software • Keep your team healthy • Always know how much work remains and how much time is left

5. Understanding & Influencing the Goals • The formal requirements • Unstated requirements • Expectations from customer, sponsor, team • Negotiation and balance are the key • Features • Specific delivery timing • Level of quality (Hot-button issues) • Essential vs. “nice-to-have” elements

6. Understanding & Influencing the Goals (cont.) Understanding the overall project plan • Development phasing (building to features, stability and completeness) • Understanding the methodology (waterfall, incremental, agile/extreme, RUP) • Test phasing (how many passes, reduction of change, entrance & exit criteria) • Key Milestones • Delivered phases • Code freeze / complete • Beta tests • Final release

7. Understanding & Influencing the Goals (cont.) • Release Criteria is the critical guide for releasing software. Should cover – • Scope • Quality • Time • Team • Good release criteria should • Define success • Learn what’s important for the project • Be Drafted and thoroughly reviewed • Be “SMART” (specific, measurable, attainable, realistic and trackable) • Assist in gaining consensus • Serve as a goal / guide throughout triage and release

8. Software Measures from Defect Data • Good defect tracking provides insight • Into the quality of the product • Into the performance and health of the team • Into the amount of remaining work and rework • Can be a hot-button issue because of misuse • New way of thinking: A “Defect” means “something to change”. More things to change means more work. • Changes stand between you and Done!

9. Measure Illustrates Defect counts or defects / KLOC Defect density, predictor for future defect trends and overall product quality Defects / Unit of testing time Testing workflow and productivity Defect count over time Time distribution of defects, looking for downward trends # found, # fixed, # remaining over time Determine overall maturity - release / ship readiness # high – medium – low severity defects Determine overall maturity - release / ship readiness Phase –containment of defects Root cause, true cost of quality Fix hours / defect Cost per defect, predict development repair efficiency Defects by state counts Defect workflow in time, potential bottlenecks Common Defect Measures

10. Defects/KLOC • Uses: • Indicate overall quality of code • Guide to trouble spots • Depends on: • Definition of a KLOC • Modular architecture with visible granularity • Defeat by: • Attacking the KLOC • Blame requirements • Writing longer code

11. Defects/Unit of Test Time • Uses: • Overall level of test “productivity” • Check test strategies or test “wearout” • Depends on: • Factors testers usually don't control • Defeat by: • Untestable/unreachable code • Cutting test time • Graphical display – similar to Defects/KLOC

12. Defect Counts over Time • Uses: • Adjust test sequencing and scheduling • Can indicate significant problems in test • Depends on: • Overall coordination • Defeat by: • Interrupting testing • Finding significant defects • Missing functionality

13. # Found, # Fixed and # Open • Uses: • Suggests when to ship • Depends on: • Reliable, repeatable test capability • Change control • Defeat by: • Curtailing test • Many small changes • Late changes

14. # High/Med/Low Severity Defects • Use: • Indicates overall readiness of a product • Depends on: • Proper triage • Effective criteria • Defeat by: • Management fiat • Subtle negotiation • Peer pressure

15. Phase-Containment of Defects • Uses: • Identify process problems • Justify process improvement • Depends on: • A defined phase-gate process • Defeat by: • Lack of commitment to the process • Lack of time to analyze raw defect data

16. Average Fix Hours per Defect • Uses: • Illustrate the cost of poor quality • Provide data for repair estimates • Depends on: • Ability to capture data • High trust culture and within the development team • Defeat by: • Misusing the data • Similar measures possible for build, test, review time

17. Defects By Status Histogram • Uses: • Show team bottlenecks • Gauge project status and product maturity • Depends on: • Consistent and reliable state update activity • Defeat by: • Gamesmanship about defect status

18. Dynamic Tracking • Defect Triage • Workflow Management • Defect Packaging

19. Defect Triage • Manage three elements • Severity How serious is this defect? • Priority How soon should this be fixed? • Impact What is the effect on team and customer? • Define a triage team and procedure early • Include QA, developers, project office, support • Set up the “drill” • Update defect reports with triage decisions – who, what, why • Map triage policy to the overall release plan

20. Workflow Management • Don’t just let work happen based on priority or chance - rather, proactively manage repairs! • Pre analyze scope, level of difficulty and potential impact – prior to assigning major repairs • Each engineer has an assigned “work queue” or a to-do list that is managed from the defect tracking system • Leverage the DTS for reporting and work flow management

21. Workflow Management (cont.) • Create guidelines per engineer 5-10 work items • 1 or 2 high priority defects • 3-4 moderate priority defects • 1-2 defects to investigate/analyze • Good idea: Focus on your “Top 10” issues, then stop and analyze • Reallocate judiciously; avoid churning • Create engineer profiles to help understand strengths and skills

22. Defect Packaging • Schedule a series of code drops or “packages” for testing • First release • Updates; new functions • “Critical fix” package • Release candidate • Final release • Each should have a primary goal • Testing involves high fixed costs; packages give you the most value (fixed defects) in each cycle • Don't waste the “power of the package”

23. Tying Things Together • Supporting the Goals • Maintaining the Level of Quality • Watching the Health of the Team • Knowing What's Left to be Done

24. Supporting the Goals • Effective defect triage helps you to maintain your focus on the goals of the project. • Triage directs your team's effort to the most important work, balancing the demands of all stakeholders • Defect data can flag roadblocks and slowdowns before they derail the project

25. Maintaining the Level of Quality • Defect measures and reports can give you a direct reading on the state of your project • How many defects there are • How defects affect the product and project • Where extra design, test, or review effort is warranted • When it's appropriate to release (or Not to release)

26. Watching the Health of the Team • Defect data helps here by - • Signaling overloaded and frustrated team members • Capturing and documenting key decisions and key changes • Communicating changes that WILL NOT be made • Preventing unnecessary work

27. Knowing What's Left to be Done • Defect tracking data helps here by indicating - • How many items must be changed • How long will these changes take • How much will they cost • What will be unfinished when we stop

28. Questions?

29. References • A. Allison, "Meaningful Metrics" The Software Testing & Quality Engineering Magazine (May/Jun 2001) • R. Black, Managing the Testing Process, 2’nd Edition, New York, NY: John Wiley & Sons Inc., 2002 • R. Galen, “Mastering the Software Project Endgame”, New York, NY: Dorset House, 2004 - forthcoming • C. Necaise, "Managing the Endgame." The Software Testing & Quality Engineering Magazine (Jan/Feb 2000) • J. Rothman, "Release Criteria: Is This Software Done?" The Software Testing & Quality Engineering Magazine (Mar/Apr 2002) • J. Rothman, "Managing Projects: Release Criteria, or Is it Ready to Ship." Newsletter Vol. 1, No. 2 (1999) • B. Schoor, “Managing Quality During the Endgame” Presentation from schoorconsulting.com website and PSQT/PSTT 2002 North conference – www.softdim.com (2002)

30. Bob Galen EMC2 Corp. & RGalen Consulting Group, LLC galen_bob@emc.com bob@rgalen.com www.rgalen.com Contact Information Erik Hemdal Independent Consultant and Instructor ehemdal@brandeis.edu