Formal Technical Reviews

1. Formal Technical Reviews Matt Graham 18 November 2004 EECS 814 University of Kansas

2. Introduction Generic Inspection and Variations Inspection throughout Development Cycle FTR Metrics Tools Conclusion Outline

3. FTR: What it is... ::Input:: • A method involving a structured encounter in which a group of technical personnel analyzes an artifact according to a well-defined process. ::Output:: • A structured artifact that assesses or improves the quality of the artifact as well as the quality of the method.

4. A structured FTR is a forum for: FTR: What it is... • Finding defect information for the Author. • Educating peers about product. • Providing fault likelihood data for testers. • Providing detailed status report for managers. • Allowing process improvement group a test to measure.

5. Walk-through No Measurements Often informal, impromptu Main purpose is developer training FTR: What it isn’t...

6. Why Review? • Improves overall product quality • Early detection of defects • Reduces rework and its associated costs • Educate the participants and provide training • Setting standards of excellence / maintain process improvement momentum • Improve schedule performance • IBM reported 83% and AT&T 92% defect detection through inspections

7. Inspection Process Overview Planning • 6 Steps • Team • 2 - 6 members • Experienced and involved in product development • Distinct and important roles Overview Defect Detection Defect Collection Defect Correction Follow-Up

8. Inspection Process Overview Planning • Roles: • Organizer – plans the inspection activities • Moderator – ensures that procedures are followed and moderates the meetings. • Inspectors – responsible for detecting defects in the product • Presenter – presents the product in logical fashion paraphrased at a suitable rate • Author – author of the developed product. • Recorder – records the defects during the meeting • Collector – collects the defects if there is no meeting. Overview Defect Detection Defect Collection Defect Correction Follow-Up

9. Planning Planning • Objective – organize inspection • Check to see if work products pass the entry criteria • Select inspection participants and assign roles • Schedule inspection meeting • Distribute inspection material • Prepare inspection documentation • Ensure product inspection readiness Overview Defect Detection Defect Collection Defect Correction Follow-Up

10. Objective Planning • Objective – educate participants on the product being inspected • Author explains the inspection materials • Can be beneficial when: • The inspection artifact is complex • If the artifact is part of large system • If new members participate in inspection • Could be rolled into previous or next phase if product is simple enough • Communicate inspection goals! Overview Defect Detection Defect Collection Defect Correction Follow-Up

11. Defect Detection Planning • Objective – to find defects • Can be divided into 2 phases: • Preparation (individual) phase. • Meeting (group) phases. • Individual preparation with the goal of detecting defects can help inspectors to be more prepared. Overview Defect Detection Defect Collection Defect Correction Follow-Up

12. Defect Collection Planning • Objective – collectively agree and document the defects (triage) • Decide on further inspection needed or not • Decisions made subjectively as a group. Overview Defect Detection Defect Collection Defect Correction Follow-Up

13. Defect Collection Planning • Objective – correct the defects collected in the previous phases • The author makes the detected changes and sends it for follow-up Overview Defect Detection Defect Collection Defect Correction Follow-Up

14. Follow Up Planning • Objective – to make sure all defects are resolved • Only one person involved and he/she verifies the defect resolution • Verifier should “return and report” to inspection team Overview Defect Detection Defect Collection Defect Correction Follow-Up

15. Formal Inspection Methods • Fagan Inspection • Asynchronous Inspection • Phased Inspections • N-Fold Inspections

16. Fagan Inspection Overview • 4 Roles • Author • Reader • Moderator • Scribe • Meeting centric – cost of scheduling and time • Meetings add little to defect detection Preparation Inspection Rework Follow-Up

17. Asynchronous Software Inspection Initialization • Team • Author not involved • Moderator and Inspectors Inspection Review Inspection Compile Final Defect List Re-Work Re-Work

18. Asynchronous Software Inspection Initialization • No meetings • Easy to assess participation • Process improvement from documenting all correspondences • Allows parallel communication • Can be distributed in space and time • Eliminates group approval Inspection Review Inspection Compile Final Defect List Re-Work Re-Work

19. Asynchronous Software Inspection Initialization • Moderator sends out material • Initial Individual Review – create list of defects • Circulate the copy of defect list to all inspectors and discuss via email • Individual Review – update defect list and send to Moderator • Moderator compiles final defect list, send it to author and follow up – eliminates group approval Inspection Review Inspection Compile Final Defect List Re-Work Re-Work

20. Phased Inspections • Consists of several coordinated partial inspections called phases • Each phase inspects for a specific property or small set of related properties • Each phase responsible for thorough checking of the properties • Some phases have single inspectors and others have multiple inspectors • Reference documents are provided to inspectors at the beginning of each phase and each inspects using checklists • Domain specific checklists • Application specific checklists

21. N-fold Inspection • N independent teams inspect the same product using traditional inspection method • Collective effort of multiple teams more faults than a single team • Moderator collects faults from independent teams and composes the final defect list

22. Requirements Inspection • Reading Techniques are techniques to analyze work product during defect detection • Ad-hoc • Checklist based reading • Scenario based readings • Perspective based reading • Defect based reading

23. Perspective based Reading • Inspectors stand in for specific stakeholders in the document to verify the quality of requirements. • Designer – detail for creating system components • Tester – detail for constructing test plans • User – correct functionality • For each perspective, reviewer creates high level work products from the requirements document

24. Code Inspections • Cognitive models for program comprehension • Bottom-up • Top-Down • Integrated • Systematic and as-needed comprehension

25. FTR Metrics • Helps you measure the effectiveness of the inspection • Aids in continuous process improvement • Provides feedback to management • A unit of work product varies often: • Usually 1000 lines of source code (not including comments) • Page counts and line counts for text

26. Some Generic Metrics • Average preparation effort and preparation rate per unit of material • Average examination effort per unit of material • Average explanation rate per unit of material • Average number of defects and major defects found per unit of material • Average hours per defect and per major defect

27. Nine Key Metrics by AT&T for Code Inspections • Total Non Comment lines of source code inspected, in KLOC • Average lines of code inspected • Average preparation rate • Average inspection rate • Average effort per KLOC • Average effort per fault detected. • Average faults detected per KLOC • Percentage of re-inspections • Defect-removal efficiency

28. Tools • Various tools for different inspection methods. • ICICLE – for inspection of C & C++ programs • Scrutiny & InspeQ for specific inspection processes • ASSIST –supports generic inspection process

29. Choosing a tool... • Threads of discussion • Sharing of Information • Train of thought • Visual Cues • Reaching a consensus • Coordination

30. The Future of FTR • FTR adoption is slow: • Managers see added cost of inspections instead of the benefits of greatly reduced defect leakage • Practitioners already have tight schedules and reluctant to take on additional responsibilities • Companies postpone adoption since peer reviews are part of CMM level 3.

31. FTR Barriers • Insufficient preparation • Moderator domination • Incorrect review rate • Ego-involvement and personality conflict • Issue resolution and meeting digression • Recording difficulties and clerical overhead

32. Conclusion • Organization’s size, culture and industry should be considered in deciding on the FTR method to use. • Review does not replace testing but can make it easier • Success of inspection lies in: • Process Adherence • Systematic Reading technique • Optimization

33. Philip Johnson: http://www2.ics.hawaii.edu/~johnson/FTR/ Tom Gilb: http://www.result-planning.com/ IEEE STD 1024-1197 FTR: References