Cutting-Edge Software Inspection Project at NASA

1. State-of-the-Art Software Inspections at NASA Forrest Shull, Fraunhofer Center for Experimental Software Patricia Larsen, Engineering – Maryland Rose Pajerski, Ioana Rus, Mike Stark, GSFC Allen Nikora, JPL

2. Project Information • This project • A collaboration between JPL, GSFC, CSC, and FC-MD • Funded by the Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program • Currently in the third of three years • Major objectives: • Improve software inspection practice and support projects in their use • To reduce cost of achieving high-quality software.

3. Project Plan Interview projects doing inspections. Analyze problems being faced. Yr1 Yr2 Introduce advanced techniques to address project issues. Measure benefits in comparison to past inspections. Yr3 If beneficial, investigate where beneficial and measure ROI. Further disseminate and measure.

4. Project Reality • Interview projects and create LLR. • Found that not many were doing them. • Most were keeping some of the practices, but not the whole process. • Selected a candidate technique (perspective-based reviews). • Did initial training, got feedback. Yr1 17 projects from 5 Centers IV&V personnel Yr2 Product: Lessons Learned Report

5. Use stakeholder perspectives to: • Give each reviewer a particular quality focus; • Make the individual review active, not passive; • Articulate what defects to detect. Roles Activities Products Inspection process overview Software Artifact 1 PlanningForm Planning organizer Defect Report Form 2 Preparation inspector 3 Defect Collection Form moderator inspectors author Meeting 4 Corrected Software Artifact Defect Correction Form Follow- through Software Inspection author

6. Project Reality Yr2 • New problem: How do we help projects with regard to their inspection practices, and show benefit of new techniques? • Modularized, restructured, tailored training • Incorporated perspectives into training • Created feasible metrics collection plan • (Collected statistics from across industry) • (Collected data from SE univ. classes) • Delivered training, collected feedback. ST5(GSFC FSB) NSF CeBASE UMCP, USC LaRC personnel Yr3 Products: Modularized and updated training Defect reduction metrics collection plan

7. Project Reality • Using perspective-based approach to address project problems and re-introduce/improve inspections. • Use perspectives & project analysis to edit checklists in existing processes • Perform tailoring, provide training to projects. Compare efficiency against baselines. • Hand off training and support materials. • Create “what if” model for decision support. Yr3 Swift BAT (GSFC FSB) JWST (GSFC FSB) KI(JPL) GSFC EPG (and others) Planned Metrics & training refinementsproducts: “What if” tool Final report

8. Example of project perspective definition • Keck Interferometer (JPL) goals: • Inspect a reusable class library to make sure that existing and future stakeholders will be satisfied with the code and documentation • Get inspections reintroduced on the project • Get individuals to take ownership of different parts of the process • Achieving these through use of perspective-based reviews: • Provides a framework for identifying stakeholders and their quality interests • Tie inspection activities to project activities

9. Manager Dev/Author Tester Dev/User Scientist Roles Activities Products Example of project perspective definition • Stakeholder analysis identifies perspectives: • Example of guidance (Dev/Author): • From your point of view as author, consider which classes/methods were most complex or contained the most functionality… Use these questions to check whether the documentation matches the functionality implemented… • We are measuring # of defects, amount of effort, etc., to show that efficient perspectives and roles can be constructed KI Reusable class library (code and documentation) Specification, design, and implementation Test-ing Handoff to next project Integration

10. Example of project defect definition • Swift BAT(GSFC FSB) in late-lifecycle code inspections and testing. • Goals: Examine discrepancy report database to: • Find out what defects took the longest to close so that we can add these 'high priority' items to the inspection checklists. • Analyze which defects were caused by problems in the requirements and understand how long they stayed in the system, so that we can feed requirements recommendations forward. • Expected result: • Improved defect taxonomy used to focus inspections within FSB • Improved requirements inspection process in FSB

11. Example of using model for decision support • Inspections Process Model and Simulator goal: • To estimate the effect of adding inspections in some/all early lifecycle phases on test effort and total rework. • To help project managers understand the cost/benefits of different ways of planning inspections.

12. Rework activity Rework activity Rework activity Rework activity Rework activity Production activity Production activity Production activity Production activity Production activity V&V activity V&V activity V&V activity V&V activity V&V activity Testing Rework Example of using model for decision support • Examine effects of possible V&V activity effort as you move through the development phases:

13. Production Activity Data V&V Activity Data Rework Data Test Data Work Product (Type, Size) EffortPerItem StaffAvailable Duration DefectsInjected (Type, Severity, Number) Work Product (Type, Size) EffortPerItem StaffAvailable Duration Effectiveness (Type, Severity) DefectsDetected (Type, Severity, Number) Work Product (Type, Size) EffortPerDefect (Type, Severity) StaffAvailable Duration TestCases (Number) EffortPerTestCase Duration Effectiveness (Type, Severity) DefectsRemaining (Type, Severity, Number) StaffAvailable Example of using model for decision support

14. Conclusions • About this initiative: • We have other potential projects - all results will be included in final report. • Research direction and results have been driven by project needs • About the software development technology: • Work with projects is showing feasibility of the perspective-based approach. • Tailorability is helping to address project needs in different contexts. • Data collection (ongoing) will demonstrate whether similar levels of efficiency are being achieved compared to historical baselines.