OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)

OSMA SAS 08Infusion of Perspective-Based Inspections forNASA IV&V(Technical Briefing) Dr. Forrest Shull, FCMD Kurt Woodham, L-3 Communications MAC-T IVV-08-162

Outline • Introduction to Perspective-Based Inspection (PBI) • Tailoring of PBI for the IV&V Context • Evaluation Plan MAC-T IVV-08-162

Software Inspection • A long history of research & application shows that structured human inspection is one of the most cost-effective practices for achieving quality software • “Cost savings rule” – Cost to find & fix software defects is about 100x more expensive after delivery than in early lifecycle phases, for certain types of defects. • IBM: 117:1 between code and use • Toshiba: 137:1between pre- and post-shipment • Data Analysis Center for Software: 100:1 • “Inspection effectiveness rule” – Reviews and inspections find over 50% of the defects in an artifact, regardless of the lifecycle phase applied. • 50-70% across many companies (Laitenberger) • 64% on large projects at Harris GCSD (Elliott) • 60% in PSP design/code reviews (Roy) • 50-95%, rising with increased discipline (O’Neill) • … many others MAC-T IVV-08-162

Software Reading Techniques: Procedural guidance incorporating tested best practices for individual preparation Roles Activities Products Improving Inspections Software Artifact 1 PlanningForm Planning organizer Defect Report Form 2 Preparation inspector 3 Defect Collection Form moderator inspectors author Meeting 4 Corrected Software Artifact Defect Correction Form Follow- through Software Inspection author MAC-T IVV-08-162

Perspective-Based Inspection (PBI) • A procedural technique incorporating three best practices: • Giving each reviewer a particular and unique focus (or perspective) on the document under review • Base perspectives on document stakeholders to help understand necessary participants, relate inspection tasks to development skills • Making individual review of a document an active (rather than passive) undertaking • Articulating the quality aspects of interest MAC-T IVV-08-162

Perspective-Based Inspection (PBI) • History • 2001-2003: SARP research funding • Refined basic approach, implemented on NASA projects and in classrooms, measured results • 2004: Tech infusion with Flight Software Branch / GSFC • Goal: Produce new Branch standards • Success metric: Based on experiences on pilot(s), how much of the recommended process is adopted by Branch? • 2004: Tech infusion with United Space Alliance / JSC • Goal: Reduce defect slippage over current increment • Success metric: Show reduced defect slippage on PB-inspected artifacts • 2007-2009: SARP funding for improved general inspection planning • Other Results and Benefits • Improved defect detection effectiveness substantially (NASA & elsewhere) • Helped re-invigorate inspection practices • Forms the basis of industrial and academic training MAC-T IVV-08-162

Adopting PBI for IV&V • Pathfinder: Infuse PBI into the IV&V practices being applied to JWST • JWST Project/Developer artifact review • JWST IV&V System Reference Model (SRM) validation • Technology Provider… • Adapts PBI to IV&V domain and JWST-specific context • Administers training • Facilitates initial reviews • Monitors efficacy of PBI and documents results/recommendations • JWST IV&V Team… • Receives training • Conducts reviews in-line with IV&V execution • Provides ancillary feedback to Technology Provider (minimal disruption) • IV&V Facility Research… • Monitors progress and determines next-step expansion based on JWST pathfinder MAC-T IVV-08-162

JWST System Overview • Launch in 2013 • Collaboration between NASA, ESA, CSA • Managed by GSFC • NGST – prime contractor • STScI – operator after launch • Integrated Instruments • Mid-Infrared Instrument (MIRI) • Near-Infrared Spectrograph (NIRSpec) • Near-Infrared Camera (NIRCam) • Fine Guidance Sensor (FGS) MAC-T IVV-08-162

JWST Observatory Diagram Image credit: http://www.jwst.nasa.gov/observatory.html MAC-T IVV-08-162

JWST IV&V Overview • IV&V Scope • Observatory and Ground System segments • SRM-based approach starting with top-level user needs flowed systematically down through segment, elements, subsystem, and component levels • Specific focus areas driven by IV&V criticality/risk analysis with close coordination with JWST project • WBS driven • SLP 09-1 Rev J (http://www.nasa.gov/centers/ivv/ims/slps/index.html) • Defines validation and verification tasking to be performed at various levels of system decomposition • Emphasis on early validation of requirements with respect to system needs, goals, and objectives in light of three validation questions • Validated SRM plays pivotal role in validation and verification activities throughout IV&V life-cycle MAC-T IVV-08-162

Document with defects High overlap Low coverage (Ad hoc) Low overlap High coverage (PBI) Adopting PBI for IV&V: Perspectives • Motivation: • Focus the responsibilities of each inspector • Minimize overlap among inspector responsibilities • Maximize union of defects found MAC-T IVV-08-162

System Ref Model Adopting PBI for IV&V: Perspectives • Tailoring required for new domain, new issues, new artifacts Is model complete, correct, consistent, and testable? Model validation Model construction Needs, goals, objectives elucidation Analysis val./ver. target Do sources adequately define user needs? Do behaviors defined in val./ver. targets map to equivalent definition in model? MAC-T IVV-08-162

Adopting PBI for IV&V: Defect Types • Interpretation of “defect” • Avoid focusing on correctness as the one and only quality property • Any issue that would cause a problem for a stakeholder downstream. MAC-T IVV-08-162

Adopting PBI for IV&V: Defect Types • Other sources of defect categories: • Checklists and defect categories from previous work • Experienced inspectors • What is it that the most experienced people are looking for? • Can get from: • Asking them! • Noting recurring questions at inspection meetings • Defect/discrepancy report databases • What types of problems seem to be common? • What types of problems take the longest to close? MAC-T IVV-08-162

Adopting PBI for IV&V: Active Review • Reading through a document one page at a time is not the most effective approach. • In PBI every inspector receives a scenario to guide his/her work. • Based on your perspective… • Imagine you are creating some other artifact you need as part of your job… • E.g. for tester perspective, think about building a test plan • Then read through the document as you normally would to get the necessary information. • Inspectors have questions about quality to answer while they follow the scenario. MAC-T IVV-08-162

Adopting PBI for IV&V: Summary • Initial perspectives for this domain: • Modeler • Uses requirements & existing models to create the next iteration • Quality foci: Appropriate level of detail in requirements; completeness and clarity of requirements & models; consistency and correctness of models • Domain expert • Assesses whether model accurately and usefully captures domain • Quality foci: Correctness of requirements (independent of models); identification of stakeholders and evaluation of usability from their POV; checking flow of control & use of reusable components • Quality assurance • Assesses whether models and system can be validated properly • Quality foci: Handling of exception cases / unexpected system conditions; robustness of system; testability MAC-T IVV-08-162

Infusion Plan • Understand team-specific quality concerns • Understand likely perspectives for the team • Refine set of perspectives and define procedures for each. E.g., • What quality concerns map to which perspectives • What is a feasible/effective process for checking those concerns • Finalize procedures and provide training • Analyze ongoing inspection results (and update procedures if necessary) • Monitor downstream defect profile • Write final report Status: Reviewed sample models & processes; Constructed draft PBI techniques; Created draft training course MAC-T IVV-08-162

Infusion Plan: Path Forward • Evaluation • Success metric: Effectiveness of PB inspection in comparison to past inspections • Current JWST approach • Project documents • Issues written against missing/incorrect/incomplete/inconsistent content in source artifacts • SRM Reviews • IV&V team maintains SRM change logs, action item lists, and other sources for the infusion team to evaluate relative effectiveness • IV&V team documents discrepancy when comparing SRM against val/ver target • Strategy is to maintain current approach (issue reports, review records), but to track how PBI assisted in identifying the items • Likely to involve follow-up conversation/interviews to assess results and collect ancillary observations from the IV&V team • Will balance this with objective to minimize impact to IV&V activities MAC-T IVV-08-162

Future Considerations • Use measurement results to further optimize, e.g., • Consistently missing a certain type of defect? • Add a new defect type and associated questions to relevant perspectives • Missing a whole set of defects? • Consider whether a new perspective should be added. • Do we never find defects of type x? Are there any perspectives that don’t seem to add much that the other perspectives didn’t find? • Consider deleting the relevant defect type or perspective. • Do we consistently face a lot of problems of type x? • Add more reviewers using relevant perspectives MAC-T IVV-08-162

Contact information Forrest Shull fshull@fc-md.umd.edu 301-403-8970 Kurt Woodham Kurt.Woodham@L-3com.com 757-644-5807 MAC-T IVV-08-162

OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)