1 / 23

SA @ WV (software assurance research at West Virginia)

SA @ WV (software assurance research at West Virginia). Kenneth McGill NASA IV&V Facility Research Lead 304.367.8300 Kenneth.McGill@ivv.nasa.gov Dr. Tim Menzies Ph.D. (WVU) Software Engineering Research Chair tim@menzies,com. Before bad software. After bad software.

rorym
Download Presentation

SA @ WV (software assurance research at West Virginia)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SA @ WV(software assurance research at West Virginia) Kenneth McGill NASA IV&V Facility Research Lead 304.367.8300 Kenneth.McGill@ivv.nasa.gov Dr. Tim Menzies Ph.D. (WVU) Software Engineering Research Chair tim@menzies,com

  2. Before bad software After bad software Why, what is software assurance? • Definition: • Planned and systematic set of activities • Ensures that software processes and products conform to requirements, standards, and procedures. • Goals: • Confidence that SW will do what is needed when it’s needed. • Why software assurance? • bad software can kill good hardware. • E.g. ARIANE 5: (and many others) • Software errors in inertial reference system • Floating point conversion overflow Ariane 5

  3. OSMA Software Assurance Research Program • Office of Safety & Mission Assurance (Code Q- OSMA) • Five million per year • Applied software assurance research • Focus: • Software, not hardware • SW Assurance • NASA-wide applicability • Externally valid results; i.e. useful for MANY projects • Organization: • Managed from IV&V Facility • Delegated Program Manager: Dr. Linda Rosenberg, GSFC

  4. Many projects • Mega: highest-level perspective • e.g. project planning tools like ASK-PETE [Kurtz] • Macro: • e.g. understanding faults [Sigal, Lutz & Mikulski] • Micro: • e.g. source code browsing [Suder] • Applied to basic: • Applied: • (e.g.) MATT/RATT [Henry]: support large scale runs of MATLAB • Basic (not many of these) • e.g. Fractal analysis of time series data [Shereshevsky] • Many, many more • Too numerous to list • Samples follow • See rest of SAS! Horn of plenty

  5. Good news! • More good proposals than we can fund • Bad news! • same as the good news Many more projects! Ratio FY02/FY01 Total proposals: 2.2 NASA centers: 1.5 Industry: 26 University: 3.7

  6. A survey of 44 FY01 CSIPs 75% with no claim for project connections Need more transitions! (but don’t forget the theory)

  7. Action plan- restructure CSIPS: more transitions! • New (year 1) • Fund many • Renewed (year 2) • Continue funding the promising new projects • Recommended: letter of endorsement from NASA project manager • Transition (year 3) • Select a few projects • Aim: tools in the hands of project folks • Required: project manager involvement • Reality check: • Transition needs time • Data drought

  8. Charles Pecheur RIACS, ASE, ARC Long transition cycles • Pecheur & practical formal methods • In-Situ Propellant Production project • Taught developers: • Livingstone model-based diagnosis • model-checking tool tools • developed by Reid Simmons, (CMU) • Technology to be applied to the Intelligent Vehicle Health Maintenance (IVMS) for 2nd generation shuttles • Lutz, Mikulski & ODC-based analysis of defects • Deep-space NASA missions • Found 8 clusters of recurring defects • Proposed and validated 5 explanations of the clusters • Explanations  changes to NASA practices • ODC being evaluated by JPL’s defect management tool team Marsatmosphere on-board CO2 + 2H2 —> CH4 + O2 fuel oxidizer Robyn Lutz JPL, CS-Iowa State (no photo) Carmen Mikulski JPL

  9. The data drought Gasp… need data…

  10. End the drought:bootstrap off other systems • Find the enterprise-wide management information system • Insert data collection hooks • E.g. JPL adding ODC to their defect tracking system • WVU SIAT sanitizer

  11. End the drought:Contractors as researchers • Buy N licenses of a defect tracking tool (e.g. Clearquest) • Give away to projects • In exchange for their data • Build and maintain a central repository for that data • With a web-based query interface • Data for all take me to your data active data repository

  12. End the drought:Contractors as researchers (2) • See also: • Titan’s new ROI project • Any contractor proposing an NRA • Galaxy Global’s metric project experience 1 4 action reflection 2 Mark Suder Titan, IV&V abstraction 3 Hypertext power browser for source code } SIAT-1 4 } Use it. 1’ For high-severity errors, recall what SIAT queries lead to finding those errors 2’ SIAT2 3’ Assess each such “power queries” Reject the less useful ones Procedures manual for super SIAT or new search options in interface 4’

  13. End the drought:raid old/existing projects • Cancelled projects with public-domain software • E.g. X-34 • Or other open source NASA projects • E.g. GSFC’s ITOS: • real-time control and monitoring system during development, test, and on-orbit operations, • UNIX, Solaris, FreeBSD, Linux, PC • Free!! • NASA project connections: • Triana, • Swift, • HESSI, • ULDB, • SMEX, • Formation Flying Testbed, • Spartan

  14. End the drought:synergy groups • N researchers • Same task • Different technologies • Share found data • E.g. IV&V business case workers • E.g. monthly fault teleconferences • JPL: • Lutz, Nikora • Uni. Kentucky: • Hayes • Uni. Maryland: • Smidts • WV: • Chapman (Galaxy Global) & Menzies (WVU)

  15. End the drought:Tandem experiments • “Technique X finds errors” • So? • Industrial defect detection capability rates: • TR(min,mean,max) • TR(0.35, 0.50, 0.65) • Assumes manual “Fagan inspections” • Is “X” better than a manual 1976technique? • Need “tandemexperiments”to check • I.e. do it twice • Once by the researchers • Once by IV&Vcontractors (baseline) fictional data

  16. Alternatively:End your own drought • Our duty, our goal: • Work the data problem (e.g. see above) • Goal of CI project year1: build bridges • But the more workers, the better • Myth: there is a “data truck” parked at IV&V • full of goodies, just for you • Reality: Access negotiation takes time • With contractors, within NASA • We actively assist: • Each connection is a joy to behold, an occasion to celebration • We don’t celebrate much • Bottom line: • We chase data for dozens of projects • Researchers have more time, more focus on their particular data needs • Ken’s law: • $$$ chases researchers who chase projects • CI year2, year3: needs a project connection

  17. Alternatively (2), accept the drought and sieve the dust • The DUST project: • Assumes a few key options control the rest • Methodology: • Simulate across range of options • Data dust clouds • Too many options: what leads to what? • Summarize via machine learning • Condense dust cloud • Improve mean, reduce variance • Case studies: • JPL requirements engineering: • Feather/JPL [Re02] • Project planning: • DART- Raque/ IVV; Chaing/UBC; • IV&V costing: Marinaro/IVV, Smith/WVU • general: Raffo, et.al/PSU [Ase02] • An analysis of pair programming: Smith/WVU • Better predictors for: • testability: Cukic/WVU, Owen/WVU [Issre02, Ase02] • faults: diStefano/WVU, McGill/IVV; Chapman/GG • reuse : diStefano/WVU [ToolsWithAI02] The answer my friend, is blowin’ in the wind But wait: the times they are changing Each dot = 1 random project plan

  18. Other WVU SA research  Testing & formal methods Bayesian approach to reliability Architectural metrics Risk assessment & dynamic UML Reliability & operational profile errors Bojan Cukic Hany Ammar Katerina Goseva Popstojanova UML (sequence diagrams, state charts) Software Specs & design (early life cycle) collaborator UML simulations Architectural descriptions Goal: accurate, stable, risk assessment early in the lifecycle Static (SIAT, Mccabe, entrophy) Code analysis (iv&v,operational usage) Dynamic (testing, runtime monitoring) Metrics(complexity,coupling,entropy) Fault, failure data on components, connectors Failure data from testing Severity of failures

  19. More WVU research (FY02 UIs) Architectural metrics Risk assessment & dynamic UML Intelligent flight controllers Testing & formal methods Bayesian approach to reliability Fractal study of resource dynamics Reliability & operational profile errors SE research chair interns DUST Ammar ISS hub controller, “Dryden application” c cccccc Cukic jc X34 ITOS jj F15 SIAT w X38 c FY03 proposals = 2.2*FY02 Goseva- Popstojanova Menzies “JPL deep space mission” DART “KC-2” IVV cost models j, ccccccc, w c = conference w = workshop j = journal new renewed

  20. Design Diversity, add eight more Design Diversity, add one more Data Diversity Function Point Metrics for Safety-Critical Software • Thesis: • Traditional function-point cost estimation • Incorrect for safety-critical software • > 1 way to skin a cat • >1 way to realize a safety critical function: • NCP=N-copy programming • NVP= N-Version Programming , • NSCP= N Self-Checking Programming, • … • With, without redundancy, • Method: • explore them all! • H2 and C2 : effort & cost, redundant systemH1 and C1: effort & cost, non-redundant system Afzel Noore

  21. } Early warning Time for graceful shutdown Pre-disaster warnings [Cukic, Shereshevsky] Can we defer a maintenance cycle and keep doing science for a while longer? Mark Shereshevsky Bojan Cukic ARTS II Crash

  22. Intelligent flight controllers [Napolitano, Cukic] (and menzies) Marcello Napolitano(Mechanical andAerospace) Bojan Cukic (CSEE) Lifecycle opportunities for V&V of neural network based adaptive control systems.

  23. The road ahead: applied & theoretical research Need both CSIPs: applied research USIPs: applied + theoretical research To boldly go…

More Related