Software Acceptance: Direct Artifact Assurance

1. Software Acceptance:Direct Artifact Assurance William L. ScherlisCarnegie Mellon University Professor, School of Computer ScienceDirector, CMU/NASA High Dependability Computing Program Director, CMU PhD Program in Software Engineering scherlis@cmu.edu412-268-8741

2. Outline • Problems • Software test and inspection inadequate to assure dependability and security • Barriers in IT supply chain: off-the-shelf, outsourcing, etc. • Example: the real story of Ariane-5 • Example: Windows device drivers and blue-screens • Software acceptance • Direct assurance of software • Contrast: CMM and NIAP-CC • Examples: MSR SLAM, CMU Fluid • What’s new: • Deep technical results informed by engineering pragmatism • Focus on scalability, decomposition, usability

3. Four barriers Contractor qualification Requirements definition Engineering acceptance “Second” sourcing Mitigation(today’s best practice) CMM / CMMI Close relationships Testing, inspection, design analysis API conventionalization Assurance of critical properties—today’s best practice Interface barriers exist between producers* and consumers at all stages of an IT supply chain Problem: Testing, inspection, and design analysis are inadequate to assure security and dependability Symptom: Software failures and security defects Challenges: Subsystem decomposition, critical properties with non-locality in code, concurrency and non-determinism Some examples… *Producers: Internal development groups; subcontractors/outsources/offshore; off-the-shelf; open source; etc.

4. Ariane 5—mission critical Ariane 5 veered off course and exploded 40 seconds into its maiden flight due to software failure The failure was due to a known unhandled exception in the software—cost $1 billion Why? “Heritage Ariane 4 code” Trust in the legacy…it worked for Ariane 4 Distrust in defined criteria…too risky to modify “working” software even when it is known to be broken “Blue screen”—desktop Most occur due to faulty 3rd party device driver code—but Microsoft “blamed” by users Reputational cost to Microsoft Examples: The inadequacy of test and inspection 3rd partydevice driver WindowsOS OS API with associated integrity constraints

5. Sources of software Internally developed Mission- or security-critical Differentiating capability Business logic Outsourced custom Whole solutions Separable subsystems Off-the-shelf components Windows, OS X, Office, Oracle Open source components Apache, Linux, Tomcat, etc Mobile code JavaScript in a web page MS Word document Free players, plug-ins “Cool screensaver” virus mail Spoofed executableenclosure Basis for accepting software Trust the source “Always trust content from __” Chain of trust – certificates Explicit test and inspection Custom and outsourced May be more costly than code development itself Limited privilege – containment Sandboxing for Java, scripts Verification of safety attributes Ada/Java type integrity Assert (often based on testing) Lack of awareness Spyware and adware Configuration mgt failures Problem: Software acceptance in today’s practice Little focus on direct assurance of software code and design artifacts …

6. Focus—direct assurance and evaluation best practice What’s needed: • Direct assurance (focused tools and ongoing research)(technology-dependent; attribute focused) • Assure the software itselfQuality, dependability, security (objective analysis) Contrast with accepted best practices for evaluation: • CMM/CMMI (ISO 9001x)(timeless; comprehensive) • Evaluate the teamCost and schedule predictability • Evaluate the process (correlates with bug reduction) • NIAP/CC (ISO 15408) (including potential EAL 7) (timeless; comprehensive) • Evaluate the processSecurity policy definition • Evaluate the designDesign compliance • Sample the product*(*sampled – no direct assurance)

7. Direct assurance—industry example: Microsoft SLAM for Windows XP http://research.microsoft.com/slam/

8. Direct assurance—research example:CMU Fluid • Assure diverse “mechanical” program properties for dependability and security • E.g., race conditions, locking policy, unaliased references • Provide Java programmers with direct positive assurance of design intent • Focused on incrementality of work and programmer early gratification • Detected numerous race conditions (and developed assured fixes) for widely used production software code • E.g., Sun Java 1.4 library BufferedInputStream http://www.fluid.cs.cmu.edu

9. Direct assurance—conclusion • Today’s software evaluation practices (testing, inspection, design analysis) are necessary but not sufficient to provide guarantees critical dependability and security attributes. • Industry and lab evidence suggests that S&T focus on defect avoidance can yield useful results • New analytical techniques for assurance are starting to emerge in research and industry • The most recently developed industry tools are based on technical results of early 6.1 and 6.2 lab projects • Focus on specific engineering attributes related to dependability and security • A priori emphasis on scalability, component decomposition, and harmony with engineering best practices.