Software Design, Verification and Validation

1. Software Design, Verification and Validation Elements of a Code Integrity Management System for Real-time Embedded Systems Joao H. Silva, Ph.D. jsilva6@visteon.com joaosilva@comcast.net

2. Problem Definition Typical Scenario: • 1 Senior Manager  178 – 200 project releases • 2 releases with major Problems • 176 releases without any major problem Question: • How do we anticipate those two “Bad Releases” before they become a problem

3. Consider These Statistics • Statistic #1: For every thousand lines of code developed by software engineers, there could be as many as 20 to 30 defects on average • Statistic #2: In 2003, Developer News reported 50 percent of all bug fixes are incorrectly performed the first time, often introducing new bugs in the fixing process • Statistic #3: As bugs progress through the development cycle, defects found become exponentially more expensive to fix – it is at least 30 times more costly to fix software in the customer versus during development – SEI 2001 • For automotive releasing a bug into the field may promote a very costly recall of that vehicle(s)

4. Software Defects • Software defects are inevitable. • “…people who write software are human first and programmers only second – In short, they make mistakes, lots of them...” The Economist 2003

5. Where is the complexity? Avionics Automotive 2010 Premium  100 M LOC 1995 – 2000  52%/Year 2001 – 2010  35%/Year Tony Scott, GM CIO • Boeing 747  0.4 M LOC • Boeing 777  4 M LOC Technology Review 2002

6. Use of Models • CMM • CMMi • Complexity Models • Metrics Maturity Model • TMM –Testing Maturity Model • Others

7. A model to understand complexity PROBLEM HW Solution #1 HW/SW Co-Design SW Solution #1 COMPLEXITY Error Proneness Size EFFORT Reliability Usability Change Maintainability Understandability MIPS Productivity Solution #1 Quality Solution #1

8. What influences the complexity of SW? Size Error-Proneness Environment Competence Methods Unnecessary functionality Etc… • Module length • Tools • Language • Features • Management • Reuse • Outsourcing • Unnecessary functionality • Etc…

9. It is a multi-dimensional problem Cost Reduction Performance improvement Meet Requirements Architecture roadmap Reuse Objectives Others Improve Scalability Identify Problems

10. Who will succeed? • The companies that will succeed will be those that can develop high-quality software faster, more reliably, and more cost-effectively than their competitors.

11. Typical Software Activities Product Requirements R Architectural Analysis Detailed Designs Requirements Analysis Coding Release R R R Integration Testing Unit Testing Functional Testing R R Hardware Design Product Validation

12. Process Maturity Levels Related to Metrics

13. Level 2: Repeatable Process on Individuals Control • Budget Construct the System • Size • Volatility • Size • Time Input Output • Size • Experience Personnel • Software Size • Non-Commented Source Lines of code • Feature Function Points • Object or method count • Requirements Volatility • Requirements changes • Engineers experience • Domain/application • Development architecture • Tools and Methods • Overall years of experience • Personnel Effort • Actual person-month of effort • Reported person-months of effort Employee turnover

14. Level 3: Process is defined and institutionalized Design method Integration Method Detailed Design method Unit Testing method • Complexity • Quality • Complexity • Quality Requirements Unit Tested Architectural Design Coding Unit testing • Complexity • Quality • Complexity • Quality High-Level Design Detailed Design Integration Testing Detailed Design Tested System • Requirements complexity • Number of distinct objects • Number of actions addressed • Test complexity • Path count • Number of interfaces to test • Design complexity • Number of design modules • Cyclomatic complexity • Quality Metrics • Defects discovered • Defect density – defects discovered per unit size • Requirements defects discovered • Design defects discovered • Code defects discovered • Fault density for each project • Code complexity • Number of code modules • Cyclomatic complexity

15. Level 4: Managed Process Reporting requirements from senior management Manage Process Product People [Metrics Database] • Design defects • Distribution of defects • Productivity of tasks • Planned vs. actuals • Resource allocation Redesign Directive High-Level Design

16. Why is it important to achieve level 4? • Process type • Assess reuse opportunities • Defect identification and classification • Analyze defect density • Assess project completion

17. Level 5: Optimized Process • An optimizing process is the highest level of the process maturity levels • Measures from the activities defined before are used to tailor the process

18. Why is it important to achieve level 5? • Continuous assessment of the process • Refine the appropriate metrics • Use metrics to recommend new metrics, tools, and techniques • Estimate project size, costs, and schedule • Create project databases • Assess project costs and schedule • Evaluate project quality and productivity

19. Building Code Integrity Management Systems • Requirements Integrity Management System • Design and Implementation Integrity management System

20. Requirements Integrity Mgt System Customer Requirements Essential Model Functionality R R Requirements Analysis Technical Constraints Computing power Memory Requirements Network Bandwidth Worst Case Execution Time Technical Model Product Validation R Functional Testing Quality Attributes Performance Maintenance Safety Reliability Flexibility R Quality Attributes Hardware Requirements Time direction Specification Baseline Essential Model Technical Model Quality Attributes

21. Requirements Integrity System New Requirements Executable Specifications Component Reuse HW Test Procedures Component Models System Requirements SW V&V Test Plans ME V&V Test Scripts COTS ETC… Change Management System Filtering and Reporting Configuration Management System

22. Problem Areas • Maturity: Mostly operating at CMMi L1 – L2 • Metrics: CMMi L1 – L2 • Testing: TMM L1 – L4 • Executable Specifications: Virtually non existent except for the HMI-Graphics, • Environment: Requirements and Validation teams need to work throughout the entire development life cycle

23. Implementation Integrity Mgt System Improvement Plan Standards Certification [Quality Group] New Code Quote Requirements Gen Code Requirements Design Legacy Code Design Coding Release Vault COTS Coding Unit Testing Release Integration Testing Validation Other Change Management System Configuration Management System Other Validation New Improvement Plan (revised) Filtering and Reporting Status Report Quality Report Productivity Report

24. Test Maturity Model

25. Test Maturity Model

26. Test Maturity Model

27. Conclusions • Maturity of Automotive Designs: Designs and architectures for automotive systems is a moving target. Change is continuous. Both the end-user needs more features and the developer devises more elaborated solutions. • SW Complexity: Complexity of the automotive software increases at an unprecedented pace. • “No Silver Bullet” or “No Silver Bullet Re-Fired” • Integrity management System: Elements of this system have been implemented and rolled out with various degrees of success… • Software reuse • Case Tools • Higher-Level Languages • Model-Driven Methodologies • Extreme programming and agile software development methods • Auto-Code and Auto-Testing

28. Q&A

29. Architecture An architecture is the set of significant decisions about the organization of a software system, the selection of the structural elements and their interfaces by which the system is composed, together with their behavior as specified in the collaborations among those elements, into progressively larger subsystems, and the architectural style that guides this organization – these elements and their interfaces, their collaborations, and their composition. Krutchten – The Rational Unified Process Booch, Rambaugh, and Jacobson –The UML Language User Guide, Addison-Wesley, 1999