Parasoft ADP solutions Automated Defects Prevention for Embedded Systems Software Development

1. Parasoft ADP solutions Automated Defects Prevention for Embedded Systems Software Development by Wiktor Grodowski

2. Observations on Software Testing • Mostly focuses on Functional Testing tied back to Requirements • Design – Code – “Integration and Test” • Early Testing is minimal and short-lived • Testing efforts are focused on the isolation and subsequent fixing of “bugs” – Error Detection • “Failure to Communicate” between Development, Test and Quality Assurance (QA) .. Not much has changed in 25 years ..

3. The Cost of Waiting

4. What is ADP all about? • ADP = Automated Defects Prevention • Defects Prevention != Error Detection • ADP = Defects Prevention + Automation • ADP is not a replacement for CMM / CMMI and alike. It is a change in a software development approach. • ADP is practical, flexible, down to earth, based on over 20 years of experience • http://www.adpqb.org

5. A brief history • ADP originates from Deming's and Jurand's works • Attempting to introduce Deming's methodology into SDLC • Follow the basic steps: • Identify a defect • Find its cause • Locate where and why it was introduced • Implement preventive methods • Monitor the process • Start preventing errors instead of detecting them!

6. Plan – establish objectives and processes Do – implement new process Check – measure the results and compare against expected outcome Act – analyze the differences, determine where to apply changes to include improvement Deming's wheel

7. ADP principles • Principle 1: Establishment of Infrastructure • Build a strong foundation through integration of people and technology • Principle 2: Application of General Best Practices • Learn from others’ mistakes • MISRA, JSF, other coding standards • Principle 3: Customization of Best Practices • Learn from your own mistakes • Principle 4: Measurement and Tracking of Project Status • Understand the past and present to make decisions about the future • Principle 5: Automation • Let the computer do it! • Principle 6: Incremental Implementation of ADP’s Practices and Policies

8. Establishment of infrastructure • People – extending traditional roles • Must promote communication, efficiency, productivity and job satisfaction • Each role must include defects prevention tasks • Understanding roles • All must understand how to adhere to the roles they are given • Define group behavior • Technology – minimum and more • SCMS • Build system • BTS • Reporting system

9. Application of best practices • Most software projects share same characteristics • Best practices can be used to limit: • Human errors • Common software defects • Where do the best practices come from? • Different granularity of best practices: • Organizational-level (EVMS – Earned Value Management System, general guidelines for use of CMS) • Design-level (suggestions for usage of specific technologies or techniques, such as design patterns, Ajax or SOA) • Code-construction level (procedures for peer programming, writing unit tests, code review etc.)

10. Customization of best practices • Must address project-specific problems • Is based on the Deming's principles • Monitor process • Identify a defect • Find cause of a defect • Locate part of a process that let the defect slip through • Modify process to prevent the defect from reappearing • Monitor process (again) • Static analysis tool should be implemented to monitor both practices automatically

11. Measurement and tracking • Measures must be quantitative • Absolute values (number of defects) • Metrics (possession of a given attribute, confidence factor) • Measurement: • What and how can be measured? • What does the measure or metric indicate? • What decisions can be based on these information? • Tracking: • What data can be tracked? • How is the data tracked? • What do we gain from tracking this data?

12. Automation • What is automation? • „Activity that can be performed completely without human intervention” • “Automation of mundane and repetitive tasks improves people satisfaction and effectiveness” • Not to substitute people but to improve working conditions for people • Automation improves product quality • Automation facilitates human communication • Automation helps to implement and enforce best practices and organizational standards • Automation improves people productivity • Automation helps to collect measurement data

13. Code reviews are time-intensive and human-intensive Review process can be done with a “static analysis” tool Quality focus is typically NOT built into product schedules so there is typically little time to do proper testing. Automation is required to leverage these precious cycles Developers write more testing procedures/code than the code they are developing for delivery! ADP “automates” much of the driver, stubs and infrastructure creation as well as provide metrics, like code coverage Developers typically do NOT like testing. Some problems will never be solved! Why Automation is Important !

14. Incremental implementation of ADP • Continuous change through: • Learning • Adaptation • Innovation • Change, however, can be overwhelming • ADP needs to be implemented gradually, on a group-by-group basis • Pilot group(s) • Division(s) • Organization • Best practices should also be introduced one-by-one

15. Incremental implementation of ADP • ADP is both iterative and incremental – the phases do not have to follow sequentially

16. code & tests SCM Source Control deployment Server Edition Test Results Reporting Professional Editions Nightly Results Company Standards TCM Team ConfigurationManager Architect Edition Exemplary ADP Deployment Target device Developer Machines Program ManagerArchitect Scheduled Build and Test Server Architect / Technical lead

17. High costs of defects Number of defects in a final product must be kept at minimal due to a high recall cost Possibility of having human life or human safety at stake, dependent on a stability of an embedded device Important time to market Late product = lost money Mandatory compliance MISRA, JSF, DO-178B, others Contractor's internal standards Relation to Embedded Systems

18. Exemplary ROI Unit testing caused drop in residual errors almost by a factor of 2. Peak revenue Peak revenue from delayed entry Revenues ($)‏ On-time Revenues ($)‏ Market fall Market rise Delayed D W 2W Time (months)‏ On-time Delayed entry entry Time Is it worth it?

19. Is it really worth it? • Non-ADP • ADP

20. C++Test Coding Standards analysis Flow Analysis Unit Testing - Coverage analysis Code Review Authoring and Automation Concerto Work Planning and Tracking Reporting and Analytics Acceptance testing Data correlation Process Visibility Insure++ Runtime memory analysis Parasoft Tools supporting ADP

21. Benefits Testing ADP Find Errors X X Uses testing results to fix errors X X Uses testing results to fix the development process X Reduces probability of errors reoccurring X Addresses the critical roles in creating software X Learns from test results and improves processes and prevents future errors X Uses testing as a process measurement technique X Understands and anticipates common coding mistakes that can lead to poor software quality X Provides tools and processes to keep team members code at same level X Benefits of ADP

22. Questions & Answers www.parasoft-embedded.com