Assurance techniques for code generators

1. Assurance techniques for code generators Ewen Denney USRA/RIACS, NASA Ames Bernd Fischer ECS, U Southampton

2. Assurance problem • Safety/mission-critical software requires assurance that it meets a certain level of “quality” • What are the issues in assuring automatically generated code? • Different forms of assurance • Different assurance techniques • Diverse generator paradigms

3. Forms of assurance What exactly might we need to assure? • Compliance with requirements • Compliance with spec/model • Certification standards • Coding standards • Absence of run-time errors • Traceability • Appropriate documentation Minimize “automation surprises” Correctness Reliability Legibility

4. Code generators in practice Practitioner survey carried out in March 2006 (Code Generators in Safety-critical Applications, J. Schumann, E. Denney); 23 responses from NASA and industry. • How are ACGs used for safety-critical applications at NASA and in industry? • Which are the primary application areas and domains? • Which tools are used? • Challenges, benefits and problems? • How could ACGs be extended to be more useful in safety-critical applications?

5. Tools and languages • The Big Three: • Real-Time Workshop • MatrixX • SCADE

6. Domains and criticality levels • Principle domains: • control • modeling/simulation • Many highly critical applications • ACG used for • production code (74%) • prototyping (52%) • simulation (48%) • testing (30%) • glue/interface code (30%)

7. System components

8. Weaknesses • Steep Learning Curve • applicable problems, features, correct usage, architecture, implied methodology, semantic ambiguities, … • substantial impact on development process • ACG customization • necessary in 1/3 of cases • often (2/3) done by tool vendor • ACG bugs • in 2/3 of applications, bugs were found in ACG

9. Qualification • A code generator is qualified • with respect to a given standard • for a given project if there is sufficient evidence about the generator itself so that V&V need not be carried out on the generated code to certify it • Must be done for every project, version • Can obtain verification credit • Generators are rarely qualified • Examples: ASCET-SE (IEC 61508), SCADE, VAPS (DO-178B)

10. Certification and V&V • Auto-generated code must be certified for safety-critical use • Techniques used: • testing (90%) • static analysis (58%) • simulation (52%) • manual review (48%) • No formal verification • No review of generator code

11. Safety properties

12. Generator features

13. Domain-specific analyses Mostly numeric issues: • stability (root locus, Lyapunov) • robustness • convergence • transience Some domain-specific design rules: • “forbidden” constructs • block structure

14. Documentation • Design information • Code derivation • Configuration management information(to “replay” generation) • Safety information • Tracing information • Interface definitions, requirements • User manuals • Installation information Should be customizable

15. Traceability • Most important: model  code • Secondary: code  V&V artifacts

16. Tool integration Also • workflow and process tools • tools for integrating legacy code

17. Survey summary • Integrated modeling, analysis, and simulation tools are most common in control domain • In-house extensions common for modeling and verification issues • Natural synergy between code generation and certification activities • perceived but not realized • autocode often treated like manual code • Iterative customization of generator should be seen as integral part of development process

18. Assurance techniques • Testing the generator (qualification) • for all specs, blocks, configurations, backends, … • Post factum verification / certification • verify / certify generated programs individually • Correctness by construction • generator inherently guarantees certain properties • Documentation • Traceability

19. Discussion questions • What are the interesting assurance artifacts, properties, etc. in your target domains? • What are suitable notions of documentation, traceability, development process? • What assurance techniques have you tried? • How is the generative knowledge represented (templates, transformation rules, etc.) and how can it be combined with assurance information? • Can we apply Design for Verification (D4V) to generators?