Assurance techniques for code generators

1. Assurance techniques for code generators Ewen Denney USRA/RIACS, NASA Ames Bernd Fischer ECS, U Southampton

2. Assurance problem • Safety/mission-critical software requires assurance that it meets a certain level of “quality” • What are the issues in assuring automatically generated code? • Different forms of assurance • Different assurance techniques • Diverse generator paradigms

3. Forms of assurance What exactly might we need to assure? • Compliance with requirements • Compliance with spec/model • Certification standards • Coding standards • Absence of run-time errors • Traceability • Appropriate documentation Correctness Reliability Legibility

4. Participants • Harold Ossher • Markus Pueschel • Julia Lawall • Ann Le Meur • Yannis Smaragdakis • Oleg Kiselyov • Tom Ellman • Gabor Karsai • Kevin Hammond • Laurence Tratt • Baris Aktemur • Walid Taha • Bernd Fischer • Ewen Denney

5. Target domains • numerical code • statistical data analysis • GN&C • physics-based animation • linear transforms • embedded systems • real-time systems • device drivers • optimizing simulators • programming language tools

6. Generator paradigms • mathematical, schema-based • templates and symbolic reasoning • source-level transformations • DSLs • AOP • template metaprogramming • staged programming • model-driven • graph-transformations

7. The Holy War?!? Thou shalt qualify thy generator vs. Certify the generated programs, Luke • Certification ≠ Verification! • Safety ≠ Correctness! • Should prove parts of the generator correct • find problems earlier: in generator rather than at compilation time • domain knowledge (much) easier to understand at higher-level than in generated code • Generate proofs that can be checked • Compositional verification • Safety is ultimately a system question

8. Some Current Approaches Distinction between generator framework and domain knowledge reflected in distinction between verification and validation • Testing generator rules in Spiral: • domain source might be wrong • formalization might be wrong • plug in parameters and check an instance of the transformation • Simulate algorithm instances in AutoFilter • Compose aspects while ensuring they don’t corrupt each other • ultimately: want behavioral equivalence • “Type systems can encode interesting things” • "Our formal abilities are laughable“

9. Traceability and Documentation • Doing it manually very tedious and error-prone • Adding "rationale system" to explain the transformation steps • Programming traceability info was harder than the rest of the system, but very important • Good for debugging, but users don't care • Relating performance model to higher-level description? • Optimization blurs boundaries • Establishing bisimulation gives trace • Tracing is much easier in “horizontal” systems rather than vertical systems

10. Bake-off A bake-off for assuring generators? • Need challenge problems, consisting of • classes of specs, • algorithms for generating programs • proofs that the algorithms are correct • …

11. Conclusions??? • In Europe, everything is proven, but nothing works. • In the US, nothing is proven, but it works. • And in code generation, nothing works and nothing is proven…