Trilinos Lifecycle Model 2.0

1. Trilinos Lifecycle Model 2.0 Roscoe A. Bartlett Trilinos Software Engineering Technologies and Integration Lead Computer Science and Mathematics Div Trilinos User Group Meeting, November 3, 2011

2. What is Trilinos? • Trilinos is a collection of unrelated unspecified software that share a common build, test, and distribution system • Software quality, usability, etc. is purely the responsibility of individual package developers • Customers must evaluate each and every package on their own to determine quality, suitability, etc. • Customers must try to use individual packages together and negotiate with various package development teams to resolve incompatibilities Extremes • Trilinos is a collection of well-designed, reliable, interoperable, composable, consistent, sustainable software components • Requires more coordination between Trilinos package developers • Requires the development of some standards and approaches to address them • Requires greater software quality practices and overhead The stated goal for Trilinos is clearly the second, but we are really only doing a great job of the first right now.

3. Trilinos Lifecycle Model 2.0 Document Table of Contents

4. Current State of Trilinos Software Engineering • Separation of Primary Stable (PS), Secondary Stable (SS), and Experimental (EX) code for testing purposes • Increasingly more extensive automated nightly regression testing and portability testing (on a growing list of platforms) of PS and SS code posting results to CDash • Synchronous (pre-push) CI testing of PS code using the Python tool checkin-test.py • Asynchronous (post-push) CI testing of PS and SS code using package-based CTestdriver posting results to CDash • Strong compiler warnings enabled with flags with g++ such -ansi -pedantic –Wall -Wshadow-Woverloaded-virtual • Strong policies on the maintenance of 100% clean PS and SS builds and tests for all Nightly and CI builds • Regular quarterly releases of Trilinos

5. PS and SS Code: Defined • Sub-categorizations of “stable” code: • “Primary Stable” code is “Stable” code that only depends on: • C, and C++ compilers • Fortran 77 compiler (optional) • BLAS and LAPACK • MPI • “Secondary Stable” code • Has additional dependencies such as: • SWIG/Python (i.e. PyTrilinos) • Fortran 2003+ (i.e. ForTrilinos) • External direct sparse solvers like UMFPACK, SuperLU, etc. (i.e. Amesos adapters) • Or, could be considered “Primary Stable” Code but is excluded from pre-checkin testing • Didasko • NewPackage • ... • “Stable” code in one package can only depend on “Stable” code in other packages. • “Stable” code should by default only build “Primary Stable” code. • Enabling “Secondary Stable” code should require extra configure-time options.

6. Trilinos Testing Infrastructure and Needed Improvements Weekly Coverage Testing Nightly Regression Tests Secondary Stable (SS) CATEGORIES [BASIC NIGHTLY] (more platforms, more TPLs) Weekly Memory (Valgrind) Testing Asynchronous CI Tests Secondary Stable (SS) CATEGORIES [BASIC NIGHTLY] (post-push CTest/Cdash, Linux/GCC) Performance Tests Secondary Stable (SS) CATEGORIES PERFORMANCE (CTest/CDash) Correctness Testing Synchronous CI Tests Primary Stable (PS) CATEGORIES BASIC (pre-push checkin-test.py) Scalability Tests? Secondary Stable (SS) CATEGORIES SCALING??? (CTest/CDash) • Primary Stable (PS), Secondary Stable (SS), and Experimental (EX) Code is about what gets tested. • PS, SS, and EX does NOT imply maturity level!

7. Self-Sustaining Software: Defined • Open-source: The software has a sufficiently loose open-source license allowing the source code to be arbitrarily modified and used and reused in a variety of contexts (including unrestricted usage in commercial codes). • Core domain distillation document: The software is accompanied with a short focused high-level document describing the purpose of the software and its core domain model (see LIME Theory Document). • Exceptionally well testing: The current functionality of the software and its behavior is rigorously defined and protected with strong automated unit and verification tests. • Clean structure and code: The internal code structure and interfaces are clean and consistent. • Properties apply recursively to upstream software: All of the external upstream software that are depended on are also themselves self-sustaining software. • All properties are preserved under maintenance: All maintenance of the software maintains all of these properties of self-sustaining software (by applying Agile/Emergent Design and Continuous Refactoring and other good Lean/Agile software development practices).

8. Overview of New Maturity Levels

9. Goals for an updated Trilinos Life-cycle Model • Allow pure research and applied research with a realistic path to productionization • Provide smooth low-effort transitions from research to production in phases • Provide maximum confidence with low cost (for research results and then for real users) • Allow the use of Lean/Agile practices and processes along the way

10. Overview of the New Maturity Levels • Purely Experimental (PE) Code • Research Stable (RS) Code • Production Growth Stable (PGS) Code • Production Maintenance Stable (PMS) Code

11. PS and SS Tested Codevs.RS, PGS, and PMS Maturity PE PS EX RS PGS PMS Testing Categories: PS = Primary Stable SS = Secondary Stable EX = Experimental Maturity Levels: PE = Purely Experimental RS = Research Stable PGS = Production Growth Stable SS

12. 1: Purely Experimental Code • Generally not developed in a Lean/Agile consistent way from the very beginning, does not provide sufficient unit (or otherwise) testing to prove correctness, • Could actually be declared to be Secondary Stable code with respect to CI and nightly Trilinos testing but in general would be considered to be untested Experimental code in Trilinos • Likely has a messy design and code base, • No one should use such code for anything important (not even for research results but in the current CSE environment, publication of results using such software would likely still be allowed), • Generally should not go out in general releases of Trilinos (but could go out in releases and is allowed by this lifecycle model), and • Does not provide a direct foundation for creating production-quality code.

13. 2: Research Stable Code • Developed in a Lean/Agile consistent way, • Strong unit and verification tests (i.e. proof of correctness) written as the code/algorithms are being developed (should have almost 100% line coverage at the very least), • Could be treated as Primary Stable or Secondary Stable code with respect to testing, • Has a very clean design and code base maintained through Agile practices of emergent design and constant refactoring, • Generally does not have higher quality documentation, user input checking and feedback, space/time performance, portability, or acceptance testing, • Would tend to provide for some regulated backward compatibility but may not, • Is appropriate to be used only by “expert” users, • Is appropriate to be used only in “friendly” customer codes, • Is appropriate to be part of a general release, and • Provides a strong foundation for creating production-quality software.

14. 3: Production Growth Stable Code • Includes all the good qualities of Research Stable code, • Provides increasingly improved checking of user input errors and better error reporting, has increasingly better formal documentation (Doxygen, technical reports, etc.) as well as better examples, tutorial material, etc., • Maintains clean structure through refactoring of the code and user interfaces to make more consistent, easier to maintain etc., • Maintains increasingly better regulated backward compatibility with few (if any) truly incompatible changes with new releases, as increasing better portability, • Has increasing better space/time performance characteristics, and • Has expanding usage in more customer codes.

15. 4: Production Maintenance Stable Code • Includes all the good qualities of Production Growth Stable code, • Primary development includes mostly just bug fixes and performance tweaks, • Maintains rigorous backward compatibility with typically no deprecated features or any breaks in backward compatibility, and • Could be maintained by parts of the user community if necessary (i.e. as “self-sustaining software”).

16. Typical (non-Lean/Agile) Lifecycle Unit and Verification Testing Portability Acceptance Testing Research Production Growth Production Maintenance Research Production Growth Production Maintenance Research Production Growth Production Maintenance Code and Design Clarity Documentation and Tutorials Space/Time Performance Research Production Growth Production Maintenance Research Production Growth Production Maintenance Research Production Growth Production Maintenance User Input Checking and Feedback Backward compatibility Research Production Growth Production Maintenance Research Production Growth Production Maintenance

17. Lean/Agile Consistent Lifecycle Unit and Verification Testing Portability Acceptance Testing Research Stable Production Growth Stable Production Maintenance Stable Research Stable Production Growth Stable Production Maintenance Stable Research Stable Production Growth Stable Production Maintenance Stable Code and Design Clarity Documentation and Tutorials Space/Time Performance Research Stable Production Growth Stable Production Maintenance Stable Research Stable Production Growth Stable Production Maintenance Stable Research Stable Production Growth Stable Production Maintenance Stable User Input Checking and Feedback Backward compatibility Research Stable Production Growth Stable Production Maintenance Stable Research Stable Production Growth Stable Production Maintenance Stable

18. End of Life? Long-term maintenance and end of life issues for Self-Sustaining Software: • User community can help to maintain it • If Trilinos project is disbanded, users can take parts of Trilinos they are using and maintain it long term • Can stop being built and tested if not being currently used • However, if needed again, software can be resurrected, and continue to be maintained NOTE: Git actually greatly helps in reducing risk and sustaining long lifetime issues.

19. Regulated Backward Compatibility

20. Need for Backward Compatibility SIERRA Y+1 (released against Trilinos SIERRA Y+1) Xyce J+1 (released against Trilinos X) VTK M+1 (released against Trilinos X+1) Trilinos SIERRA Y+1? • Multiple releases of Trilinos presents a possible problem with complex applications • Solution: • => Provide sufficient backward compatibility of Trilinos X through Trilinos SIERRA Y+1

21. The Cost of Maintaining Backward Compatibility • Static interfaces and design degrades as new unanticipated functionality is added • Leads to messy software that is hard to understand and dangerous to modify • Hacks to get around design problems make the software more fragile • Backward compatibility must be tested • New tests must be written and maintained by manual labor • Backward compatibility tests must be built and run • Example: Change in [TEUCHOS_]TEST_FOR_EXCEPTION(…) macros. Bottom Line => Maintaining backward compatibility increases “technical debt”

22. The Paradox of Backward Compatibility and Agile Development • Agile software development: • Design changes as functionality is added and modified over multiple releases • Emergent Design: Design changes as the domain is better understood and functionality is added. • Continuous Refactoring: Keep “conceptual integrity” with clean design and clean code. • Requires close customer interaction and feedback to help drive software development • However: • Many customers don’t want to use software while it is constantly changing • The paradox: • Agile developed software must change as it is developed and must have real feedback from customer use • Customers driving Agile developed software don’t want to be constantly chasing changes. The solution => Regulated Backward Compatibility!

23. Regulated Backward Compatibility • Trilinos Version Numbering X.Y.Z: • X: Defines backward compatibility set of releases • Y: Major release (off the master branch) number in backward compatible set • Z: Minor releases off the release branch X.Y • Y and Z: Even numbers = release, odd numbers = dev • Makes logic with Trilinos_version.h easier • Special Exception: Let X+1.0 be the development version leading to X+1.0 release • Allows X+1.0 to be the first major release as clear message to users • Backward comparability between releases • Example: Trilinos11.6 is backward compatible with 11.0 through 11.4 • Example: Trilinos 12.X is not compatible with Trilinos 11.Y Drop backward compatibility of 12.0 with 11.Y! How to manage this? 12.2 12.4 12.5 (Dev) 12.0 (Dev) 11.2 11.4 11.6 [Future] Test backward compatibility of Dev with current release every night! 11.0 12.0

24. Three Apps that Depend on Trilinos App3 App2 Trilinos App1

25. Releases with Three Apps and Trilinos App3 J (App2 M, App1 K, Trilinos X) App3 J+1 (App2 M+1, App1 K+1, Trilinos X+3) App3 App2 M (App1 K, Trilinos X) App2 M+1 (App1 K+1, Trilinos X+2) App2 App1 K (Trilinos X) App1 K+1 (Trilinos X+1) App1 Trilinos X+1 Trilinos X+2 Trilinos X+3 Trilinos X Trilinos Time

26. Regulated Backward Compatibility: Guidelines • Prepare users for the break in backward compatibility: • Deprecate code to create compiler warnings in previous versions (see __deprecated__ attribute in GCC and Intel) • Fail big and hard when backward compatibility is dropped: • Code should not even compile, and compile errors should be clear • Otherwise, code should not run at all and fail hard • Generate good compile or runtime error messages • Provide for safe straightforward upgrades: • Provide sed-like scripts for making baulk changes? • Example: Change all class and files names ‘VectorBase’ to ‘Vector’, etc. • Allow namespace changes? • Example: TpetraNew::Vector => Tpetra::Vector for Map refactoring Take Home Message: Plan for it and think about the user!

27. Deprecation Approaches • Deprecate classes and functions using the standard <UCPACAKGENAME> DEPRECATED macro: This macro expands to the GCC deprecated attribute when Trilinos is configured with Trilinos SHOW DEPRECATED WARNINGS = ON. • Deprecate macros by calling dummy deprecated functions using the standard <UCPACAKGENAME> DEPRECATED macro: #define OLD_MACRO_NAME(…) { SomeDeprecatedFunction(); NEW_MACRO_NAME(…) } • Deprecate old class names using deprecated typedefs or macro defines: Preferred (nontemplates): typedef UCPACKAGENAME_DEPRECATED NewClassNameOldClassName; Templated classes: #define OldClassNameNewClassName; • Must also deprecate the header file with #warning (see below) • Deprecate header files by inserting protected #warning directives: #ifdef __GNUC__ # waning This header OldHeader.hpp is deprecated. Please use NewHeader.hpp #endif

28. Other Considerations

29. Risk Analysis and Acceptance Testing • NOTE: • Trilinos provides tools for creating end-user applications but has no end-user applications • Risk analysis must be driven by specific end user use cases • Customer responsibility: • Do their own risk analysis • Communicate their risks down to Trilinos developers as requirements (through Agile feedback) • Trilinos Developers responsibility: • Write good algorithms with good tests, good user input checking, etc. • Listen to customers through Agile feedback

30. Grandfathering of Existing Packages • ???

31. Next Steps? • How to identify and enable software based on maturity level: • Allow users to set the minimum maturity level: • -D Trilinos_ENABLE_MINIMUM_MATURITY_LEVEL=PRODUCTION_MAINTENANCE_STABLE • (or PRODUCTION_GROWTH_STABLE or RESEARCH_STABLE) • Assign a maturity level to each package in TrilinosPackages.cmake • Assign a maturity level to each subpakage in a package (High maturity level package can have some lower maturity level subpackages) • Allow code to be otherwise ifdefedetc. to control enables based on Trilinos_ENABLE_MINIMUM_MATURITY_LEVEL. • How to assess maturity levels? • Define standards and metrics for various areas to help define maturity levels? • Set up process to review packages and assign maturity levels? • Let package teams pick their own maturity levels? • Other issues: • How do we deal with existing legacy packages? => All modifications should be done according to Legacy Software approach (1. cover with tests, 2. add functionality, 3. refactor for simplicity)