1 / 26

Trilinos Usage of CMake/CTest/CDash

Learn about the motivations for switching to Trilinos CMake/CTest/CDash, the gains and losses, and the design principles for the build system.

tbaldwin
Download Presentation

Trilinos Usage of CMake/CTest/CDash

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2009-1098P Trilinos Usage of CMake/CTest/CDash Roscoe A. Bartlett http://www.cs.sandia.gov/~rabartl/ Department of Optimization & Uncertainty Estimation Sandia National Laboratories Kitware/SNL CMake/CTest/CDash Workshop May 7, 2009 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

  2. Primary Motivations for Switching Trilinos • Native support for MS Windows (MS Visual Studio projects and self-extracting installers) and Mac (XCode project files and binary installers) • Improve robustness of our development and testing environments • Improve the stability of important Trilinos code • Support stronger collaborations with important customer codes (i.e. Almost Continuous Integration) • Better support R&D of deep stacks of vertically integrated algorithms CMake Transition • Decrease the maintenance costs of the framework and testing infrastructure • Reduce maintenance cost by 1/2 to 1/3! • Allow Jim Willenbring and Brent Perschbacher to do more important things Autotools Total Effort CMake Time

  3. Gains & (Initial) Looses Switching to CMake for Builds • What we gain: • Full dependency tracking of every kind possible on all platforms (i.e. header to object, object to library, library to executable, and build system files to all built files) • Support for shared libraries on a variety of platforms • Support for MS Windows (i.e. Visual Studio projects, Windows installers, etc.) • Simplified build system and easier maintenance (extremely easy to add new packages and maintain existing packages) • Improved mechanism for extending capabilities (as compared to M4 in autotools) • Ability to affect the development of the build tools with good existing collaborations • Significant ``in house'' knowledge-base • One hundred percent automated intra-package dependency tracking and handling (built into the prototype Trilinos/CMake build system) • What we lose: • CMake requires that all uses have 'cmake' installed on their machine when building from source and users will need to have at a very recent version of cmake. (However, cmake is very easy to build from source)

  4. Gains (no losses) Switching to CTest/CDash for Testing • What we gain: • Test time-outs (this is a major maintenance issue for the current Perl-based test harness) • Memory testing with Valgrind and purify that is backed up by Kitware and a larger development community • Line coverage testing that is backed up by Kitware and a large development community • Support for selecting and excluding subsets of tests based on regular expressions or named keywords • Better integration with the build system (e.g. easier to support more advanced features like PBS batch systems and flexible testing control) • Better tracking of specific tests (i.e. each and every test can have a unique name that is easy to find) • What we lose: • Nothing!

  5. Design Principles for Trilinos CMake Build System: #1 • Make it exceedingly easy to define CMake files for new packages and to define libraries, tests, and examples in those packages. • Create a design for building individual package CMake files that automatically results in uniformity of how things are done. • Use standard macros to define every package's main features • Simplify maintenance. • Allow changes to logic and functionality that apply to all Trilinos packages without having to touch each individual Trilinos package's CMake files. • Provide 100% automatic intra-package dependencies handling • Avoid mistakes • Avoid duplication • Robustifies a number of important features. • Provide built-in automated support for as many critical software engineering practices a possible • Proper pre-checkin testing when continuous integration is being performed

  6. Design Principles for Trilinos CMake Build System: #2 • Avoid duplication of all kinds as much as possible. • Fundamental software maintenance issue. • The build system should be able to reproduce 100% update-to-date output by simply typing ‘make’. • E.g. Coping test input files to binary directory • Aggregate as much common functionality as possible to the top-level CMake files but allow individual CMake packages to refine the logic if they really need to. • Accept extra complexity at the global framework level to reduce it at the package level • Apply solid software engineering design principles to manage the complexity • Remove duplication

  7. Outline of CMake PackageArch Build System Trilinos/ CMakeLists.txt # Top-level build file cmake/ # Cmake support TrilinosPackages.cmake # List of Trilinos packages TrilinosTPLs.cmake # Lists of Trilinos TPLs ... package_arch # Generic package architecture macros PackageArchGlobalMacros.cmake PackageMacros.cmake PackageLibraryMacros.cmake PackageAddExecutable.cmake ... ... packages/ ... epetraext/ CMakeLists.txt # Top-level package build file cmake/ Dependencies.cmake # Defines intra-package dependency and TPL lists EpetraExt_config.h.in # Copied from ../src/ and hand modified src/ CMakeLists.txt # Defines library sources and library(s) test/ CMakeLists.txt MatrixMatrix/ CMakeLists.txt # Define actual test executables and test runs ...

  8. Adding Packages in TrilinosPackages.cmake SET( Trilinos_PACKAGES_AND_DIRS_AND_CATEGORIES Teuchos teuchos PS RTOp rtop PS Epetra epetra PS Triutils triutils PS EpetraExt epetraext PS Thyra thyra PS RBGen rbgen SS ForTrilinos ForTrilinos EX PyTrilinos PyTrilinos SS ... ) • Adding a new Trilinos Package is a 1-line addition at the Framework Level! • NOTE: The packages must be listed in a order of strictly increasing dependences! • NOTE: If you get the ordering wrong, the automated dependency handling CMake scripts will automatically detect this and issue a very good error messages before the build is performed! • PS = Primary Stable Code, SS = Secondary Stable Code, EX = Experimental Code

  9. Defining a Package’s CMake Build Files Trilinos/packges/ epetraext/CMakeLists.txt • Utility macros provide framework hooks into package functionality • Defines common behavior across all packages • Avoids duplication • Facilitates maintenance INCLUDE(PackageMacros) INCLUDE(AddOptionAndDefine) # # A) Define the package # PACKAGE(EpetraExt) # # B) Set up package-specific options # ADD_OPTION_AND_DEFINE(${PACKAGE_NAME}_BUILD_TRANSFORM HAVE_TRANSFORM "Enable transform functionality." ON ) ... # # C) Add the libraries, tests, and examples # ADD_SUBDIRECTORY(src) PACKAGE_ADD_TEST_DIRECTORIES(test) # # D) Do standard postprocessing # PACKAGE_POSTPROCESS() epetraext/cmake/Depencencies.cmake SET(LIB_REQUIRED_DEP_PACKAGES Epetra Teuchos) SET(LIB_OPTIONAL_DEP_PACKAGES Triutils) SET(TEST_REQUIRED_DEP_PACKAGES) SET(TEST_OPTIONAL_DEP_PACKAGES) SET(LIB_REQUIRED_DEP_TPLS) SET(LIB_OPTIONAL_DEP_TPLS) SET(TEST_REQUIRED_DEP_TPLS) SET(TEST_OPTIONAL_DEP_TPLS) • Intra-package dependencies ! • Used in all intra-package dependency handing • All header-file paths and link libraries and directories are handled automatically • These dependencies can not be wrong! (i.e. the libs and execs would not build and link otherwise) • Optional dependencies create variables <PACKAGE>_ENABLE_<DEP> and HAVE_<UCPACKAGE>_<UCDEP> !

  10. Automatic Intra-Package Dependency Handling $ ./do-configure -D Trilinos_DUMP_PACKAGE_DEPENDENCIES:BOOL=ON Configuring Trilinos build directory Printing package dependencies ... -- Teuchos_FORWARD_LIB_REQUIRED_DEP_PACKAGES='RTOp;EpetraExt;Isorropia;Thyra;Galeri;Amesos;Ifpack;Belos;RBGen' -- Teuchos_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='AztecOO;ML' -- Epetra_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Triutils;EpetraExt;Isorropia;AztecOO;Galeri;Amesos;Ifpack;Belos' -- Epetra_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='Thyra;ML;RBGen' -- Zoltan_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Isorropia' ... -- EpetraExt_LIB_REQUIRED_DEP_PACKAGES='Epetra;Teuchos' -- EpetraExt_LIB_OPTIONAL_DEP_PACKAGES='Triutils' -- EpetraExt_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='Isorropia;Thyra;Galeri;Amesos;ML' -- EpetraExt_FORWARD_TEST_OPTIONAL_DEP_PACKAGES='Belos' -- Isorropia_LIB_REQUIRED_DEP_PACKAGES='Teuchos;Epetra;Zoltan' -- Isorropia_LIB_OPTIONAL_DEP_PACKAGES='EpetraExt' -- Isorropia_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='ML' -- Thyra_LIB_REQUIRED_DEP_PACKAGES='RTOp;Teuchos' -- Thyra_LIB_OPTIONAL_DEP_PACKAGES='EpetraExt;Epetra' -- Thyra_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Stratimikos' ... -- Stratimikos_LIB_REQUIRED_DEP_PACKAGES='Thyra' -- Stratimikos_LIB_OPTIONAL_DEP_PACKAGES='Amesos;AztecOO;Belos;Ifpack;ML' -- Stratimikos_TEST_OPTIONAL_DEP_PACKAGES='Triutils'

  11. Adding a Library in PACKAGE/src/CMakeLists.txt • Dependent package header directories, libraries, and library link directories handled automatically! • Macros provide uniform behavior across all libraries across all packages! • Test-only libraries are supported with ‘TESTONLY’ flag (see NOX for an example) INCLUDE(PackageLibraryMacros) # A) Package-specific configuration options PACKAGE_CONFIGURE_FILE(${PROJECT_NAME}_config.h) # B) Define the header and source files (and directories) INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}) SET(HEADERS EpetraExt_ConfigDefs.h ... ) SET(SOURCES EpetraExt_ProductOperator.cpp ... ) ... # C) Define the targets for package's library(s) PACKAGE_ADD_LIBRARY( epetraext HEADERS ${HEADERS} SOURCES ${SOURCES} )

  12. Adding a Test in PACKAGE/test/CMakeLists.txt epetraext/test/CMakeLists.txt # Compile against epetra_test_err.h in all tests? INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}) ... ADD_SUBDIRECTORY(MatrixMatrix) • Add common header include dirs • Just include the subdirectories epetraext/test/MatrixMatrix/CMakeLists.txt INCLUDE(PackageAddExecutableAndTest) INCLUDE(PackageCopyFilesToBinaryDir) PACKAGE_ADD_EXECUTABLE_AND_TEST( MatrixMatrix_test SOURCES cxx_main.cpp COMM serial mpi ) PACKAGE_COPY_FILES_TO_BINARY_DIR( EpetraExtMatrixMatrixCopyFiles DEST_FILES infileAB infileATBT infileAB2 infiles infileABT infileAB3 infileATB infileATB2 C.mtx C4x4.mtx C4x12x12x4.mtx C4x6.mtx C6x4.mtx C6x6.mtx mat6x4.mtx mat6x6.mtx mat12x4.mtx mat4x12.mtx mat4x4.mtx mat4x6.mtx Y.mtx YTC.mtx Y_transp.mtx roman roman2 romancase romancase2 cdt cdt_case cdt_d.mtx cdt_m.mtx cdt_tce.mtx SOURCE_DIR ${${PACKAGE_NAME}_SOURCE_DIR}/test/MatrixMatrix SOURCE_PREFIX "src_“ EXEDEPS MatrixMatrix_test ) • All header paths, link libraries etc are handled automatically! • Define executable and test in one shot! • 100% correct dependency tracking!

  13. Defining slightly more complex tests thyra/test/operator_solve/CMakeLists.txt INCLUDE(PackageAddExecutableAndTest) PACKAGE_ADD_EXECUTABLE( test_linear_op_with_solve SOURCES test_linear_op_with_solve.cpp COMM serial mpi ) PACKAGE_ADD_TEST( test_linear_op_with_solve NAME test_linear_op_with_solve_n1_n2 ARGS "--n=1" "--n=2" NUM_MPI_PROCS 1 COMM serial mpi ) PACKAGE_ADD_TEST( test_linear_op_with_solve NAME test_linear_op_with_solve_n4 ARGS "--n=4" NUM_MPI_PROCS 1 COMM serial mpi XHOST s858352 s903186 ) • Define test cases separately from executable if needed!

  14. Defining More Sophisticated Tests Current Limits of built-in CTest tests: • Runs a single executable (or script) with a set of arguments • Pass/fail determined by return value (0), or regex fail or regex any pass on STDOUT • Runs test in current directory What we would like: • Run multiple commands • Define pass/fail for each command in various ways (examine other output files, examine output from multiple commands, etc.) • Run tests in subdirectories created for the test Solution: • Write user-defined CMake function PACKAGE_ADD_ADVANCED_TEST(...) • Write ‘cmake –P’ scripts

  15. Defining More Sophisticated Tests Define advanced test: PACKAGE_ADD_ADVANCED_TEST( <testName> TEST_0 (EXEC <execTarget0> | CMND <cmndExec0>) ... [TEST_1 [EXEC <execTarget1> | CMND <cmndExec1>] ...] ... [TEST_N [EXEC <execTargetN> | CMND <cmndExecN>] ...] [OVERALL_WORKING_DIRECTORY <overallWorkingDir>] [FAIL_FAST] [KEYWORDS <keyword1> <keyword2> ...] [COMM [serial] [mpi]] [OVERALL_NUM_MPI_PROCS <overallNumProcs>] [HOST <host1> <host2> ...] [XHOST <host1> <host2> ...] [FINAL_PASS_REGULAR_EXPRESSION <regex> | FINAL_FAIL_REGULAR_EXPRESSION <regex>] ) Where: TEST_<i> EXEC <execTarget> [NOEXEPREFIX] [NOEXESUFFIX] [ADD_DIR_TO_NAME] | CMND <cmndExec> [ARGS <arg1> <arg2> ... <argn>] [MESSAGE "<message>"] [WORKING_DIRECTORY <workingDir>] [NUM_MPI_PROCS <numProcs>] [OUTPUT_FILE <outputFile>] [NO_ECHO_OUTPUT]] [PASS_ANY | PASS_REGULAR_EXPRESSION "<regex>" | PASS_REGULAR_EXPRESSION_ALL "<regex1>" "<regex2>" ... "<regexn>" | FAIL_REGULAR_EXPRESSION "<regex>" | STANDARD_PASS_OUTPUT ]

  16. Defining More Sophisticated Tests PACKAGE_ADD_ADVANCED_TEST( ValidateParameters_compareTestOutput COMM serial mpi OVERALL_NUM_MPI_PROCS 1 # Should be 1,4,9 TEST_0 EXEC ml_Validate ARGS input1.xml OUTPUT_FILE ValidateParameters_compareTestOutput.out TEST_1 CMND ${PYTHON_EXECUTABLE} ARGS ${PROJECT_SOURCE_DIR}/commonTools/test/utilities/compareTestOutput evaluateCriteria baseline1.txt ValidateParameters_compareTestOutput.out PASS_REGULAR_EXPRESSION "Test passed." FAIL_FAST )

  17. Defining More Sophisticated Tests XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Advanced Test: ML_ValidateParameters_compareTestOutput Running test commands: TEST_0, TEST_1 ================================================================================ TEST_0 Running: "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/BUILD/packages/ml/test/ValidateParams/ML_ml_Validate.exe" "input1.xml" Writing output to file "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/BUILD/packages/ml/test/ValidateParams/ValidateParameters_compareTestOutput.out" -------------------------------------------------------------------------------- ... -------------------------------------------------------------------------------- TEST_0: Return code = 0 TEST_0: Pass criteria = Return code TEST_0: Result = PASSED ================================================================================ TEST_1 ...

  18. Defining More Sophisticated Tests XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Advanced Test: ML_ValidateParameters_compareTestOutput Running test commands: TEST_0, TEST_1 ================================================================================ ... ================================================================================ TEST_1 Running: "/usr/bin/python2.5" "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/Trilinos/commonTools/test/utilities/compareTestOutput" "evaluateCriteria" "baseline1.txt" "ValidateParameters_compareTestOutput.out" -------------------------------------------------------------------------------- Test passed. -------------------------------------------------------------------------------- TEST_1: Return code = 0 TEST_1: Pass criteria = Match REGEX {Test passed.} TEST_1: Result = PASSED ================================================================================ OVERALL FINAL RESULT: TEST PASSED XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX http://trilinos-dev.sandia.gov/cdash/testDetails.php?test=182284&build=17561

  19. Trilinos CMake Workshop Outline Automatic Intra-Package Dependency Handling

  20. Quickstart: Special Configuration Modes for Trilinos • Configuring Trilinos to build all packages with all tests and examples: -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_TESTS:BOOL=ON • Configuring a package(s) along with all of the packages it can use -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_TESTS:BOOL=ON • Configuring Trilinos to disable a package(s) and all packages it depends on: -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Amesos:BOOL=OFF

  21. Automatic Intra-Package Dependency Handling • Trilinos/cmake/python/data/TrilinosPackageDependenciesTable.html • http://gabriel.sandia.gov/TrilinosPackageDependenciesTable.html

  22. RTOp Teuchos Epetra Triutils Thyra EpetraExt Package Dependency Structure for Thyra Required Dependence Optional Dependence

  23. Example: Enabling a Package and All Optional Packages $ ./do-configure –D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_Thyra:BOOL=ON \ -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON Configuring Trilinos build directory ... Enabling all optional packages for current set of enabled packages ... -- Setting Trilinos_ENABLE_EpetraExt=ON because Trilinos_ENABLE_Thyra=ON -- Setting Trilinos_ENABLE_Epetra=ON because Trilinos_ENABLE_Thyra=ON -- Setting Trilinos_ENABLE_Triutils=ON because Trilinos_ENABLE_EpetraExt=ON Enabling all remaining required packages for the current set of enabled packages ... -- Setting Trilinos_ENABLE_RTOp=ON because Trilinos_ENABLE_Thyra=ON -- Setting Trilinos_ENABLE_Teuchos=ON because Trilinos_ENABLE_Thyra=ON Enabling all optional intra-package enables that can be if both sets of packages are enabled ... -- Setting EpetraExt_ENABLE_Triutils=ON since Trilinos_ENABLE_EpetraExt=ON AND Trilinos_ENABLE_Triutils=ON -- Setting Thyra_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_EpetraExt=ON -- Setting Thyra_ENABLE_Epetra=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_Epetra=ON Final set of enabled packages: Teuchos RTOp Epetra Triutils EpetraExt Thyra 6

  24. RTOp Teuchos Epetra Triutils Thyra EpetraExt Lib Only Libs & Tests Dependency Handling for Pre-Checkin Testing $ ./do-configure \ -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_Epetra:BOOL=ON \ -D Trilinos_ENABLE_ALL_FORWARD_DEP_PACAKGES:BOOL=ON \ -D Trilinos_ENABLE_TESTS:BOOL=ON

  25. RTOp Teuchos Epetra Triutils Thyra EpetraExt Lib Only Libs & Tests Dependency Handling for Pre-Checkin Testing $ ./do-configure \ -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_RTOp:BOOL=ON \ -D Trilinos_ENABLE_ALL_FORWARD_DEP_PACAKGES:BOOL=ON \ -D Trilinos_ENABLE_TESTS:BOOL=ON

  26. Summary • There is a lot of functionality in the current tools CMake/CTest/CDash • CMake/CTest scripting gives great flexibility to extend functionality • Various Sandia projects duplicate effort in build, testing, and reporting tools • However, complex computational code projects require more functionality! LET’S TALK ABOUT WHAT ELSE IS NEEDED!

More Related