CMT

1. Define the development work models Structuring the packages • Project management • Split the complete software base into natural sub-projects • Based on the release cycle constraints • Typical: • Core software • Offline/online • Simulation/reconstruction/analysis/… • Reflect the multi-project organization • Eg. Atlas uses Gaudi which uses LCGCMT • Decomposition based on the responsibilities • Each individual project have • A specific release cycle • Every n-week • Random • A dedicated CMTPATH entry • An interface package to its used sub-projects • Defines the upstream CMTPATH items • Selects the release id of all used sub-projects • Development/Integration work models • The work area becomes the bottom project • i.e. it is the top left entry in $CMTPATH • Used packages override those from the selected production release • Structure the software base with levels of integration areas • A package belongs to a project • It is located in the corresponding CMTPATH entry • May be located at the top position or below any directory tree • A package has a unique name • must be unique in the complete software base • Unique amongst the different projects • A package has versions • Implicitly specified in the directory structure • Or explicitly in the version.cmt text file • A package is recognized as a CMT package by • a cmt directory below its directory structure • a requirements file in the cmt directory Structuring the projects Project2 Project3 Project1 CMTPATH=/…/Project1 : /…/Project2 : /…/Project3 • Projects are located in dedicated disk areas • Each project has an entry in the CMTPATH search list • Follows the standard platform dependent syntax • ‘:’ on Unix and ‘;’ on Windows • The search list is prioritized like usual PATH search lists • Left (bottom) entries are considered first • Right (top) entries are considered last • Packages of a project may use packages of upstream projects • Packages belong to projects • they are always installed in one of the CMTPATH entries • This is where packages are looked for • Each project may define its own strategies • In /…/Project/cmt/project.cmt • Valid for all packages in this project • Valid for all bottom packages if not overridden The project file /.../Project/cmt/project.cmt /A/cmt/requirements /cmt/version.cmt /B/v1/cmt/requirements /C/D/E/cmt/requirements /version.cmt Package A with explicit version definition Package B with implicit version definition Package E In a sub-directory project P2 project P1 project P3 package B use A package A package E use B Identify the elements of the configuration package F package D use B Structure the packages package G use E use F package C use B • Projects • Project names are defined in the project files • <project>/cmt/project.cmt • Packages • The name is the primary identifier of packages • Must be unique • Constituents • Libraries • Applications • Generated documents • Activities • Code generation • Build • One make target per constituent • Actions • Generic activity • Activities can be grouped by type (testing, documentation, installation, deployment…) • Contextual parameters values according to • Sites • Platforms (hardware, OS, …) • Step in the Life cycle • User defined conditions • Specified in symbols • Macros, sets, paths, actions • Variants with tags or tag mixtures Integration area Structure the projects development area package C use B package D use B package G use E use F Define the development work models Define recurrent patterns in configuration management Identify the elements of the configuration • Build strategy • E.g. Define whether the result of the build will go to a global installation area • Setup strategy • E.g. Define what kind of automatic environment variables will be generated by CMT • Compiling, linking • E.g. Construct DLL on different platforms • Generating code, documentation • E.g. Building rootcint generator • Complec Doxygen generation with tag-file management • Include file search style • E.g. Specifying the project-wide conventions for include search strategy • Test patterns • Driving CppUnit • Enforcing the Unit tests • Managing the integration tests • QA activities Define recurrent patterns in configuration management Identify and describe the activities of the software production Query the knowledge base to parameterize the development tools Query the knowledge base to parameterize the development tools • The cmt command • A C++ application • Available on any platform (Unixes, MacOSX, native Windows) • All configuration parameters can be reached by query commands • cmt show macros • cmt show uses • cmt show macro xxx • … • The cmt broadcast command • Uniquely sort the dependency graph (leaves first, root last) • Make it an ordered list of packages reached from the current package • Traverse the ordered list • Apply an action to every node • A normal shell command • May stop at first error or continue • May select a subset of the graph • May exclude a subset of the graph • The cmt filter command transforms any text files • expand all $(mmm) and ${mmm} patterns in the file using symbol values Identify and describe the activities of the software production • Versioning activities • Generally handled through tools like CVS or SVN • CMT provides a plugin to CVS to perform queries about CMT structures • helper command to perform conformant checkout operations: cmt co • SVN is more flexible and can be taught to recognize CMT packages • Through SVN properties • Build activities • Standard compiling and linking is automatically handled from the definitions of constituents • Specialized sequences of operations defined in CMT patterns • Testing, QA assessment • Driven by external tools like JUnit, CppUnit, Qmtest, OVAL, … CodeWizard, RuleChecker • Enforced by automatic applications through CMT patterns • Deployment • CMT query mechanisms to the knowledge base used to generate the manifest files of different packaging systems: Pacman, RPM, Fink • Specify adaptation scheme to configuration parameter values when installed on remote sites • The common options • -quiet • -tag_add=t1,t2 • -public • -private > cmt broadcast gmake > cmt broadcast –select=/Graphics/ gmake test CMT Christian Arnault arnault@lal.in2p3.fr http://www.cmtsite.org/

2. The configuration tags • Describe the contextual and operational conditions of the software configuration • The current sub-project • The platform and operating system • The site • The compiler • The step in the life cycle • The user defined conditions • Examples: • Debug or optimized mode • Selection of a graphical environment • Running some special firmware • The version tags of CMT itself • Various production modes: • Some are automatically constructed by CMT • By discovery of the context (OS, compiler version) • To reflect the project strategies • During the activities • While building a constituent • While running a user defined activity (through actions) • Some are externally specified by the user through environment variables • CMTSITE: the current site name • CMTCONFIG: the binary tag name • Tags can be specified on the cmt command line • Using the –tag_add=<tag-list> common option • Tags can be explicitly activated in requirements file • Using the apply_tag <tag> statement • Some are deduced from other tags • Using the tag association statement • Active tags are visualized using cmt show tags The dependency graph • Describe the primary relationship between packages • Three identification elements • The name • The version request • The directory offset in the project • The dependences form a Direct Acyclic Graph • Clients packages use used packages • Packages may be reached by several paths • CMT supports cycles in the graph • Cycles introduce non deterministic traversal paths • A package exports its public features to its clients • All requirements files of all (visibly) used packages are concatenated to form the complete configuration • Constituents are always private • Automatic (global) patterns are applied to all visible packages • Use statements installed in private sections define a hidden region of the dependency graph when observed from the client perspective tag i686-rh73-gcc32 Linux redhat73 gcc-3.2 opt tag i686-rh73-gcc32-opt Linux redhat73 gcc-3.2 opt tag i686-rh73-gcc32-dbg Linux redhat73 gcc-3.2 debug tag i686-rh73-gcc32-pro Linux redhat73 gcc-3.2 opt profiled tag redhat73&gcc-3.2 redhat73-gcc32 tag OSF1 Unix tag Linux Unix tag HP-UX Unix tag Darwin Unix The symbolic parameters Dependency graph • Generically define a configuration parameter • May receive different values under different conditions • According to tag expressions • Is exported to client packages (if installed in a public section) • Can be modified/overridden by client packages • Modifications installed in a private section are not propagated to clients. • Several possible semantics share the same syntax • macro generic internal CMT variable • and make’s macro definition • set environment variable definition • alias alias definition of the shell • path PATH-like environment variable • action shell command • And associated modifiers • <kind>_append • <kind>_prepend • <kind>_remove • <kind>_remove_all • <kind>_remove_regexp • <kind>_remove_all_regexp The constituents Configuration tags • Describe constituents to be constructed from sources • Applications executable • Libraries static, shared, dynamically loadable • Documents anything else • Applications and libraries obey a predefined strategy • Sources are always explicitly specified • Default source location is ../src • But can be explicitlyspecified • Absolute location • Or relatively to ../src/ • Sources are transformed by a languagecompiler • Support for new languages is described by users • The language statement • Automatic detection of languages from file suffixes • Documents construction method can be freely designed • Make fragments Constituents Symbolic parameters Actions Configuration patterns macro m1 “default value” \ Unix “on any Unix” \ rh73 “only on this machine” \ Win32&debug “for Windows and debug” macro m2 “je réutilise $(m1)” \ Linux “but on any Linux” path_remove PATH “/A/bin” path_prepend PATH “” \ CERN “/afs/cern.ch/sw/lcg/A/bin” \ LAL “/lal/A/bin” action directory “ls $(dir_options)” \ Win32 “dir $(dir_options)” Scoping sections Client package Package semantics doc_header ../cmt/fragments/ /doc_header /doc /doc_trailer doc (a.txt) doc (b.txt) package doc_provider make_fragment doc_header make_fragment doc_trailer make_fragment doc \ –header=doc_header \ –trailer=doc_trailer cmt do directory doc (c.txt) doc_trailer mydoc.make use package doc_client document doc mydoc a.txt b.txt c.txt Provider package Configuration patterns • Container and release packages: • Purely managerial packages • Not involved when using the software • Only provide a CMT requirements file with a set of explicit use statements • No wild cards • Purpose: • Specifies the set of exact version for a release of a software domain • Drive the cmt broadcast commands over the complete software domain • One top container can provide a release package<project>Release. • It describes the complete software release of the project • Every new release of the project implies a new release of the <project>Release container. Scoping sections • Describe recurrent configuration operations or policies in a project or a software domain • Enforce project policies • Automate and parameterize complex operations • Extend the CMT concepts • Pattern application can be: • Explicit (manual) using the apply_pattern keyword • Application is on a package basis • Automatic when patterns have the –global keyword • Applied to all packages that see the pattern in the dependency graph • Definition of a pattern • A sequence of templated cmt statements • Templates: • Automatic: <package> <PACKAGE> <version> • <path> <project> • User defined: <xxx> • Examples: • Defining the project-wide policy for include search path • Defining automatic shared library access path • Enforcing a systematic QA test application macro m1 “a public value” private macro m2 “a private value” use my_package end_private # back to public section • Interface packages: • Provide an interface between a software project managed by CMT and a so-called external product (i.e. not managed by CMT) • Specify the native version value defined by the product (as opposed to the CMT version tag) • Adapt the policies and conventions to the project’s ones • Eg Include directory • Construct the compiler and linker options for this product • Set the runtime options (PATH, LD_LIBRARY_PATH, etc…) • Specify the material to export when constructing a distribution kit • Examples • MySQL • ROOT • SEAL • Geant4 • Java kits • The LCGCMT project is entirely made of such interface packages • Connect to LCG internal (SEAL, POOL, PI, …) or external products pattern -global include_path include_path ${<package>_root}/<package>/ pattern ld_library_path \ path_remove LD_LIBRARY_PATH "" Unix "/<package>/" ; \ path_append LD_LIBRARY_PATH "" Unix "${<PACKAGE>ROOT}/${CMTCONFIG}" ; \ path_remove PATH "" Win32 "/<package>/" ; \ path_append PATH "" Win32 "${<PACKAGE>ROOT}/${CMTCONFIG}" package XercesC use LCG_Settings v* macro XercesC_native_version "2.3.0" macro XercesC_home "$(LCG_external)/XercesC/$(XercesC_native_version)/..." include_dirs $(XercesC_home)/include macro XercesC_linkopts "-L$(XercesC_home)/lib -lxerces-c -lpthread" \ WIN32 "/LIBPATH:$(XercesC_home)/lib xerces-c_2.lib" path_prepend LD_LIBRARY_PATH "$(XercesC_home)/lib" WIN32 "" path_prepend PATH "" WIN32 "$(XercesC_home)/bin" macro XercesC_export_paths " $(XercesC_home)/include $(XercesC_home)/lib" In ProjectPolicy pattern quality_test \ application <package>test –s=../test <package>test.cxx <other_sources> package Pa use ProjectPolicy quality_test other_sources=*.cxx In package Pa CMT Christian Arnault arnault@lal.in2p3.fr • A private section is not exported to the clients • A public specification found in the graph overrides a private region of the dependency graph • The cmt broadcast and cmt show uses reach the private sections by default • The –public common option always hides the private regions • The –private common option always reaches the private regions • Scoping sections end: • At the end of the requirements file • With the end_private or end_public statements • Ending statements return to the previous scope • Policy packages: • Provide policy statements for a software domain • include dir strategy • CMT strategies • CMT patterns • Generic symbols (macros, sets, …) • Examples: • AtlasPolicy is the main policy package, mandatory for all Atlas packages. • ExternalPolicy, AtlasCxxPolicy, LCG_Policy