Handling of Unix Application Software

Handling of Unix Application Software Stephan Wiesand DESY -DV - May 25, 2004

Motivation • in early 2003, we looked for a replacement for our Unix software deployment scheme: • install software in /afs/<cell>/products • create symlinks in /afs/cell/@sys/usr/local • all systems have these symlinks • /products -> /afs/<cell>/products • /usr/local -> /afs/<cell>/@sys/usr/local • the replacement was designed, implemented and deployed since, as time allowed (no dedicated resources), and is presented here

The Problem • provide additional software on all client systems • sw not included with the distribution, or • needed in different or multiple versions, or • built with different options (kerberized...) • targets: • Linux, Solaris, & any other potentially relevant platform • desktops, farm nodes, (workgroup) servers, controls, notebooks • many have disks too small to install all software locally • some must function properly without network or other infrastructure

Additional Requirements • automatic (un)installation, up/downgrade • configurable per client (tests...) • keep systems in a well defined, working state • reproducible without backup • verifyable • decent dependency handling (shared libraries allowed) • within add-on software and against the host system • for every client, even if some software isn't local • lightweight solution, don't invent a new framework • use standard tools, unmodified

Solution • use a standard package format • and a package database separate from the host system's • split packages into • a main package that can be replaced by a single symlink • into a central “reference installation” • an optional “default” package (always installed locally) • populating directories in PATH, MANPATH etc. with symlinks • populate the reference installation from same packages that would be installed locally • on the client, maintain the package database as though all packages were installed locally

Packager Choice: RPM (all platforms) • available and working on all potential targets • has all required features • versioned dependencies/conflicts, virtual packages • automatic for shared library dependencies • --justdb switch for updating the database only • structured package building from specs is a plus • build procedure is fully documented automatically • powerful macro facility (one spec for all platforms) • writing specs is becoming a common skill • excellent documentation for beginners available

Closing the Dependency Chain • using a separate RPM DB • is possible and consistent across platforms • doesn't interfere with any mechanism maintaining the client • but leaves a dependency gap between system and add-ons • solution: glue packages, providing • virtual packages (ABI version of shared libraries, ...) • ghost files (/bin/sh, ...) • safeguards in pre[uninstall] script enforce --justdb • verifying a glue package will check that the system installation keeps its promises

Glue Packages • to build a glue package, • define macros for the missing dependencies • for virtual/real packages: add Provides: <macro> • for files, add %ghost <macro> in files section • provide a verify script • checking existence of files (test -f <macro>) • or using /bin/rpm -q --whatprovides <macro> • can have one or more (glue-devel, glue-legacy, ...) • possible enhancement: mirror package for system DB • requiring everything the glue package provides & vice versa

Filesystem Layout & Packaging • deliberately chose not to use /usr/local /opt/ products/ bin/ perl->/opt/products/perl/5.8.2/bin/perl pine->/opt/products/pine/4.58/bin/pine perl/ 5.8.2/ bin/ perl* pine/ 4.58/ bin/ pine* perl-5.8.2-2.i586.rpm pine-4.58-1.i586.rpm perl-default-5.8.2-2.i586.rpm pine-default-4.58-1.i586.rpm

Packages: Rules • base packages • contain /opt/products/<name>/<version> • + content • nothing else • can be replaced by a single symlink • to <reference install>/<name>/<version> • default packages • contain links to files from base packages • anywhere in /opt/products • except in any directory owned by a base package • must require their base package

Refinements • what if we want /opt/products/bin/perl5.8.0 ? • -> alias packages (like default, but multiple versions ok) • what if we want to add files in a base directory ?(additional perl modules, configuration files) • -> daughter packages • allowed to write anywhere below parent's directory • must require their parent • names prefixed by <parent_name>_<parent_version>- • what if we want to maintain certain files by other means? • (again, configuration files) • -> local packages (like default, but links point to local fs)

Shared Library Dependencies • we do populate /opt/products/lib (and include) • but use is discouraged • and it's not searched by the dynamic loader by default • instead, set runtime search path for shared libs at build time • (link) ... -Wl,-rpath,/opt/products/<name>/<version>/lib ... • sometimes hard to do (improper use of libtool...) • means we have to rebuild dependent packages more often • also means rolling out new versions doesn't break anything • needs a modified find-requires not processing shared libs in /opt/products (spec macro __find_requires)

Deployment • manual/scripted installation is possible: • mkdir -p /opt/products/RPMDB • rpm -p --verify --verbose .../glue-2.0-2.i586.rpm • rpm --dbpath ... -i --justdb .../glue-2.0-2.i586.rpm • rpm --dbpath ... -i .../pine-4.58-1.i586.rpm • mkdir -p /opt/products/perl • ln -s <reference>/perl/5.8.2 /opt/products/perl • rpm -i --justdb .../perl-5.8.2-2.i586.rpm • rpm -i .../perl-default-5.8.2.i586.rpm • rpm -i .../pine-default-4.58-1.i586.rpm • possible and straightforward, but tedious...

Tools • instead, implemented ppm • perl wrapper calling rpm • finds packages in a repository on central filesystem • locatedb for locating package files • RPM DB (created with -i --justdb) for dependencies, content • accepts input like this: [install] justdb glue 2.0 2 local pine 4.58 1 default link perl 5.8.2 2 default,alias

ppm features • accepts one or more flat file names as input • processed sequentially • conflicting/redundant input ok • last one takes precedence • -> overrides, delegation (also has an include mechanism) • there's a [remove] tag as well • set of input files defines absolute final state • not an incremental change • final state is verified & sanitized - if possible • resolve dependencies automatically • correct children of base packages installed as link ...

Status • so far, we mostly built packages for software that needed a rebuild or were unavailable before • and a few essential ones • and their dependencies • in full production on DESY Linux 5 • most important packages available, few missing • in use on DL4 and Solaris 8 hosts to varying degrees • DL4 now obsolete, Solaris actually started this year • on both sites (independent of host management solutions) • also in use on Linux Notebooks

Experience • improved preservation and transfer of knowledge about building an application or library properly • no diligence (installation notes...) required • package builders have to go through a learning curve • about one day for building a first package not yet built • starting from a similar example • once used to it, little overhead to other procedures • at least any producing transparent, reproducible results • rebuilds and small changes much faster and more reliable • we're probably past break even

Handling of Unix Application Software