Software repository replication using the ASIS Local Copy Manager

1. Software repository replication using the ASIS Local Copy Manager IT/DIS/OSE, CERN ASIS Team Presented by: German Cancio (German.Cancio@cern.ch)

2. Presentation Plan • Introduction to ASIS • ASIS structure • Transactional model • Repository replication using ASISlcm • Project status

3. Software Distribution: Goals and Requirements • Centralized management of SW applications necessary for: • standard and homogeneous user environment • better support (less diversity!) • Equally accessible by all supported platforms • same availability • same configuration • User nodes to see ‘ready to use SW’ trough a distributed file system

4. Software Distribution at CERN: ASIS • ASIS: stands for Application Software Installation Service, offering: • A multiplatform and multiversion SW repository with ready-to-use applications • A toolsuite for • assisted application compilation and configuration • accessing, manipulating and replicating the repository • Contents: > 700 packages for ~8 platforms • GNU software (compilers, system tools, editors, desktops) • LaTeX • X11 and contributions • CERN program library • Others, like Perl, Tcl/Tk, Java, licensed software

5. Platforms currently supported ASIS (AFS) name Platform i386_redhat51 RedHat Linux 5.1 sun4x_56 Sun Solaris 2.6 hp_ux102 HP_UX 10.20 ACE3 alpha_dux40 Digital UNIX 4.0b rs_aix41IBM AIX 4.1 .5 rs_aix43 IBM AIX 4.3 .2 sgi_64 SGI IRIX 6.5

6. Software Distribution at CERN: ASIS (II) • Main tools: • ASIShappi: concurrent software compilation and configuration on multiple platforms • ASISwsm: client workstation access to the repository • ASIStm: checks and validates repository modification requests • ASISlcm: repository update and replication • ASIStar: generate tarfiles from ASIS packages

7. Logical organization • Packages are classified into Families, Products and Versions • A family groups together products with similar origin or functionality (i.e. MAIL, KDE, GNU.EDIT,CERN.LIB) • A product may contain one or several versions (emacs-19.34, emacs-20.2) • Each version is usually available for all supported • platforms • ‘share’ platform: contains common files (documentation, fonts, scripts)

8. Logical organization (II) • A model has been defined for the software processing from the sources release until delivery to users. • Based on a state-transition diagram, 2 phases: generation and deployment • A deployed package may be on one of the following states: • In Production: default version, formally supported • Certified: proved to work in the proper environment • Under Certification: for evaluation by beta users • ASIS stores InProduction packages in a common area for improved efficiency

9. ASIS ASISwsm ASISwsm ASISwsm ASISwsm ASISwsm Client access to the repository ASIS repository • ASISwsm manages the links from the repository to the workstation (in /usr/local or /cern) • The workstation administrator can customize the access to the packages (i.e. default version, specific, ignore) • Packages can be stored as ‘local copies’

10. Transaction Model In order to group together related repository modifications, a transactional system is used • A transaction is a sequence of operations... • … either • a modification of the state of a package • a ‘patch’ operation (modifying a package) • The atomicity of the execution of the transaction is guaranteed.

11. Transaction: example • Description: update emacs & related packages • RemoveFromProduction GNU.EDIT/emacs-19.34 (sun4x_55,alpha_osf32,hp_ux102) • RemoveFromProduction GNU.EDIT/vm-5.96beta (share) • IntroduceInProduction GNU.EDIT/emacs-20.2 (sun4x_55,alpha_osf32,hp_ux102,alpha_dux40) • IntroduceInProduction GNU.EDIT/vm-6.32 (share)

12. Transaction System Based on the cooperation between ASIStm (ASIS transaction manager) and ASISlcm (ASIS local copy manager) • A transaction is submitted to ASIStm, which checks its validity and coherence. • The transaction is queued if accepted. • Repository modifications are carried out by ASISlcm

13. ASISlcm: Repository Replication ASISlcm ... • Reads transactions which are new since last run • Determines what has to be done according to local configuration • Performs locally all selected transactions. • Only new packages are copied • ASISlcm can be run asynchronously, (i.e. at nighttime)

14. ASISlcm (II) • Fault tolerance: ASISlcm rolls back the currently executed transaction in case of failure (i.e. when network problems arise) • Transaction atomicity is always guaranteed. • Star scheme: modifications are propagated to the replicas (master/slave) • ASISlcm runs locally on the replicated site and only accesses the master when necessary • Replication costs proportional to the size of changes

15. Replication: example ASIStm ASISlcm ASIS ASIS CERN AFS master repository Replica 1 on site XXX WAN file system (AFS) LAN file system (NFS) LAN file system (NFS) ASISlcm ASIS ASISlcm CERN NFS replica ASIS Replica 2 on site XXX

16. Full repository replication ASISlcm also offers a ‘full’ replication mode • State of all packages is checked against the master and updated accordingly • When to use one mode or the other? • Transactional (or incremental) replication: • frequent repository updates, • i.e local replicas for load balancing • Full replication: • initial repository cloning • independent replicas (occasional updates) • recovery after crashes • the full replication does not offer atomic properties

17. ASISlcm: configuration ASISlcm can be configured to match your site’s needs: • Filtering by architecture • list of platforms to be replicated • Filtering by family / product: • take all product versions • skip a family / product / version • ‘freeze’ a family / product / version • based on the product’s state (Certified, In Production, Under Certification)

18. Architectures: sun4c_411 sun4x_55 hp700_ux90 hp_ux102 rs_aix41 alpha_osf32 sgi_52 i386_linux2 Products: <All>/<All> InProduction # Lightweight copy GNU.LANG/<All> None GNU.LANG/cvs InProduction GNU.LANG/rcs All GNU.MISC/<All> Frozen GNU.MISC/gzip InProduction MISC/<All> None MISC/a2ps InProduction MISC/tcsh 6.06 # Specific version nr MISC/arc Frozen ASISlcm: configuration example

19. How to set up an ASIS local copy? 1st. Get the ASIS software • by mirroring from the CERN AFS tree or • by downloading it from our ftp server 2nd. Configure and run ASISlcm (full) on your server(s) 3rd. Configure and run ASISwsm on your workstations 4th. Decide frequency and update type (full, incremental) Please contact us for assistance!

20. ASIS project status • ASIS runs at CERN as part of the standard setup: • > 2000 nodes, > 10000 Users • ~ 700 packages on ~ 20 GB • ASISlcm: • in production for the CERN NFS server for 18 months • on other sites: ??? • EPFL (Swiss Federal Institute of Technology, Lausanne): acts as co-developer and owns a private repository

21. Further documentation and contacts ASIS has an URL on the web: http://wwwinfo.cern.ch/dis/asis with links to: • Guides (not up to date…) • Product Documentation • Site Usage Statistics • Download area Contacts: ASIS team at CERN: asis.support@cern.ch