1 / 27

CHEP 2000, 10.02.2000

Roberto Barbera. “Farming” with VENUS. CHEP 2000, 10.02.2000. Meeting INFN-Grid, 18.10.2000. Roberto Barbera. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy. ALICE Collaboration. Goals of VENUS. Simplify administration of heterogeneous Unix/NT clusters

shellie-farmer
Download Presentation

CHEP 2000, 10.02.2000

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Roberto Barbera “Farming” with VENUS CHEP 2000, 10.02.2000 Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  2. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Goals of VENUS • Simplify administration of heterogeneous Unix/NT clusters • Increase efficiency of administration • Increase productivity of users • Providing a single system image • Simplify software integration and installation • Central point of administration • Integration of Unix with WindowsNT • Considering special needs in complex engineering and scientific computing environments Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  3. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Design of today‘s clusters HP Pre-, postprocessing CAD/CAE Simulations Batch, Jobflowmanagement Backup Archiv PDM Plotprocessing Databases Communication Firewall SGI WEB IBM Email SUN Compaq DEC Printer Backup Servers Printer Compute Servers File Servers Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  4. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Today‘s problems in system management I Complexity of structure: grouping of hosts, users with different working environments Complexity of data and filesystemstructures: Many projects, databases, centralized and decentralized filesystems Maintaining consistency in login behaviour: user management Fast changes in hard and software Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  5. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Today‘s problems in system management II • Complexity of software: • many software packages used: graphical (CAD, Pre/postprocessing) • sometimes big packages (>500 MB) • many software releases per year, patch management • licensing using different mechanisms (flexlm, netls,...) • integration into the desktop (CDE, KDE, ...) • integration into the filesystem • integration into the printer environment Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  6. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Today‘s needs in system management • Methods to install software and patches to be fast and reliable • User management to be simple, providing best user environment for users • Well defined filesystem structure to provide a global file name space • Well defined operating system status (which OS, which patches) to assure quality and reliability • Independence of special knowlegde of system administrators • Quality driven system management instead of „quick“ but undocumented solutions • Flexibility to solve daily problems methodically Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  7. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Single system image (SSI) I • Definition: For users, an ideal cluster presents itself as a single system image, i.e. a heterogeneous workstation server cluster appears like a single big computer with many (intelligent) terminals attached. • Single point of authentification • Same password on any node in the cluster • Unique access rights on each node • Unique file space : Global file name space Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  8. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Single system image (SSI) II • Common login environment on each node • common (architecture dependent) variables (e.g. $path) • shell aliases • desktop configuration (CDE, KDE, GNOME,...) • Considering needs of users and goal to simplify the login environment • Access to all printers, peripherals available for the user or user group Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  9. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Overview of VENUS functionalities • User- and file system • management • Uniform view of the • file system • /home/$user • add • sc del usr • ch group • list • /$map/$key • add • sc del key • ch • list • Configuration • Management • central storage of all • configuration • differences • to original OS • As global as possible • as specific as • necessary • common • architecture • groups • hosts • VENUS methods • to configure • subsystems • Software • Management • Software depot server • Full documented • software packages • in VENUS format • Pre/post/install/ • remove methods • and procedures • VENUS tools to • create VENUS • software packages • scsoftpkg, scsoftfile • scsoftinfo • sc{add|ch|del|list}pkg Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  10. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration User management I • scadduser, scchuser, scdeluser • modify passwd map and automounter maps or corresponding information bases on NT • Create, move, or delete user homes on target nodes scadduser -u calvin -h school school: /v/school/home/calvin scchuser -u calvin +h playground moves calvin from school to playground scdeluser -u calvin removes user calvin Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  11. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration User management II VENUS Master Unix Local file system Unix Local file system Unix: passwd map Automounter map Unix NFS Unix PDC,samba SMB File Server NT/DFS Server WindowsNT: PDC configuration DFS configuration NT NT NT Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  12. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Configuration management II • Methods properties • Methods are tasks to be executed on a client to configure a client subsystem • Methods are (usually) shellscripts • subsystems are: tcpip, nis, nfs, mail, resolver, xwin,cde, printer, etc. • Methods are „dataless“, i.e. they are automatically configured in their specific context • Methods are able to communicate with the configuration depot on the VENUS master • Methods are able to install VENUS software packages • Methods use the VENUS lookup hierarchy: host > groups > architecture> common • Methods have an „install“ and an „uninstall“ section • Methods can be modified, extended and administrators can develop their own methods Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  13. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS architecture VENUS master VENUS client Accesscontrol VENUS commands rshd scrsh scinstall inetd scvenus venusd Depots Methods Context Software Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  14. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS software management I • Goals: • Installing Software by a simple command including configuration, licensing, pre and postinstallation • Automatic maintenance of a software inventory • Parallel installation of software • Installing the correct software version (architecture, i.e. OS) • Ability to create software packages from any 3rd party vendor • Easy removal of software from a client Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  15. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS software management II • VENUS Software packages consist of: • All files to be installed • pre/post installation procedures and/or VENUS methods • filelist, version, architecture, author • Application integration • Integration using wrapper scripts: • setting path and environment • creating symbolic links (in installation method !) • wrappers are located in the standard path • (e.g. /usr/local/bin) Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  16. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS software management III HP SGI VENUS master IBM Depots SUN Compaq Methods Context Software DEC HP SGI Parallelized Software Installation ! Software Depot IBM SUN Compaq DEC Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  17. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS software management IV • Commands for software management • scaddpkg lsf lsfhosts • This command will install LSF on each node contained in the • hostgroup „lsfhosts“. Each client asks the VENUS master to provide • the proper software package • for the correct operation system type • and the latest version • The software package will be transfered from the „nearest“ software depot to each client. This makes it possible to parallelize the software • installation process. If the installation is finished successfully, VENUS • makes an entry into the software inventory. • The installation of the configuration files is done automatically in the installation method. Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  18. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS software management V Commands for software management scdelpkg lsf lsfhosts This command will remove LSF on each node contained in the hostgroup „lsfhosts“. If done successfully, the entry in the software inventory is also removed. sclistpkg hostname This command creates a list of software installed on „hostname“ and displays the current version of the software. Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  19. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references I DaimlerChrysler: 1600 CAD Workstations in car design (SGI, HP, IBM,SUN) Main appl.: CATIA 300 CAE Workstations in engineering (SGI, IBM, HP, SUN, CRAY, NEC) 100 different applications 80 CAE Workstations in truck engineering (SGI, IBM, HP) 200 Workstations in DaimlerChrysler Research Center Stuttgart (SGI, Linux, IBM, HP) 100 Workstations in bus design and engineering (EVOBUS) (SGI,HP,IBM) Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  20. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references II DaimlerChrysler: Details in CAE • 5 departments with different tasks: Structure • mechanics, crash, engine development, • various applications (> 100) • High end Unix workstations and > 30 compute • servers (HP, SGI Origin, NEC SX4, IBM SP) • Using VENUS specials: • Managing all printers with original VENUS printer method • Installing complex Pre/postprocessors using VENUS methods • e.g. Medina from debis, abaqus, etc. • Configuring batch (LSF, NQS), Jobstarter and batchsystem using • VENUS depots Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  21. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references III BMW: 1600 CAD Workstations in car design (SGI, HP, IBM,SUN) Main appl.: CATIA 400 CAE Workstations in engineering (SGI, IBM, HP, SUN, CRAY) Compute Center with CRAY SV1, T90, Origin IBM SP, SUN HPC 100 different applications 50 Workstations at BMW Technik Gmbh (Designers of Z1) Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  22. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references IV 300 CAD/CAE Workstations in engineering and design (SGI, HP, IBM,SUN) 30 Workstations at BOSCH Research Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  23. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references V 330 CAD Workstations in ECAD design (SUN) planned: 1200 Workstation (Autumn) Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  24. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references VI Details • ECAD at Infineon in the memory products division: • >250 workstations using cadence, mentor software • Big fileserver, used as file and codeserver • All nodes are completely managed using VENUS methods: • standard VENUS NIS, NFS, but using SUNs automounter • TCPIP, DNS setup • printer method modified: using lpngr • Installing backup client software (legato networker) using VENUS • Installing and configuring LSF using VENUS • security method, SUN patch method, method to create CAD-Admin-users Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  25. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references VII Details cont´d • Benefits: • Reducing costs by increasing the efficiency for administrators: • > 50 Workstations managed by one administrator • Parallel execution of tasks (methods) • documented and well defined status of any clients • Quality driven administration in contrast to manually changing configurations • Increasing efficiency of users by high stability and uptime of clients • Avoiding conflicts between different administrators by VENUS • multi-administrator ability: Locking of methods and central files • e.g. NIS sources, passwd. • Error prevention ! Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  26. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration VENUS references VIII Details cont´d Benefits: ROI (Return on investment) Break even Costs to introduce VENUS and licensing Service costs before using VENUS Service costs using VENUS 0 3 months Time Meeting INFN-Grid, 18.10.2000 Roberto Barbera

  27. Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy ALICE Collaboration Costs Meeting INFN-Grid, 18.10.2000 Roberto Barbera

More Related