The common Software Repository for SL and ST (The SCaM Project)

1. The common Software Repository for SL and ST(The SCaM Project) A. Bragg (SL/CO), E. Hatziangeli (SL/CO), J. Patino (ST/MO)

2. Plan • The project • The solution • Razor • The elements of SCM • The Functionality • The system • The development environment • The Work • Documentation and Training • The Benefits

3. Why the SCaM project? Provide the means for users to develop and exchange software, using a common repository, based on available industrial products • Replace and extend the functionality of SL/CO software management system (SLAPS) • Give control to the users over their software development process • Parallel development • Quick fixes by Operators without the need of administrator • Allow version and release management for in-house software, as well as external software integration • Improve the maintenance of the Operational Software • Unify all diverse versioning systems efforts (SLAPS, SL/BI cvs, ST/MO sccs, PS/CO rcs and sccs, ...) • Create a software management system that is easy to extend and maintain

4. Primary Team • SL/CO Andrew Bragg & Eugenia Hatziangeli [pr. leader] • ST/MO Jose Patino Advisors • SL/BI Jannes de Vries • ST/MO Roberto Bartolome • LHC/VAC Isabelle Laugier • PSSL Convergence Project Alessandro Risso Observers • PS/CO Alessandro Risso • IT/IPT Arash Khodabandeh (LHC experiments: SPIDER SCM sub-project) Project Sponsors • SL/CO/AP Pierre Charrue • ST/MO Pierre Ninin The people involved

5. Major Project Milestones • Identify problem and create project SCaM [SL-Note 97-44 (CO), Jun. 97] • Collect User Requirements [SL-Note 97-59 (CO), Oct. 97] • Market survey for industrial and public domain tools => Evaluation report and recommendations for tool and procedures to SL-CO and ST-MC management [SL-Note 98-022(CO) Mar 98] • Re-commit SL-CO & ST-MC management to solution chosen [SL-Note 98-34 (CO), Jun. 98] • Purchase and installation of the chosen tool (Nov. 98) • Software management system implementation (end Aug. 99)

6. The solution package 1) Tool => Razor 2) Software Management Procedures => User and administrator 3) Administration and User support for the: Software Management tool and the Repository Software Management Procedures New User Training The team will ensure that the Software Management activities for Operational Software development are carried out as planned.

7. Why Razor? • It demonstrates a good all around functionality • good management of software versions, • supports baseline and release management • provides problem reporting management • Very adaptable, with GUI and command line interface • It is simple to learn, to use and to maintain • Good level of technical support and constant product evolution • It is commercial and well priced • It is based on floating or site license scheme • 14 floating licenses available (28 concurrent developers)

8. Architecture • Network License Manager Repository Server Win95/NT Licenses License Manager LAN Connection Win95/NT WAN Connection UNIX UNIX UNIX UNIX Repository Server Win95/NT Archive Engine Repository Files Razor User Interface Win95/NT • Platforms • HPUX • Linux • Windows NT/95 • AIX (IBM) • SCO • Solaris 2.4 (SunOS 5.4) or higher • Solaris x86 • SunOS 4.1.x • IRIX (SGI) • OSF (DEC) More on Razor ...

9. Control and track “what changes” during development Version Management Software Repository Release Management Problem Management Software Configuration Management ?? It is about “what changes” in our development environment, while we are working and programming

10. Archives changes and coordinates efforts amongst developers Versions Manages all files related to a released product Repository Issues Threads Version Management Records and coordinates things yet to be accomplished Problem Management Release Management Trilogy of Razor

11. File: hello.c Version 1.0 File: hello.c File: hello.c Version 1.2 Version 1.1 #include <stdio.h> main(void) { printf(“hello world\n”); } #include <stdio.h> main(void) { printf(“Hello World\n”); } #include <stdio.h> main(void) { printf(“Hello New World\n”); } Version 1.0 Version 1.4 Version 1.5 Version 1.3 Version 1.2 Version 1.1 Version 1.2.1.2 Version 1.2.1.1 Version Management “….what the hell, let’s call him Ramses, too….” Ramses the 1st, inventor of the versioning, at the birth of his son, 1784 B.C.

12. get_thin 1.2 get_thin 1.3 Release Management calories.c 1.3 1.4 1.5 1.6 1.7 food.h 1.2 1.3 1.4 1.5 meal.data 2.4 2.5 2.6 2.7 2.8 • Grouping (threading) software as a collection of specific versions of files is done for: • release or/and build management • try out certain combination of files for testing or prototyping • The exact state of each file, as they relate to a specific named release, can be reproduced • Each release is given a unique name and evolves over time. • The meaning and the utility of a release is up to the user

13. File: hello.c Version 1.0 File: hello.c File: hello.c Version 1.1 Version 1.2 File Versions #include <stdio.h> main(void) { printf(“hello world\n”); } #include <stdio.h> main(void) { printf(“Hello World\n”); } #include <stdio.h> main(void) { printf(“Hello New World\n”); } Release Versions Release: get_thin Version: 1.2 calories.c 1.4 Release: get_thin Version: 1.3 food.h 1.2 meal.data 2.4 calories.c 1.6 food.h 1.5 meal.data 2.7 Problem Management Change Requests Bug Reports • Software changes, arising from Bug Fixes or Change Request are managed through the software lifecycle • Track the changes by relating the versions of the files/releases with the Problem Reports or the Change Requests

14. All versions of files Software Releases All previous development investment in RCS or SCCS can be injected in the new repository Software Change Requests, Problem Reports We are independent from the tool RCS SCCS Repository “…. you know, I preferred the way you had the pyramid yesterday….” Ramses the 5th, inventor of the vault, shortly before his entombment, 1637 B.C. • Located on REPSRV • HPUX 10.20 • 8GB of space • on regular backup schedule • 365 days/year available • on power fail safe

15. ASCII Files RCS : The last version is kept along with backward deltas for the older versions SCCS: The first version is kept along with forward deltas for the newer versions Binary Files Stored in compressed format • Source files • Header files • Data files • Scripts • Documentation files • Help files • Icons • Bitmaps • Other graphics • Configuration files • Spreadsheets • Off the self software Reproducible code (binaries, output data files,…) Repository Contents All versions of files

16. User Several repositories with software related by a common theme, set up using standard defaults Repository Login Interface ALARMS LHC …... ST SL ST-MO projects Alarm Software LHC/VAC Software SL-CO projects SL-OP projects SL-BI projects SPS2001 PS-SL-conv …. . . . The set up of the repositories Minimize user inconvenience User should not need to traverse an enormous tree to have access to his software Possibility to customize according to the local development philosophy

17. groups SL-CO SL-BI SL-OP SPS2001 PSSL-conv folders The set up of each repository SL A software group is a “narrower” collection, contained within a repository The folders are the simplest collection. They contain files and are a way of grouping related files

18. The plan… • The philosophy • Access Control • PEOPLE files • The structure of a project in the repository • File name/extension, Symbolic-Link Control • Accessing the right Repository and starting the GUI’s

19. The philosophy “Do unto others as you would have them do unto you” Someone a long time ago • Underlying philosophy: • Who do I want to have write access to my software? (individual files and projects) • Myself and my project team • Others in my group in case of my absence (holidays, sickness etc etc…) • There is a complete history of repository activity linking actions to users

20. Access Control • Applies to both (Unix) Command Line and GUI PC Windows95/NT Client • Transparent to user - No additional passwords • Possible to configure, per Repository per Project • Any action which can affect a file or the attributes associated with a file are under access control • Two levels of validation • 1. PEOPLE file • 2. User’s CERN Division/Group

21. PEOPLE file? • Reflects the project team / collaborations • A PEOPLE file is a list of valid UNIX user id’s • Separated by ‘white space’ • Allow write access to the structure below • Used to obtain e-mail information for group validation • Maintained by the project team

22. Scenario .3. Scenario .2. Scenario .1. measlib bct inc lib PEOPLE SL-CO SPS2001 ST-MO WS AP cryo tds inc lib src PEOPLE inc lib src PEOPLE PEOPLE src Repository The file PEOPLE contains the group of people that they have write access under that structure. The lowest level PEOPLE file has precedence, over the upper one, for the directory structure below. Any connection to an existing or proposed Division/Project is entirely co-incidental. The names have been changed to protect the innocent.

23. File/Extension, Symbolic-Link Control • Only valid file extensions may be introduced, e.g. *.c, *.java, *.doc • Certain file’s without extensions may be introduced, e.g. PEOPLE, Makefile, readme • Acceptable extensions are defined per software group Why? - Avoid unwanted files e.g. core • New file(s) / extension(s) are entered by the developers • Symbolic links are NOT allowed!

24. Logging into the repository Starting the Razor Suit Accessing the right Repository and starting the GUI’s • From a Unix environment - GUI’s

25. Accessing the right Repository and starting the GUI’s(Continued…) • From a PC windows95/NT client environment

26. Problem Reporting(Issues) 26

27. Plan • What is the ISSUES program • Operations on a Problem Report • Problem Report lifecycle • Attributes of a Problem Report

28. It is the Problem Tracking System part of the Razor trilogy What is the ISSUES program? • Main list: Displays each problem report or change request made within a software domain • The main display:

29. GUI • Email templates • Web • Create new problems reports Operations on a Problem Report • View/modify problems reports • Delete problem reports • Print problem reports • Relate problems reports to an activity in either the file version control or release management aspect of Razor • Make filtered selections on all the archived reports • Generate reports

30. Submitted Accepted Rejected Assigned Feedback Closed Suspended Problem Report lifecycle • The possible STATES and TRANSITIONS of an issue are: (e-mail notifications on state changes)

31. Attributes of a Problem Report (I) • Common Attributes

32. Attributes of a Problem Report (II) • Category • Assigned person • State

33. Attributes of a Problem Report (III) • Two text panes • Problem Description (Originator) • Actions Taken (Assigned Person)

34. Operational Software Area (Used to operate the machines) Products:tz_drive, logbook, shiftlog,… Complete releases with sources Install software release Repository Install public software Public Software Area (For compilation and linking of user software) Libraries: SL_EQUIP, SL_RPC, SL_MEAS,SL_UMMI,… Includes: nc.h, Mequip.h, sl_measlib.h, binaries: Configuration:.Xdefaults, etc. Installation procedure Installation procedure Check in Create/build/test a release Check out Operational Software Installation User Development Area

35. Public SW Area lib include bin Current Next Previous ……….. Newly released public software Current (default) public software Previous versions of public software Build Support Environment • Support for csh environment • Support the existing SLAPS environment • $SL_INCLUDE, $SL_LIB, $SL_BIN,... • $SL_RPC, $SL_EQUIP,… • Support for sh like environment • General Makefile (new projects)

36. Operational software will be released in and run from a designated distribution area Each release will be complete with sources to allow quick bug fixes The Operator will be able to quickly revert to previous working version of a product (global for all consoles) The Operator will be able to access any available version (previous or future) of a product (local to a console) The Operational Software

37. Defined per software group according to group requirements Roles and Capabilities Developers • introduce new files in his software project • check out their software • check in their software • create/modify releases of their software projects • extract releases of software products from repository • build releases of their software projects • install software releases in the Operational Software Area • receive/create/modify Problem Reports and Change Requests Librarians • install public software (libraries, includes,…) in Public Software Area Administrator • manage the tool, the repositories, the software management procedures and new user training

38. Policies and Recommendations “Write your code for somebody who does not understand it, because by the time you come back to it, that somebody else will be you….” Some of the Policies and Procedures • PEOPLE file (Access control to a product in the repository) • Control over the type of files in the repository (File Extension control) • Public software installation via an Automatic Installation Procedure • Operational software installation via an Automatic Installation Procedure Some of Recommendations • Files self-identification (RCS keywords) • Relation of file/release versions to PRs or SCRs • Naming scheme of an Operational release

39. Who does the work? • Project team • Tool installation • Set up of the software repositories • Set up of user development environment • Set up of Operation Software distribution area • Implementation of automatic installation procedures • User training and documentation • Project team + Developers • Porting of software project (SLAPS, SL/BI, ST/MO, LHC/VAC, ALARM, …) inside the repository • Administrator(s) • Administration of the tool and management of software repository • User support • User training

40. Documentation • User Documentation • Administration Documentation • SCaM project web page: http://venice.cern.ch/~slaps/hot/SCAM_II/index.html • Razor documentation available: http://venice.cern.ch/~slaps/hot/razorDocs/rz_web/razortoc.htm • The official Razor internet site: http://www.tower.com

41. Training • We will provide training • Users • Librarians • Small groups of users - Targeting their specific needs • Lasting no more the 1½ to 2 hours maximum • There will be several training sessions • Future training sessions will be arranged as required

42. The situation today Operational Software • unable to identify the versions of the majority of the operational software • unable to identify/control all elements that influence the operational software • unable to control changes of the operational software and identify the initiators • not easy to run another version of the operational software Development Environment • public software change often and without warning • difficult to debug, since reproduction of the exact version of the software concerned is often not possible • non unique software and hardware configuration of the development platforms External Software • software written by external contractors cannot be properly synchronised with software written in CERN, because rules to integrate software are not well specified

43. Free some resources What will it change? • All software from different groups will reside in a single repository • Good working practices and procedures • Version management (at least) for existing project in maintenance phase • Software improvements • knowledge of where each software component is and what state is in • stability, because changes affecting operational software will be under control • useful reports available about the evolution and changes of the software • Centralization of repository system management

44. Benefits for the Operation team • Delivery of consistent operational software • Being able to trace and identify any component of an operational system • Being able to incorporate quick bug fixes in the operational environment • Minimisation of uncontrolled changes of the operational software • Able to run other (future or past) versions of the operational software • Able to “revert to previous working” version of the operational software

45. Version management Software is kept safely in the repository “Care free” parallel development Manage the process used to develop your software Project lifecycle management Transparency of software and documentation Important for subcontracted software development Benefits for developers Projects in maintenance New Projects All Projects A single software configuration management environment Platform independent repository access (HPUX, Win95/NT, AIX, Linux,..) Guidelines and procedures concerning the integration and installation of operational software and the introduction of software modifications

46. Iterative process Help and feedback from the software developers Why this presentation? • Make you aware of this project • Receive feedback on our proposed solution • Encourage you to try the new system We would like to produce a Software Management system that works

47. Thank you for coming Any questions?