CVS Reorganization, Installation Reorganization & A Simple Build System

1. CVS Reorganization,Installation Reorganization & A Simple Build System Steve Fischer October 24, 2002

2. Goals • Make GUS portable • Easily build and configure runnable instances of our projects • Avoid conflicting with the pre-existing structure of external sites • Self-explanatory CVS structure • Improve collaborators’ ability to understand our code structure • And ours (particularly new folks) • Improved ability to release projects on a schedule • Projects wait for other projects only when necessary • Easily maintain released versions while development continues • Support development on a unified file system (new file server) • Don’t rely on local file systems having different installed versions • Encourage a move away from individual ownership of projects

3. CVS Reorganization

4. CVS Structure • At the root there are projects, which have components, which have parts: $CVSROOT/ Project1/ Component1/ bin/ lib/ Component2/ Project2/ Projectn/ parts

5. Projects • Units of software development having their own release schedule • Have their own release number, eg, GUS 3.1.2, Annotator 1.0.0 • May depend on other projects • Either on a specific release or ‘latest’ • No circular dependencies • Dependency example: • DoTS->GUS->CSP

6. Proposed Projects (phase 1) • AllGenes • Annotator (new) • CSP (CBIL Style Police: web form assistance) • DJob (distributed job controller) • DoTS (build, including DoTS related plugins) • GLE (grammars for transcription) • GUS • ParaDBs • PlasmoDB • RAD (plugins & website) (or should these be separate?)

7. Projects Contain Components • For example: GUS/ Common/ DBAdmin Model/ • A component is usually formed from one or a couple of package directories (eg, java packages). • In addition to code, components have: • Documentation • TODO list • Configuration • Required data • A component may depend on others within its project

8. Proposed GUS/ Components • SeqMatch/ • Common/ • DBAdmin/ • GenBank/ • GOPredictor/ • Model/ • ObjRelJ/ • ObjRelP/ • Pipeline/ • PluginMgr/ • SchemaBrowser/ • Util/ • WebDevKit/

9. Proposed GUS Components in Detail - 1 • SeqMatch/ • Blast, BLAST2, and BLAT object models • Common/ • Commonly used schema dependent stuff, such as common plugins • DBAdmin/ • Database admin scripts • GenBank/ • Data model for genbank parser • GOPredictor/ • Predict GO function • Model/ • Schema definition files • Hand edited objects (eg, GUS::Model::DoTS::Assembly.pm.man)

10. Proposed GUS Components in Detail - 2 • ObjRelJ/ • Java object layer: superclasses • ObjRelP/ • Perl object layer: superclasses and code generator • Pipeline/ • Pipeline management API • PluginMgr/ • GA and friends • SchemaBrowser/ • Web based schema browser • Util/ • Commonly used schema-independent utilities • WebDevKit/ • Servlet and/or JSP based

11. Components Contain Standard Parts • As needed: Component1/ bin/ cgi-bin/ config/ data/ doc/ htdocs/ lib/ src/ test/

12. Component Parts in Detail - 1 • Bin • executables • Cgi-bin/ • Cgi executables • Config/ • Properties files for the component • Files typically named component.prop • Doc/ • Specifications • User guides • TODO

13. Component Parts in Detail - 2 • Data/ • Extra data needed by component, eg, matrices, • Htdocs/ • Static web pages • Test/ • Test data

14. Component Parts in Detail - 3 • Lib/ • Contains linkable object files • Perl includes full package path (Project::component::module) lib/ perl/ Project/ component/ *.pm xml/ *.xml

15. Component Parts in Detail - 4 • Src/ • Contains source that needs to be compiled or otherwise transformed before it can be installed src/ c/ java/ edu/ cbil/ project/ component/ org/ gusdb/ component/

16. Process to Migrate CVS Structure CVS checkout copy origCVS/ GUS/ perl/ … www/ perl/ lib/ Fasta.pm CSP/ fromCVS/ GUS/ perl/ … www/ perl/ lib/ Fasta.pm CSP/ newCVS/ DoTS/ Common/ lib/ perl/ Common/ Fasta.pm CSP/ GUS/ Transform script orig pruned new

17. Process to Migrate CVS Structure • Do transform • Check out old to origCVS/ • Copy to fromCVS/ • Run script to transform from fromCVS/ to newCVS/ • moves files (thus pruning fromCVS/) • Validate transform by: • Examing transform script (see where YOUR stuff is going to be) • Examine fromCVS/ to see what I left behind • Create new CVSROOT with newCVS/ (losing history) • How can I get everybody’s OK on the transform??

18. Releases • A project is the unit of release • When it is released, it is tagged in cvs with a release number, eg 2.1.1 • At that time, other projects can declare a dependency on that particular release (discussed in detail later) • Bug fixes are applied to the tagged branch

19. Installation Reorganization

20. Objectives • Install into a single relocatable location: $GUS_HOME • Don’t conflict with the site’s pre-existing structure • Easy to install and uninstall • Avoid path and classpath conflicts • Be able to find (almost) all GUS installed resources there: • Executables • Libraries • Documentation • Configuration • Some data • Third party resources • Support multiple running instances on a machine, eg dev, beta • Make explicit the versions of each included resource • In a file called $GUS_HOME/versions • Also have installation targets for websites and appl. servers

21. Installing Project1 to $GUS_HOME P1/ C1 C2 $GUS_HOME/ Bin/ Lib/ Doc/ P2/ C1 C2 P3/ C1 C2 = dependencies

22. Understanding $GUS_HOME • The home for a single GUS related installation • Contains one or more projects, and the projects they depend on • Including third party resources (such as BioJava or BioPerl) • May contain projects that are running separately, as long as they don’t have conflicting dependencies • (May even contain the entire set of GUS projects) • It contains a version file, eg: DoTS 1.2.1 GUS 2.0.0 CSP 3.0.1 BioJava 1.1.1

23. Sample $GUS_HOME $GUS_HOME/ bin/ dotsbuild extractSeqs config/ dotsbuild.prop csp.prop gus.prop data/ matrices/ doc/ DoTS/Dotsbuild/TODO GUS/Model/uml/*.uml CSP/UserGuide.html (cont’d)…

24. Sample $GUS_HOME (cont’d) $GUS_HOME/ … lib/ java/ gusmodel.jar guswdk.jar perl/ DoTS/DotsBuild/*.pm GUS/Model/*.pm GUS/ObjRelP/*.pm GUS/WebDevKit/*.pm CSP/*.pm xml/ *.xml

25. Installation Targets - Website • Contains cgi-bin and htdocs $/world/www.allgenes/ cgi-bin/ *.pl htdocs/ *.html

26. Installation Targets – Application Server • EG, Tomcat • Will copy .jar files and other stuff as needed

27. Configuration • Code never refers to absolute file paths • Many resources are conveniently located relative to $GUS_HOME, so don’t need configuration • Otherwise, code relies on property files for configuration • We provide a perl and java PropertySet object to make this easy • Property files are found in… $GUS_HOME/config • Named after the component that is using them: • gusmodel.prop • Consider using properties instead of macro substitutions when possible

28. A Simple Ant-based Build System

29. Objectives • Install projects to installation targets • Serves two kinds of installs: • From $PROJECT_HOME to installation targets (developers) • From .tar file to installation targets (external users) • Support inter-project dependencies • Support dependencies on third party resources • Handle object layer code generation • Be easy to use • Be relatively easy to maintain • Be generic across projects and components, but also flexible

30. $PROJECT_HOME -> $GUS_HOME (developers) • Install the build system • Create $PROJECT_HOME • Check out the install/ project • Copy build to a local bin/ • Check out the project of interest • build DoTS install $GUS_HOME –co • Now $PROJECT_HOME is an image of CVS containing your project and all projects it depends on. • Edit away • Install for testing • build DoTS install $GUS_HOME

31. .tar -> $GUS_HOME (external users) • Untar the download file • cd install/ • setenv $GUS_HOME /usr/local/somewhere • ./build DoTS install $GUS_HOME • Also works for installing to a website • ./build GeneDB installweb /world/www.genedb

32. The Mechanics • Uses Ant (http://jakarta.apache.org/ant/) • Uses xml to declaratively specify the build • Used extensively in the java community • Each project contains its own build.xml file • The install/ directory contains the main build.xml file • Calls the appropriate project’s build.xml • Houses “subroutines” • The project build.xml file • Specifies the project’s dependencies on other projects • Specifies the project’s component’s dependencies on each other • Calls default project and component build routines, unless custom ones supplied

33. The Mechanics (cont’d) • Java compilation • The build compiles java into the classes/ directory of a component • It places .jar files into lib/java/ directory of a component • Does compilation in dependency order • Copying from $PROJECT_HOME to target (eg, $GUS_HOME) • Copies and merges all dirs, such as bin/, lib/ doc/ • Can perform macro substitution on the way • Creates version file • Because Ant is kind of slow, build can do just a component instead of a whole project

34. Special case: object layer code generation • Generates into: • Perl: • $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/DoTS • $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/Core • Etc • Java: • $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/dots • $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/core • Etc • Hand created files have .man suffix, eg: • GUS/Model/DoTS/Assembly.pm.man • Only generates if files are older than “schema definition” file: • $PROJECT_HOME/GUS/Model/schema/definition.sql • This file is generated by schema modification process (and lives in CVS)

35. Making it All Happen

36. Goals • Getting it done quickly • With a minimum of disruption

37. Overarching strategy • Work on projects and components one at a time • Start with ones that are most central and not in development • GUS/ObjRelP • GUS/Model/lib/perl/Model/DoTS • GUS/PluginMgr • GUS/Common • DoTS • DJob • Get that core working with new build system and $GUS_HOME • In parallel, convert to GUS 3.0 • In parallel, bring in GUS/ObjRelJ and Annotator (Dave) • Upgrade WebDevKit install apparatus • Bring in Web sites one at a time

38. Inconveniences • During the time that a project is being upgraded: • Everybody will need to commit their work to cvs • Will need to maintain 2 copies in cvs: • Old • New • Will need to merge changes to both • Hopefully this will be short for most projects