Japan UNIX Society September 18, 2002

1. Japan UNIX Society September 18, 2002 FreeBSD Release Engineering Murray Stokely FreeBSD Mall, Inc. murray@FreeBSD.org

2. Outline • Introduction • Terminology • FreeBSD Development Model • Release Process Overview • Code Review • Release Checklist • CVS Operations • Release Building

3. Outline • Extensibility • FreeBSD 4.6 / 4.6.2 • Future Directions

4. SCM Terminology • SCM - Software Configuration Management • Branch -A variant codeline that is separate from the main line of development. • Merge - The process of integrating tested changes across branches. • Tag - An identifier for the code in an SCM system at a given point in time. • CVS - A widely used open-source SCM system.

5. FreeBSD Development Model • FreeBSD development continues along two parallel branches; FreeBSD-CURRENT and FreeBSD-STABLE. • FreeBSD-CURRENT is the main trunk of our CVS repository. All new development should happen here. • FreeBSD-STABLE is the branch from which major releases are made. Changes enter this branch at a different pace, and with the general assumption that they have been well tested by our user community running -CURRENT.

6. FreeBSD Development Model Recent FreeBSD Branch Points

7. FreeBSD Development Model • Thousands of developers around the world contribute code to FreeBSD. • Only the "committers" have write access to the CVS repository • 324 total committers in FreeBSD (includes source, documentation, and ports committers). • 212 active committers to src/ in the last 12 months. • 160 active committers to src/sys in the last 12 months. (active is loosely defined as making a single commit.)

8. FreeBSD Development Model • Users and external developers should submit bug fixes, enhancements, or problem reports with GNATS. • Online submission forms, send-pr command installed by default on FreeBSD systems, etc..

9. Open PRs vs Time

10. Release Process Overview Releases are made from the -STABLE development branch at approximately 4 month intervals. • Reminder / Announcement email sent to developers <45 days> • MFC Sweeps ("Merge from -CURRENT") • Code slush / freeze <30 days> • Release Candidates <15 days> • Eventually a final release • Upcoming release schedule maintained at http://www.FreeBSD.org/releases/4.7R/schedule.html

11. Release Checklist • Creating the Release Branch • Branch point tag $ cvs rtag -rRELENG_4 RELENG_4_6_BP src • Creating the branch $ cvs rtag -b -rRELENG_4_6 _BP RELENG_4_6 src

12. Bumping up the Version Number • Before the final release can be tagged, built, and released, the following files need to be modified to reflect the correct version of FreeBSD. • doc/${LANG}/books/handbook/mirrors/chapter.sgml • doc/share/sgml/freebsd.dsl • src/Makefile.inc • src/UPDATING • src/gnu/usr.bin/groff/tmac/mdoc.local • src/release/Makefile

13. Bumping up the Version Number (continued) • src/release/doc/${LANG}/share/sgml/release.dsl • src/release/doc/share/examples/Makefile.relnotesng • src/release/doc/share/sgml/release.ent • src/share/examples/cvsup/standard-supfile • src/share/misc/bsd-family-tree • src/sys/conf/newvers.sh • src/sys/sys/param.h

14. Bumping up the Version Number (continued) • www/${LANG}/releases/* • www/${LANG}/docs.sgml • www/${LANG}/cgi/ports.cgi

15. Creating the Final Release Tag • When the final release is ready, the following command will create the RELENG_4_6_0_RELEASE tag. $ cvs rtag -rRELENG_4_6 RELENG_4_6_0_RELEASE • The Documentation and Ports managers are responsible for tagging the respective trees with the RELEASE_4_6_0 tag. • "Sliding" tags after the fact : $ cvs tag -d tagname filename

16. Release Building • Tools necessary for building a FreeBSD release are available in src/release/ • These tools aim to provide a consistent way to build FreeBSD releases. • A complete release can be built with only a single command, including the creation of ISO images suitable for burning onto CDROM, installation floppies, and an FTP install directory. This command is aptly named "make release"

17. Release Building • To successfully build a release, you must first populate /usr/obj by running "make world" or simply "make buildworld". The release target requires that several variables be set properly : • CHROOTDIR - The directory to be used as the chroot environment for the entire release build. • BUILDNAME - The name of the release to be built. • CVSROOT - The location of a CVS repository. • RELEASETAG - The tag corresponding to the sources to build.

18. "make release" (1/4) There are many other variables that can be used to customize the release build process. Most of these are documented at the top of src/release/Makefile. The exact command used to build FreeBSD 4.6 (x86) was : make release CHROOTDIR=/local3/release \ BUILDNAME=4.6-RELEASE CVSROOT=/host/cvs/usr/home/ncvs RELEASETAG=RELENG_4_6_0_RELEASE

19. "make release" (2/4) • The release makefile can be broken down into several distinct steps. • Creation of a sanitized system environment in a separate directory hierarchy with "make installworld". • Checkout from CVS of a clean version of the system source, documentation, and ports into the release build hierarchy. • Population of /etc and /dev in the chrooted environment. • chroot into the release build environment, to make it harder for the outside world to taint the build.

20. "make release" (3/4) • Steps of "make release" (continued) • "make world" in the chrooted environment. • Build Kerberos-related binaries. • Build 'GENERIC' kernel. • Creation of a staging directory tree where the binary distributions will be built and packaged. • Build and installation of the documentation toolchain needed to convert the documentation source (SGML) into the HTML and text documents that will accompany the release.

21. "make release" (4/4) Steps of "make release" (continued) • Build and install of the actual documentation (user manuals, release notes, tutorials, hardware compatibility lists, etc..) • Build of the "crunched" binaries to be used on the installation floppies • Creation of the distribution tarballs of the binaries and sources. • Create the boot media and a fixit floppy. • Create the FTP installation hierarchy. • (optionally) Create ISO images for CDROM media.

22. Ports and Packages • The FreeBSD Ports Collection is a collection of nearly 7,500 third-party software packages available for FreeBSD. • The ports team (portmgr@FreeBSD.org) is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany a given FreeBSD release.

23. Ports and Packages • In order to provide a consistent set of third-party packages, every port is built in a separate chroot environment, starting with an empty /usr/local and /usr/X11R6. • The requisite dependencies are installed as packages before the build proceeds. • By starting the package build in a pristine environment, we can assure that the package metadata (such as required dependencies) are accurate, and so we will never generate packages that might work on some systems and not on others depending on what software was previously installed.

24. Ports Build Cluster • The "Ports Cluster" for the x86 architecture currently consists of a master node (Dual Pentium III 733 Mhz) and 8 slave nodes (Pentium III 800Mhz) to do the actual package builds. With this configuration, a complete package build takes over 24 hours. • The Ports Cluster for the Alpha architecture consists of 7 PWS 500A machines donated by Compaq. Faster Alpha machines have recently been donated by Compaq/HP and will soon be available for the ports cluster. • The clusters for both architectures are co-located with the other FreeBSD Project equipment on Yahoo's campus in Santa Clara, California.

25. Ports and Packages • For FreeBSD 4.4, over 4.1 gigabytes of packages were created. This causes a problem for CDROM distributions because we would obviously like to ship as many packages as possible without making the user insert another disc to satisfy dependencies. • The solution is to put "clusters" of like packages with similar dependencies onto specific discs. • The package split is performed by Steve Price and the ports team in coordination with the general wishes of the general user community with respect to which packages get to appear on the first CD.

26. FreeBSD CDROMs • Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the BSDi/WRS "official" CDROM distributions. • Each of the four discs must contain a README.TXT, CDROM.INF, and filename.txt file. • The file manifest can be created with : find . -type f | sed -e 's/\^.\///' | sort > filename.txt

27. Disc #1 • Output from "make release" with the following additions : • Addition of a 'tools' directory to aid would-be new users working in other operating systems. • XFree86 • As many additional software packages as will fit.

28. Disc #2 • Also largely created by "make release". • Contains a "live filesystem" that can be used from sysinstall to troubleshoot a FreeBSD installation. • Contains a compressed copy of the CVS repository in the CVSROOT directory • Contains commercial software demos (such as Perforce, Accelerated-X, etc..) in the commerce directory. • Discs #3 and 4 simply contain as many additional packages as will fit.

29. Extensibility • A release can be customized in many different ways. • If you have access to the staging directory of an existing "make release" build hierarchy then you may find it easiest to apply patches or additional files as necessary to the chroot build directory. rm ${CHROOTDIR}/usr/obj/usr/src/release/release.[48] • Rebuild sysinstall, the kernel, or whatever parts of the system your change affected. chroot ${CHROOTDIR} ./mk release.4 release.8

30. FreeBSD 4.6 • Code slush began on May 1, 2002. After which time all commits to the RELENG_4 branch (-STABLE) had to be approved by re@FreeBSD.org • The first release candidate for the x86 architecture was released on May 17. • 2 more release candidates were released leading up to the final release on June 15. • Over 1,100 emails were sent to re@FreeBSD.org in little over a month.

31. FreeBSD 4.6.2 • Unfortunately, there were a number of security problems discovered in OpenSSH and OpenSSL immediately after the release of FreeBSD 4.6. • Many people also reported problems with the ATA subsystem. • The release engineering and security officers decided that there were enough problems to warrant a rare point release of FreeBSD. • Previously, approach used with 4.1.1 didn't work. A more conservative approach was needed. • 4.6.1 - The "Phantom Release"

32. Future Work • More documentation • Regression Testing • Better Installation Tools • Graphical User Interface • Message Catalogs for Internationalization.

33. Additional Information • Detailed documentation on "make release" is available in release(7). (Also translated to Japanese). • A schedule of upcoming releases is maintained at http://www.FreeBSD.org/releng • The release engineering paper is now maintained in CVS and the most recent version is available from http://www.FreeBSD.org/docs.html • If all else fails, just email re@FreeBSD.org and we will be happy to answer your questions.