O’Reilly Open Source Con October 19, 2005

1. O’Reilly Open Source Con October 19, 2005 FreeBSD Release Engineering Murray Stokely murray@FreeBSD.org

2. Outline • Introduction • FreeBSD in a Nutshell • FreeBSD Development Model • Source Code Revision Control • Project Management / “Hats” • Release Process Overview • Code Review • Release Checklist • CVS Operations • Release Building

3. Outline • Extensibility • FreeBSD 6.0 and beyond • Summer of Code - FreeSBIE, BSD Installer, launchd, etc.. • Future Directions

4. FreeBSD in a Nutshell • Freely available Unix-like operating system • Runs on x86, Alpha, Sparc64, IA-64, AMD64 architectures • Over 13,000 software packages available • Many commercial users • Thousands of developers around the world contributing to it • Used as an operating systems research platform • So far, so Linux...

5. Development Organization • Two layers of FreeBSD organization • The committers, and everyone else • Committers have write access to the source tree • Everyone else submits patches or bug reports using FreeBSD's problem reporting system (GNATS), and waits for a committer to commit the change • Individuals who submit many patches (that work) are invited to become committers • 9 committers form the elected "core team", for dispute resolution

6. Development Organization Thousands of contributors 9 core team Sourcecode 350+ committers

7. FreeBSD Development Model • Thousands of developers around the world contribute code to FreeBSD. • Only the "committers" have write access to the CVS repository • 353 total committers in FreeBSD (includes source, documentation, and ports committers). • 212 active committers to src/ in the last 12 months. • 149 active committers to src/sys (kernel) in the last 12 months. • 80% of major src/ commits in CVS made by roughly 45 senior developers. (active is loosely defined as making a single commit.)

8. FreeBSD Development Model • FreeBSD development continues along two parallel branches; FreeBSD-CURRENT and FreeBSD-STABLE. • FreeBSD-CURRENT is the main trunk of our CVS repository. All new development should happen here. • FreeBSD-STABLE is the branch from which major releases are made. Changes enter this branch at a different pace, and with the general assumption that they have been well tested by our user community running -CURRENT.

9. Source Code Control • The entire source code for FreeBSD is stored in a CVS repository. • The logs, and individual changes for each file can be traced back to 1994. • The source tree can be checked out at any state, or corresponding to any release tag. • CDs are available taking the history back a further 20 years.

10. Source Code Control Perforce • Limitations of CVS (vendor branches, poor three way merges, lack of atomic changelists, etc.) can really hamper large development projects. • As an open source project, we strongly believe we should support and use open source tools. • CVS is our official system, but next-generation development is often done in Perforce first before being merged to our -CURRENT CVS branch.

11. Source Code Distribution • FreeBSD Source Code • Available on CD (freebsdmall.com, others) • Can be downloaded from ftp.FreeBSD.org • Changes to the source code • Can be updated using CVS • Can be updated using CVSup (faster CVS) • Changes can be automatically e-mailed in, and integrated with your local source tree • Can be browsed, with history, on the web, at http://www.FreeBSD.org/cgi/cvsweb.cgi

12. FreeBSD Branch Model • Head of the tree is now FreeBSD 7.0-CURRENT. • Multiple “STABLE” branches for more mature releases. RELENG_4 branch from which 4.x releases are made, RELENG_5, RELENG_6, etc.. • Point releases are made at 4 month intervals off of STABLE release branches until the release branch becomes obsolete. • 4.11 was last release on RELENG_4. Probably one more 5.x release. • 6.0 real soon now -- we expect a long and successful future for 6.x releases.

13. March 2000, FreeBSD 4.0

14. FreeBSD-Current (became 5.0) FreeBSD-Stable March 2000, FreeBSD 4.0

15. FreeBSD-Current (became 5.0) FreeBSD-Stable 4.3, April 2001 4.2, November 2000 4.1.1, September 2000 (crypto) March 2000, FreeBSD 4.0 4.1, July 2000

16. “Hats” in FreeBSD • Originally, large percentage of FreeBSD commits made by set of “core” developers. • As the project has grown, the number of significant contributors has also grown. • The “core” team model needed to be modified with the creation of additional teams to delegate project management responsibilities.

17. “Hats” in FreeBSD • Release Engineering Team • Security Officer Team. • Port Manager Team. • Documentation Engineering Team. • Donations Liaison Team. • Marketing / Press Team. • Architecture Review Board. • CVS / Perforce Meisters, Admins, postmaster, etc..

18. FreeBSD Build System • The complete system is available in CVS : kernel, compilers, libraries, headers, userland, third-party packages, etc. • The entire system is built with a single command in /usr/src : “make world” . • Some GNU software included (e.g. gcc, gdb). These are imported on CVS vendor branches with BSD Makefiles instead of default configure based Gmake build (for parallel makes, make -j N, etc..) • Over 13,000 third party applications in FreeBSD Ports Collection.

19. Ports Tree Growth

20. Release Process Overview • Releases are made from the -STABLE development branch at approximately 4 month intervals. • Reminder / Announcement email sent to developers <45 days> • MFC Sweeps ("Merge from -CURRENT") • Code slush / freeze <30 days> • Release Candidates <15 days> • Eventually a final release • Upcoming release schedule maintained at http://www.FreeBSD.org/releases/6.0R/schedule.html

21. Release Engineering Team • A principal release engineer drives the process for each release. • Other team members have different responsibilities to help principal release engineer: • help evaluate MFC merge requests to evaluate risk / reward tradeoff. • Run builds for release candidates and releases on different architectures. • Translate release notes, work on DocBook based documentation infrastructure. • Etc.

22. Building Releases • One can build a complete release of FreeBSD, including FTP install directories, floppy images, and ISO images for CDROMs with one command. • “make release” is used by many large companies to produce special versions of FreeBSD with special patches or additional software installed by default. • It is also the well documented way in which the release engineering team makes all official releases of FreeBSD. • Schedules, checklists, process documentation available at http://www.FreeBSD.org/releng.

23. Release Building • Tools necessary for building a FreeBSD release are available in src/release/ • These tools aim to provide a consistent way to build FreeBSD releases. • A complete release can be built with only a single command, including the creation of ISO images suitable for burning onto CDROM, installation floppies, and an FTP install directory. This command is aptly named "make release"

24. Release Building • To successfully build a release, you must first populate /usr/obj by running "make world" or simply "make buildworld". The release target requires that several variables be set properly : • CHROOTDIR - The directory to be used as the chroot environment for the entire captive release build. • BUILDNAME - The name of the release to be built. • CVSROOT - The location of a CVS repository. • RELEASETAG - The tag corresponding to the sources to build.

25. "make release" (1/4) There are many other variables that can be used to customize the release build process. Most of these are documented at the top of src/release/Makefile. The exact command used to build FreeBSD 4.6 (x86) was : make release CHROOTDIR=/local3/release \ BUILDNAME=4.6-RELEASE CVSROOT=/host/cvs/usr/home/ncvs RELEASETAG=RELENG_4_6_0_RELEASE

26. "make release" (2/4) • The release makefile can be broken down into several distinct steps. • Creation of a sanitized system environment in a separate directory hierarchy with "make installworld". • Checkout from CVS of a clean version of the system source, documentation, and ports into the release build hierarchy. • Population of /etc and /dev in the chrooted environment. • chroot into the release build environment, to make it harder for the outside world to taint the build.

27. "make release" (3/4) • Steps of "make release" (continued) • "make world" in the chrooted environment. • Build Kerberos-related binaries. • Build 'GENERIC' kernel. • Creation of a staging directory tree where the binary distributions will be built and packaged. • Build and installation of the documentation toolchain needed to convert the documentation source (SGML) into the HTML and text documents that will accompany the release.

28. "make release" (4/4) • Steps of "make release" (continued) • Build and install of the actual documentation (user manuals, release notes, tutorials, hardware compatibility lists, etc..) • Build of the "crunched" binaries to be used on the installation floppies • Creation of the distribution tarballs of the binaries and sources. • Create the boot media and a fixit floppy. • Create the FTP installation hierarchy. • (optionally) Create ISO images for CDROM media.

29. Ports and Packages • The FreeBSD Ports Collection is a collection of over 13,000 third-party software packages available for FreeBSD. • The ports team (portmgr@FreeBSD.org) is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany a given FreeBSD release.

30. Ports and Packages • In order to provide a consistent set of third-party packages, every port is built in a separate chroot environment, starting with an empty /usr/local and /usr/X11R6. • The requisite dependencies are installed as packages before the build proceeds. • By starting the package build in a pristine environment, we can assure that the package metadata (such as required dependencies) are accurate, and so we will never generate packages that might work on some systems and not on others depending on what software was previously installed.

31. Build Clusters • Several large companies have donated machines, rack space, power, and network bandwidth for build clusters for all supported architectures to allow release engineers and port masters to build releases and packages quickly. • Current ports cluster can build 13,000+ third party packages in just over 24 hours for i386 (somewhat longer for most other architectures).

32. Extensibility • A release can be customized in many different ways. • If you have access to the staging directory of an existing "make release" build hierarchy then you may find it easiest to apply patches or additional files as necessary to the chroot build directory. • rm ${CHROOTDIR}/usr/obj/usr/src/release/release.[48] • Rebuild sysinstall, the kernel, or whatever parts of the system your change affected. • chroot ${CHROOTDIR} ./mk release.4 release.8

33. Summer of Code • Google’s Summer of Code Program sponsored 19 students to spend the summer working on FreeBSD (nearly 400 total with other open source projects) • Many interesting projects in security, power management, networking, file systems, and release engineering. • Specifically: • Dario Freni - FreeSBIE (FreeBSD LiveCD) • Andrew Turner - BSD Installer (new installation tool) • R. Tyler Balance - launchd (porting from MacOS X 10.4 to speed up Unix boot process.) • All student code in Perforce (perforce.FreeBSD.org), some being merged into main CVS repository.

34. Future Work • More process documentation • More automated performance testing • Better Installation Tools / Graphical User Interface • Better integration with FreeSBIE livecd

35. Additional Information • Detailed documentation on "make release" is available in release(7). • A schedule of upcoming releases is maintained at http://www.FreeBSD.org/releng • The release engineering paper is now maintained in CVS and the most recent version is available from http://www.FreeBSD.org/docs.html • If all else fails, just email re@FreeBSD.org and we will be happy to answer your questions. • Excellent Case Study on FreeBSD Development in September IEEE Transactions on Software Engineering. • http://wikitest.FreeBSD.org

36. Questions?