OpenSUSE Build Service

1. openSUSE.org Build Service ??? Novell ??????

2. Current Challenges The open source communities provide lots of source code, but building & installation is often hard for unexperienced users. Current challenges: maintain sources for different target platforms maintain patches during upstream updates integrate external contributions merge code back to external source repositories no version updates for released distributions Deal with differences on various distributions

3. Goals of the Build Service Open the openSUSE distribution development itself Make it simple to provide binary packages of software Maintain sources in one place, offer packages for everybody Support the �Release early, Release often� approach Involve and connect the open source communities Make it easy and secure to install new software

4. What is the Build Service ? Server Infrastructure The Build Service server hosts all sources provides the build systems to create packages provides download and mirror infrastructure for packages is the communication framework A set of tools: The tools are used for local operations on the workstation or for remote operations on the openSUSE server.

5. The open design of the Build Service Everyone will be able to use the Build Service. The Build Service tools are open source. The Build Service provides a public API. The Build Service can get integrated into existing tools. The Build Service is not limited to SUSE based distributions Integration with existing web pages is possible.

6. Trust Model The Build Service does guarantee that the binary package got build from the sources, but it can't judge about the sources itself. Everyone can submit source, this causes a potential security problem. Installating software from external sources requires trust in these. The decision to trust a package or not is up to the end-user. A trust rating system helps the end-user to judge about software. The trust level of project depends on the trust level from its contributors.

7. Build Service Architecture

8. Components Overview

9. Backend Building Packages Storage for sources (version controlled) Farm of build hosts for building packages Run build in specified environment Build for multiple hardware architectures (currently i586, x86_64) Storage for built packages Provide build status and logs

10. Frontend Programming Interface to the Build Service Public API for client tools Access to sources Access to build status and logs Access to built packages Control build User management

11. Frontend Utilities Additional Frontend Services and Utilities SPEC file generation News feeds about new packages Project rating Interfacing with upstream repositories (CVS, SVN, etc.) Interfacing with other developer portals and software index sites (Forge, Ticker, private project page, etc.) Automatic version upgrades ... (most of this isn't implemented yet)

12. openSUSE API Public Programming Interface for the openSUSE Build Service REST-based API XML over HTTP Frontend functionality exposed through HTTP operations Control and meta data is transferred and stored as XML Schema validation for all XML data

13. openSUSE API Documentation http://api.opensuse.org/apidocs

14. Client Tools User Interface for Developers and Packagers Web Client Easy browsing and project administration Editing and uploading of sources Downloading of built packages Command Line Client Editing and uploading of sources Start local build for debugging

15. Web Client http://build.opensuse.org

16. Command Line Client user@host> osc

17. Mirror Interface Interface to mirrors Interface to end users through installation tools Syncing with mirrors (rsync, drpmsync) Installation source meta data Redirecting users to mirrors Support end users with finding and installing packages from openSUSE

18. Project Modell

19. Project Model 1/2 A project is a workspace which can be created by any user. It may contain: a list of users with write access to it sources or a description how to download them link to existing sources to be built in a different enviroment changes for existing packages a list of build targets to build binary packages for The result will be one or more package repositories.

20. Project Model 2/2

21. Setting up the build environment The build service parses BuildRequires / Build-Depends from spec file / dsc file. these packages get added to a �base system� packages get automatically added so that all of the run-time dependencies are met

22. Breaking dependencies To get rid of excess packages one can break the unwanted dependencies. Dependencies can be broken on the project level (affects every package of the project) or on the package level: project level: by adding �Ignore:� lines to the project configuration Ignore: tetex:xorg-x11-libs package level: by adding �#!BuildIgnore� lines to the specfile #!BuildIgnore: xorg-x11-libs

23. Dealing with ambiguities Ambiguities can happen if two packages provide the same functionality. The system treats ambiguities as errors: Specfile: BuildRequires: apache2 expansion errors: have choice for apache2-MPM needed by apache2: apache2-prefork apache2-worker

24. Dealing with ambiguities To solve ambiguities, either select one of the choices or deselect all unwanted ones: project level: �Prefer� lines Prefer: apache2-prefork Prefer: -apache2-worker package level: �BuildRequires� / �#!BuildIgnores� BuildRequires: apache2-prefork #!BuildIgnore: apache2-worker

25. Automatic dependency rewriting Problem: packages get renamed or are named different for different distributions. Example: package containing shared libraries for canna SUSE: canna-libs Fedora: Canna-libs Mandriva: libcanna1 Debian: libcanna1g Project can specify per repository dependency rewrite rules: Substitute: <package> <replacement packages>

26. Project specific build data A project consists of: a number of packages and repositories macros for the project information for setting up the build environment The build process concatenates the configuration of every involved project. the repository search path defines which projects to use

27. Adding a service SUSE: postinstall: %{fillup_and_insserv -f <srv>} preuninstall: %stop_on_removal <srv> postuninstall: %restart_on_update <srv> %insserv_cleanup Mandriva: postinstall: %_post_service <srv> preuninstall: %_preun_service <srv>

28. Adding a service (cont.) Fedora: postinstall: /sbin/chkconfig --add <srv> preuninstall: if [ �$1� = 0 ] ; then service <srv> stop >/dev/null 2>&1 /sbin/chkconfig --del <srv> fi postuninstall: if [ �$1� -ge 1 ] ; then service <srv> condrestart >/dev/null 2>&1 fi

29. Adding a service (cont.) Proposed macros: postinstall: %service_add <srv> preuninstall: %service_del_preun <srv> postuninstall: %service_del_postun <srv>

30. Adding specials to spec files Used statements: %if 0%{?suse_version} < 1010 do something %endif %if 0%{?mandriva_version} Name: libopensync %else Name: opensync %endif  %endif

31. openSUSE Factory buildHow it is

32. Status Quo � How it is What is the problem here ? * The workflow does work only from SUSE internal to openSUSE, but not vice versa. * The distribution itself is not buildable within the build service, so it is not reproducable for everyone * This means that all contributions have to be routed via SUSE internal systemWhat is the problem here ? * The workflow does work only from SUSE internal to openSUSE, but not vice versa. * The distribution itself is not buildable within the build service, so it is not reproducable for everyone * This means that all contributions have to be routed via SUSE internal system

33. Status Quo � How it is Some open source projects dislike distribution patches (samba). Some open source projects build on being patched by distributions (kernel) SUSE/Novell contributors must fullfill the SLE requirements. openSUSE.org relies on contributions of SLE base. Some open source projects dislike distribution patches (samba). Some open source projects build on being patched by distributions (kernel) SUSE/Novell contributors must fullfill the SLE requirements. openSUSE.org relies on contributions of SLE base.

34. Status Quo � How it isExample Quality Requirements

35. Contribution ProcessesHow it could be So knowing how it is and having requirements for the process we sat down and looked for a contribution process we want to use for the openSUSE distribution. There are already some ways of handling contributions to distributions in place and used today. We looked at the most prominent ones and analyzed if they would work for us.So knowing how it is and having requirements for the process we sat down and looked for a contribution process we want to use for the openSUSE distribution. There are already some ways of handling contributions to distributions in place and used today. We looked at the most prominent ones and analyzed if they would work for us.

36. How it could be Traditionally the way to grant contributions is to allow selected people, that have somehow proven to be good contributers, the right to do so. You create a contributers �caste�. This puts all the responsibility for getting the right to contribute on the person that wants to do it. The distribution has nothing to do but look out for active and reliable contributers. Then you make those sign a contract or some other kind of understanding that is about the quality and reliability of their contributions. This is a nice way to do it if you want to keep the numbers of contributers low. It also assures that the initial skill of the contributer is very high and that hes commited to deliver contributions. On the other hand this way of handling the contribution process is not very attractive to most contributers. The entry hurdle to become one is very high. This represents a problem because often the drive for doing contibutions is the act itself. Contributions have to be visible in a timely fashion so they spark the interest of the contributer. The best example for this is the wiki concept. So this way of doing it does not match up with our goal of really broaden the group of contributers to the openSUSE distribution. Also it does not really reflect what we understand as an open development process. Traditionally the way to grant contributions is to allow selected people, that have somehow proven to be good contributers, the right to do so. You create a contributers �caste�. This puts all the responsibility for getting the right to contribute on the person that wants to do it. The distribution has nothing to do but look out for active and reliable contributers. Then you make those sign a contract or some other kind of understanding that is about the quality and reliability of their contributions. This is a nice way to do it if you want to keep the numbers of contributers low. It also assures that the initial skill of the contributer is very high and that hes commited to deliver contributions. On the other hand this way of handling the contribution process is not very attractive to most contributers. The entry hurdle to become one is very high. This represents a problem because often the drive for doing contibutions is the act itself. Contributions have to be visible in a timely fashion so they spark the interest of the contributer. The best example for this is the wiki concept. So this way of doing it does not match up with our goal of really broaden the group of contributers to the openSUSE distribution. Also it does not really reflect what we understand as an open development process.

37. How it could be To open this up a little bit you can introduce an extra �layer�. Frankly spoken you split the distribution in an �important� and an �unimportant� part. For the unimportant (non-core) part you make the entry hurdle, to contribute something, lower. For the important (core) part of the contribution you leave the entry hurdle where it would be in the previous process layout. You make it way more attractive to contribute and at the same time guard the important part of your project and keep the base stable. This way of doing it is nice for attracting first time contributers because most of the traverse trough some evolutionary process. They begin with small, easy and maybe even simple contributions. By doing that they learn what contributing something to a project means. What requirements there are. What it means to commit to maintaining something. In essence how the whole, not very simple, process of contributing works. On the other hand you keep the entry hurdle for people that already know all this high. They still have all the burden of proof on their shoulder that they can contribute to the important part. So this also does not match our requirements because it is not very attractive to the most valuable contributers. Its somehow better but still not good enough.To open this up a little bit you can introduce an extra �layer�. Frankly spoken you split the distribution in an �important� and an �unimportant� part. For the unimportant (non-core) part you make the entry hurdle, to contribute something, lower. For the important (core) part of the contribution you leave the entry hurdle where it would be in the previous process layout. You make it way more attractive to contribute and at the same time guard the important part of your project and keep the base stable. This way of doing it is nice for attracting first time contributers because most of the traverse trough some evolutionary process. They begin with small, easy and maybe even simple contributions. By doing that they learn what contributing something to a project means. What requirements there are. What it means to commit to maintaining something. In essence how the whole, not very simple, process of contributing works. On the other hand you keep the entry hurdle for people that already know all this high. They still have all the burden of proof on their shoulder that they can contribute to the important part. So this also does not match our requirements because it is not very attractive to the most valuable contributers. Its somehow better but still not good enough.

38. Contribution to the DistributionWhich packages should be part of the distribution? So given that the contribution processes that are already there today do not fit our idea of an open development process we started to work out another way of doing it. After a lot of talk we came to the conclusion that we want to have a process that is package based instead of people based. So the entity you grant the right to contribute to the project is a package and not some person. Of course there are still people behind each package but they are only one of many factors. So given that the contribution processes that are already there today do not fit our idea of an open development process we started to work out another way of doing it. After a lot of talk we came to the conclusion that we want to have a process that is package based instead of people based. So the entity you grant the right to contribute to the project is a package and not some person. Of course there are still people behind each package but they are only one of many factors.

39. Package based Contribution ProcessRequirements Make contributions as easy as possible Involve all types of contributers. From Beginners to the most powerful ones Only the best packages become part of openSUSE openSUSE must be innovative and at the same time stable openSUSE must be secure and maintained Remembering the requirements that we already gathered in the first part of the talk and thinking about other requirements we have we came down with the following list.Remembering the requirements that we already gathered in the first part of the talk and thinking about other requirements we have we came down with the following list.

40. Package based Contribution Process So to reach these goals we propose to take each package and attach a set of attributes to it. So to reach these goals we propose to take each package and attach a set of attributes to it.

41. Package based Contribution Process Attributes can be of various types. Like numbers or true/false values. Up until now we came up with attributes that say something about Attributes can be of various types. Like numbers or true/false values. Up until now we came up with attributes that say something about

42. Package based Contribution Process

43. Package based Contribution Process Info Provider: The Package owner The distribution owner A user rating/trust system A package popularity system Bugzilla BuildService Different openSUSE Teams (Security, Usability etc.) Of course these attributes need to get their values from somewhere. We can have all kinds of info providers that set these. Of course these attributes need to get their values from somewhere. We can have all kinds of info providers that set these.

44. Package based Contribution Process With the values of these attributes we can then create functions that deliver a value that expresses if the package suffices our requirements for something that is part of openSUSE. For instance you can test if a package is relevant for the security of the system. Or you can calculate a quality value of the package. Or even test if the package is enhancing the distribution.With the values of these attributes we can then create functions that deliver a value that expresses if the package suffices our requirements for something that is part of openSUSE. For instance you can test if a package is relevant for the security of the system. Or you can calculate a quality value of the package. Or even test if the package is enhancing the distribution.

45. Contribution to PackagesWorking on Sources/Packages We want to have a access system that is project/packages based. So the entity you grant access to the distribution is a package. This way we can really open up the development of the openSUSE distribution.We want to have a access system that is project/packages based. So the entity you grant access to the distribution is a package. This way we can really open up the development of the openSUSE distribution.

46. Future Source Handling Process I ADRIAN Source checkout Source edit Source commit Server rejects commit offers diff creation & package setup in home requests merge ---------- Project owner sees wanted changes Project owner merges back or refuses with reasonADRIAN Source checkout Source edit Source commit Server rejects commit offers diff creation & package setup in home requests merge ---------- Project owner sees wanted changes Project owner merges back or refuses with reason

47. Future Source Handling Process II ADRIAN Source checkout Source edit Source commit Server rejects commit offers diff creation & package setup in home requests merge ---------- Project owner sees wanted changes Project owner merges back or refuses with reasonADRIAN Source checkout Source edit Source commit Server rejects commit offers diff creation & package setup in home requests merge ---------- Project owner sees wanted changes Project owner merges back or refuses with reason

48. Future Source Handling Process III ADRIAN Source checkout Source edit Source commit Server rejects commit offers diff creation & package setup in home requests merge ---------- Project owner sees wanted changes Project owner merges back or refuses with reasonADRIAN Source checkout Source edit Source commit Server rejects commit offers diff creation & package setup in home requests merge ---------- Project owner sees wanted changes Project owner merges back or refuses with reason

49. A Possible Example Setup Which effectively boils down to this. So sometime ago the openSUSE core team sat down and talked about all these possibilities and we came to the conclusion that none of these fits our idea of an open development process. Which effectively boils down to this. So sometime ago the openSUSE core team sat down and talked about all these possibilities and we came to the conclusion that none of these fits our idea of an open development process.

50. Another Possible Example Setup Which effectively boils down to this. So sometime ago the openSUSE core team sat down and talked about all these possibilities and we came to the conclusion that none of these fits our idea of an open development process. Which effectively boils down to this. So sometime ago the openSUSE core team sat down and talked about all these possibilities and we came to the conclusion that none of these fits our idea of an open development process.

51. Missing Parts We want to have a access system that is project/packages based. So the entity you grant access to the distribution is a package. This way we can really open up the development of the openSUSE distribution.We want to have a access system that is project/packages based. So the entity you grant access to the distribution is a package. This way we can really open up the development of the openSUSE distribution.

52. Missing Components Efficient patch handling Merge handling Defined package requirements for Factory -> en.opensuse.org/Contribute User Trust Relationship System Relation of bug statistics to packages Overall UI concept

53. Clustering Visions

54. Building additional packages To build packages for supported distributions

55. Building modified distributions Different compile flags Different hardware architecture Replace base components (other compiler for example)

56. Next steps / Roadmap

57. Future Ideas Future ideas are collected at http://www.opensuse.org/Build_Service/Future_Ideas Template based package creating Integration of distribution creation system LSB conform builds Translation framework QA and automated test case framework

58. Roadmap A detailed and updated roadmap is available at http://www.opensuse.org/Build_Service Preview Phase (Q1 2006) build for addon packages does work Alpha Phase (starting in Q2 2006) Hardware is in place Import of SUSE Linux core packages does start Beta Phase (starting in Q1 2007) write access for everybody core features are complete

59. Resources http://www.opensuse.org/Build_Service Overview about the project and links for further informations and the source code. http://build.opensuse.org A running instance of the Build Service. opensuse-buildservice@opensuse.de The mailing list for discussing the Build Service.

60. Join the Lizard Blizzard!

61. Contact http://www.openSUSE.org/ openSUSE@openSUSE.org irc://irc.freenode.net/openSUSE

63. Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.