Introduction

As of August 2009, we're moving our automation system from the prototype homebrew creation to Buildbot. (We also tried Atlassian Bamboo, but found it too java-focussed.)

Buildbot, written in Python, is tremendously flexible as is being used in complex environments such as Chromium; it has plenty of flexibility for OOI. Some of the things it can do:

• Run unit tests, either Python UnitTest or Trial/Twisted.
• Run the pylint and/or pyflakes static code analyzers
• Compile, package and upload the results to the buildmaster
• Run any system-callable script (as of 6/23/10, we're using to run the java-based Maven)
• Run periodically or when code is committed (post-update hook)

OOI buildbot configuration and notes

Buildbot runs a single master and one or more buildslaves. Each buildslave is typically a different operating system or hardware, so that you can run your testing and packaging on all supported platforms automatically.

Here's a sketch:

Right now, we have build slaves for:

All buildslaves are run from within virtualenv - see this page I had to move the python 2.5 buildslave from OSX to EC2 when we added pydap. Numpy refused to build on 2.5+osx, so it was easier to move to EC2. The buildslave name is therefore outdated. More buildslaves are easy to add and might be worthwhile, depending on what platforms we need to support.

Details of setup

• On amoeba, the buildmaster and buildslave are both run from the 'buildbot-runner' user account that we setup for this purpose. To modify the configuration, you have to sudo or su to buildbot-runner.
• The webserver for amoeba is in /var/www and if you want to auto-generate doxygen docs the target directory tree is /var/www/doxygen
• The entire configuration for buildbot is in Git as 'buildbot'

Modifying the OOI buildbot

To make this easier, the entire buildbot setup is in git as its own project.

git@amoeba.ucsd.edu:buildbot.git

The project is non-gitweb and non-public due to the static passwords in master.cfg and buildslaves. Kind of a pain, that.

The project has one directory for the master and another for slaves, if you want to add a new buildslave please follow the same pattern.

Once you've edited master.cfg, you can do a sanity check with

buildbot checkconfig master.cfg

before you go live. To start or restart the buildbot:

cd $HOME/buildbot/master
buildbot restart .

You can also use that same restart command for the slaves.

Master.cfg

You'll quickly figure out that the buildslaves basically don't have any configuration; it's all in master.cfg. Understanding it requires some basic concepts: Schedulers, builders, factories and the glue around them.

• Change sources notify buildbot of a commit. There's a PBChangeSource that is used for Git. Right now, we're not using this due to issues below. We just use a periodic instead.
• The post-receive hook in git just fires the generalized buildbot trigger, caught by the PBChangeSource:

  python /usr/share/buildbot/contrib/git_buildbot.py $1 $2 $3

  (that's the entire contents of /home/git/repositories/attribute_store.git/hooks/post-receive)

• The PBChangeSource fires the Scheduler, which decides which builds to run.
• The Scheduler is keyed on branch name, which since we have multiple repositories with branch 'master' is a problem. You can fix this by writing lots of Buildbot source code; instead I chose to use a Periodic scheduler instead, which runs all the build factories every ten minutes. Sorted.
• The build factories contain the actual recipes to be run, as well as the git URL.
• Buildslaves can be reused by multiple factories.

Code and project organization

As you can see, there is project-specific bits encoded into master.cfg, so it makes sense to try and minimize the number of times you have to manually keep buildbot in sync with projects.

Coding conventions

The master.cfg expects the following from python projects:

• Git name is the module name. For example, 'magnet.git' has module 'magnet'. If not, edit the _internal_name routine to add an exception.
• Doxyfile in the project directory that will create docs/html/*
• setup.py in the project directory that will accept an 'sdist' argument
• Source code under module name and tests under module name.test. For example, the DX code is under ooidx, and the tests are under ooidx.test. This allows Trial to be run simply as

  trial module_name

  to automatically pick up any unit tests in the directory.

• Static analysis is run from the module directory
  • pyflakes module_name
  • pylint module_name
  • pep8 module_name

Dependencies and packaging

One of the major challenges was automatically satisfying the package dependencies before running the unit tests. It took some effort; please see this page for the details.

Current status

Buildbot is up and running here. Doxygen is generated from each and moved to the webserver, and results are visible on the waterfall page. As noted, it runs every 30 minutes. We also generate packages and upload them as well; see this page for details.

References and documentation

• Buildbot manual
• Source code docs for buildbot
• How monotone uses it
• How Mozilla uses it
• How Puppet uses it
• How Cairo uses it
• A random tutorial
• How ReviewBoard uses it - subclassing, etc

References: http://buildmonkey.wordpress.com/2009/02/27/continious-integration-with-buildbot-and-git/ http://djmitche.github.com/buildbot/docs/0.7.11/