Buildbot at OOI

Introduction

As of August 2009, we're moving our automation system from the prototype homebrew creation to Buildbot

Buildbot, written in Python, is tremendously flexible as is being used in complex environments such as Chromium; it has plenty of flexibility for OOI. Some of the things it can do:

Run unit tests, either Python UnitTest or Trial/Twisted.
Run the pylint and/or pyflakes static code analyzers
Compile, package and upload the results to the buildmaster
Run any system-callable script
Run periodically or when code is committed (post-update hook)

OOI buildbot configuration and notes

Note that this is preliminary as we explore buildbot; please check back to this page!

Buildbot runs a single master and one or more buildslaves. Each buildslave is typically a different operating system or hardware, so that you can run your testing and packaging on all supported platforms automatically. Right now, we have build slaves for Ubuntu/64bit (on amoeba), and OSX 10.5 (spare mini in 3604.) We plan to add buildslaves for Ubuntu/32 and windows/32, and are considering ubuntu/32 (to match our typical AMI install) and perhaps Redhat.

Details of setup

On amoeba, the buildmaster and buildslave are both run from the 'buildbot-runner' user account that we setup for this purpose. To modify the configuration, you have to sudo or su to buildbot-runner.
The webserver for amoeba is in /var/www and if you want to auto-generate doxygen docs the target directory tree is /var/www/doxygen
The entire configuration for buildbot is in Git as 'buildbot'

Modifying the OOI buildbot

To make this easier, the entire buildbot setup is in git as its own project.

git@amoeba.ucsd.edu:buildbot.git

The project is non-gitweb and non-public due to the static passwords in master.cfg and buildslaves. Kind of a pain, that.

The project has one directory for the master and another for slaves, if you want to add a new buildslave please follow the same pattern.

Once you've edited master.cfg, you can do a sanity check with

buildbot checkconfig master.cfg

before you go live. To start or restart the buildbot:

cd $HOME/buildbot/master
buildbot restart .

You can also use that same restart command for the slaves.

Master.cfg

You'll quickly figure out that the buildslaves basically don't have any configuration; it's all in master.cfg. Understanding it requires some basic concepts: Schedulers, builders, factories and the glue around them.

Change sources notify buildbot of a commit. There's a PBChangeSource that is used for Git. Right now, we're not using this due to issues below. We just use a periodic instead.
The post-receive hook in git just fires the generalized buildbot trigger, caught by the PBChangeSource:
```
python /usr/share/buildbot/contrib/git_buildbot.py $1 $2 $3
```
(that's the entire contents of /home/git/repositories/attribute_store.git/hooks/post-receive)
The PBChangeSource fires the Scheduler, which decides which builds to run.
The Scheduler is keyed on branch name, which since we have multiple repositories with branch 'master' is a problem. You can fix this by writing lots of Buildbot source code; instead I chose to use a Periodic scheduler instead, which runs all the build factories every ten minutes. Sorted.
The build factories contain the actual recipes to be run, as well as the git URL.
Buildslaves can be reused by multiple factories.

c['schedulers'].append(Periodic(name='10-minute', branch='master', periodicBuildTimer=10*60,\
                                builderNames=['AS-linux64', 'AS-osx', 'DX-linux64', 'DX-osx', 'MS-linux64', 'CPE-linux64', 'TXR-linux64']))

# Factory for amoeba and attribute store
f1 = factory.BuildFactory()

f1.addStep(Git(repourl='git@amoeba.ucsd.edu:attribute_store.git'))
f1.addStep(PyFlakes, command=['pyflakes', '.'])
f1.addStep(Trial(testpath='test', tests='attribute_store.test'), description='Run unit tests')
f1.addStep(ShellCommand, description='Build distribution', command=['python', 'setup.py', 'sdist'])
f1.addStep(ShellCommand, command=['doxygen'], description='Generate doxygen docs')
f1.addStep(ShellCommand, description='Move doxygen docs to web folder', command=['/home/buildbot-runner/bin/create_doxy.sh', 'attribute_store'])

# Simpler factory for slave
f2 = factory.BuildFactory()
f2.addStep(Git(repourl='git@amoeba.ucsd.edu:attribute_store.git'))
f2.addStep(Trial(testpath='test', tests='attribute_store.test'), description='Run unit tests')
f2.addStep(ShellCommand, description='Build distribution', command=['python', 'setup.py', 'sdist'])

# Other factories look the same, see master.cfg...

b1 = {'name': 'AS-linux64',
      'slavename': 'amoeba-buildbot',
      'builddir': "AS-l64",
      'factory': f1,
      }
b2 = {'name': 'AS-osx',
     'slavename': 'osx-buildbot',
     'builddir': 'AS-osx',
     'factory': f2,
     }
# Other builders the same too

c['builders'] = [b1, b2, b3, b4, b5, b6, b7]

Code and project organization

As you can see, there is project-specific bits encoded into master.cfg, so it makes sense to try and minimize the number of times you have to manually keep buildbot in sync with projects. I think a toplevel project for each area makes sense:

data exchange
messaging service
cloud provisioning environment
attribute store
txrabbitmq

So that's what's in there for now. We're also investigating Atlassian Bamboo to see if it'd be easier to setup and use.

Coding conventions

The master.cfg expects the following:

Doxyfile in the project directory that will create docs/html/*
setup.py in the project directory that will accept an 'sdist' argument
Source code under module name and tests under module name.test. For example, the DX code is under ooidx, and the tests are under ooidx.test. This allows Trial to be run simply as
```
trial ooidx.test
```
to automatically pick up any unit tests in the directory.
pyflakes is also run and is good for catching code lint

Current status

Buildbot is up and running, on attribute store (linux, OSX), data exchange (linux, osx), CPE, Magnet and txrabbitmq. Doxygen is generated from each and moved to the webserver, and results are visible on the waterfall page. As noted, it runs every ten minutes.

References and documentation

Issues

The big hassle so far is code dependencies - attribute store requires redis, magnet and txamqp. Figuring out how to integrate virtualenv into buildbot would be of great value here.

References:

http://buildmonkey.wordpress.com/2009/02/27/continious-integration-with-buildbot-and-git/

http://djmitche.github.com/buildbot/docs/0.7.11/

Page Operations

Browse Space