Dashboard > CI Development > ... > System Development Environment > New developer tutorial
Log In   View a printable version of the current page.
CI Development
New developer tutorial
Added by Paul Hubbard , last edited by Michael Meisinger on Aug 15, 2010  (view change)
Labels: 

What this page is

This page is for developers new to OOI, and attempts to collect information to get you started and productive. It's a wiki, so please feel free to edit it as you learn.

Big Picture

Please see the Overview for New Developers for more information on the project, the CI technology choices, and the engineering processes we use.

Sample code, for the truly impatient

LCAARCH - Complete OOI CI architecture prototype in Python

As of April 2010, there a complete OOI CI architectural prototype project in Python available called 'LCAarch' to jump start you right in. See the LCAARCH Development Project page to get you started and for more details. Grab a copy with

git clone git@amoeba.ucsd.edu:lcaarch.git

If this fails, you might not yet have an account with the GIT server (see below). Use anonymous checkout instead to move ahead:

git clone git://amoeba.ucsd.edu/lcaarch.git

Skeleton - Complete simple example Python project

If you prefer to look at a simple example project, look at the "skeleton" project explained below. The documentation is doxygen based, with a generated copy online for you here. See below for more information on git.

Skeleton has unit tests, doxygen, setup automation and such; please have a look. You can browse the source code online here if you don't have git installed yet.

Java - Python interaction

As of August 2010, there is a Java client/example project available for interaction with the Python based LCAarch system. In addition to LCAarch (see above), get ioncore-java:

cd mycodedir
git clone git@amoeba.ucsd.edu:ioncore-java.git
cd ioncore-java

And then follow the instructions on the IONCore-Java page.

Other tools

Confluence/crowd and dev mailing list

See Qian or Rita to get a login. This will give you access to the Confluence wiki, Jira, FishEye and the other Atlassian tools via an LDAP server. It will also subscribe you to the cidevelopment@ooici.ucsd.edu mailing list.

IRC/XMPP chat

We've installed an ejabberd (chat daemon) on ooici.net, connected to our LDAP server. You should be able to join as soon as you've a Crowd/Atlassian login from Qian. Sign in using username@ooici.net.

For apps, try Apple iChathttp://psi-im.org/ as a client. Psi works as well.

Documenting Your Work

Copy the Task Development Template to an appropriate place in the system (usually under the current iteration in the CI Development pages), and use it to provide the key information about your task as you go along.

Development Environment

See here for more details for the Developer Environment. The material below is to get you started.

Directory Structure

You don't have to follow this structure, but doing so makes life easier. Execute the following commands in the terminal:

cd ~/Documents
mkdir Dev
cd Dev
mkdir virtenvs
mkdir code
mkdir apps

The directory structure should look like as follows. Modify/refine as you please.

~/Documents/
~/Documents/Dev/           # This is where all development related stuff goes
~/Documents/Dev/virtenvs/  # For python virtual environments (keeps libraries apart)
~/Documents/Dev/code/      # For downloaded packages and your own code
~/Documents/Dev/apps/      # For downloaded binaries, such as Redis

Git and gitosis

See the Configuration Control page for more about git, gitosis, and other version management processes as used on OOI CI.

Python development environment

  1. Python comes with OSX and Linux, so that's sorted. (Note: we use Python 2.6+, not Python 3.0)
  2. Setup the directory structure as explained above or think about your own structure. The most important thing is to separate virtual environments (different sets of libraries) from the actual working code.
  3. Get the Apple Developer tools (XCode) first. This is a large download (700 MB). Take some time
  4. Get virtualenv, see this article to learn more on these tool. 

    sudo easy_install virtualenv

  5. Get and install virtualenvwrapper . 

    sudo easy_install virtualenvwrapper

    Edit ~/.profile and add the following lines

    export WORKON_HOME=~/Documents/Dev/virtenvs
source /usr/local/bin/virtualenvwrapper.sh

    Question
    Should the WORKON_HOME environment be defined at this point? Note the instructions in virtualenvwrapper.sh are a bit different. Precedence?
  6. Prepare an initial virtual environment 

    cd ~/Documents/Dev/virtenvs
virtualenv ion --no-site-packages
source ion/bin/activate

    Add more virtual environments as needed to keep sets of python libraries separate, e.g. for experimentation with different versions. Note that the --no-site-packages option for creating the new virtualenv states that NO other third party packages are shared with this virtualenv. You have to install all of them in your virtualenv manually, which provides maximum isolation.

  7. Having done this (through virtualenvwrapper), we can use the very clean 

    workon ion

    or

    workon lcaarch

  8. Install a few utility packages as prerequisites for subsequent steps. See Installing Dependencies for several more packets and libraries. Note: you don't need any of them for initial progress
  9. Logging, logfiles and such are documented here. Python logging == good.

Python debugging

Ahh, debugging, always a challenge. Here are some starting points to help:

  • Python has a native debugger call PDB that resembles GDB. Nothing fancy, but it's there.
  • Tim has nicely documented a really useful tool - you can invoke an ipython shell at any point in the execution of your program, which is very helpful for when you have complex runtime state.
  • You can solve a lot of simple problems by using an interactive iPython shell and just running code by hand.
  • If you're using Komodo:
    • It has a pretty good debugger (breakpoints, etc) built in
    • You can also connect to remote code using a proxy.
  • There's nothing like printf debugging sometimes. See this page on python logging for some help. You can do all sorts of flexible and configurable stuff with python logging that are really helpful, e.g. crank up the verbosity on module 'foo' and silence everything else, log to a socket, etc, etc.
  • Twisted adds its own set of complexity, and Deferreds/inlineCallbacks especially

EC2 Development environment setup

Several prototypes are running on Amazon's EC2 (Elastic Compute Cloud service), part of the Amazon Web Services (AWS). Setup of an EC2 development environment is encouraged but not required for initially getting started.

See here for a more detailed tutorial.

Empty Python Development Project

There's a complete Python sample project called 'skeleton' to help you get started if you want to set up a new GIT project. Grab a copy with

git clone git@amoeba.ucsd.edu:skeleton.git

Use it as reference to start own development projects.

Code editors

At present, the group is split between vim (tutorials here and here) and Komodo (note there is also a free editor only version with Komodo). Use whatever makes you happy. Emacs works too, as do Netbeans and Eclipse. A more recent entrant is geany (X11, GTK2, cross-platform). The Wing IDE is pretty clean, snappy, X11-based, and cross-platform with a decent debugger...but is python only.

Coding style guide

For Python, please follow PEP-8.

Static code analysis

For quick first-pass analysis, we use pyflakes which is easy to install and use:

easy_install pyflakes
cd code/magnet
pyflakes .

Try to always have zero pyflakes warnings.

There's also pylint, coverage/figleaf and others TBD.

Unit and integration tests

We encourage you to write unit tests for your code, either TDD-style or however suits your style. We have an in-depth tutorial on writing unit tests for communications code on this page.

Here's how trial is started to run unit tests with Twisted (or without)

cd lcaarch
trial ion
trial ion.core
trial ion.services.coi.test.test_resource_registry

Note also that the unit test suites are run by Buildbot, so they need to be cross-platform and not hardwired for your laptop!

Fudge may be useful for mock objects.

Doxygen

Documentation is generated by buildbot and posted to the website, but if you want to do so locally, then download the binaries and symlink them into your path:

cd /usr/local/bin
sudo ln -s /Applications/Doxygen.app/Contents/Resources/mscgen mscgen
sudo ln -s /Applications/Doxygen.app/Contents/Resources/dot dot
sudo ln -s /Applications/Doxygen.app/Contents/Resources/doxygen doxygen
sudo ln -s /Applications/Doxygen.app/Contents/Resources/doxytag doxytag

You'll also need to download doxypy.py and save it in the path:

chmod 755 doxypy.py
sudo mv doxypy.py /usr/local/bin/

Test it out:

cd magnet
doxygen
open docs/html/index.html

Contents should look just like these on amoeba

See this page for instructions on adding Doxygen to your Python code.

Packaging and distribution

The combination of distutils and PyPi makes for one-step installs with dependencies:

easy_install --find-links http://ooici.net/packages data_exchange

which will download and install the data exchange and its rather complex dependencies. For this to work automatically, you have to

  1. Have a proper setup.py (metadata, version, etc)
  2. Have the tarball available for download on ooici.net/packages (done for you by buildbot)
  3. Have correct project setup (README, MANIFEST.in, license, etc)

There's a good tutorial here. Note that we presently have a local repository until the source code license issue is settled.

Automated deployment

See this page for instructions on using Fabric to automate more complex deployments.

Automated build and test

If you're working on an existing module in git, e.g. Magnet, then buildbot is already setup and you can see results at the status page

If you have a new module, please see this detailed page for how to hack on our buildbot.

We have simple recipes for testing under linux-64 and OSX-32.

Sample code

  1. Start with the skeleton project - no actual code in it, but it does have a correct Doxyfile, setup.py, unit test suite, etc.
  2. Proceed to the lcaarch project - complete OOI architecture prototype
  3. After that, the attribute store is perhaps the next place to look:

Things to learn from it:

  • Namespace and modules
  • setup.py for distutils and setuptools (note the dependency list in particular)
  • Doxyfile and doxygen tags in the source
  • Use of Twisted - service, interface, factory, protocol and twistd (as.tac)
  • Message logging
  • Unit test suite, run with 

    trial attribute_store.test

Learning Twisted

Unless you've coded on it before, one major time sink is learning to write code in Twisted. Just another library, you say? Hah! Nope. It's an entire component framework, using asynchronous code to tackle the C10K problem. Expect to spend quite a while on this.

Here are some resources to help you learn Twisted.

  1. This series on Krondo is excellent.
  2. Rabbits and Warrens is a great intro to AMQP and why it's a win for distributed programming. It'll help you understand Twisted, too.
  3. There's a book too, but it's not great other than as a recipe collection.
  4. Eventually, you're going to need to read and grok the infamous finger tutorial. It's hard. Expect to do it multiple times, learning a bit more each time.
  5. There are a number of example codes here that might help.

We also have a number of local docs to help:
Powered by Atlassian Confluence 2.7.1, the Enterprise Wiki. Bug/feature request - Atlassian news - Contact administrators