New developer tutorial

What this page is

This page is for developers new to OOI, and attempts to collect information to get you started and productive. It's a wiki, so please feel free to edit it as you learn.

Big Picture

What's OOI? From the Ocean Leadership page:

The Ocean Observatories Initiative (OOI) will construct a networked infrastructure of science-driven sensor systems to measure the physical, chemical, geological and biological variables in the ocean and seafloor. Greater knowledge of these variables is vital for improved detection and forecasting of environmental changes and their effects on biodiversity, coastal ecosystems and climate.

and

OOI is the National Science Foundation's contribution to the U.S. Integrated Ocean Observing System (IOOS). While the science-driven OOI will focus on discoveries enabled by new technologies, IOOS will concentrate on direct applications to everyday societal needs. IOOS data will feed into the Global Ocean Observing System, an international program with similar goals.

Technology Big Picture

The primary language in use is Python
Most of our programming is communications, and for that we use and love the Twisted component framework.
All of our communications are transported on AMQP, primarily with the RabbitMQ broker.
Magnet is our Python-based messaging layer, built on top of txAMQP.
Version control is Git, with admin via Gitosis and gitweb.
Documentation is source-code-based, using Doxygen.
We're transitioning to test-driven development, using Twisted's Trial.
Automated build and test is done with Buildbot.
Since you're here now, you know that our wiki is Confluence.
Deployment environment is primarily EC2 and also Nimbus

There are others, but that's the short list. We try to maintain complete lists for the Project Infrastructure and System Development Environment.

Process: Small Impression

There is a central set of architectural documentation that lays out the work. Read this to get some context on the project.
There is an engineering model into which all this work fits.
We describe how to perform each iteration in some detail.

Getting Started

Every developer I know shares a certain amount of impatience. So this section should get you started, checklist-style. This generally assumes the use of a Mac client, and we'll use Magnet as example code.

Confluence/crowd and dev mailing list

See Thomas to get a login. This will give you access to the wiki, Jira, FishEye and the other Atlassian tools. It will also subscribe you to the cidevelopment@ooici.ucsd.edu mailing list.

IRC/XMPP chat

We've installed an ejabberd (chat daemon) on ooici.net, connected to our LDAP database. You should be able to join as soon as you've a Crowd/Atlassian login from Thomas. Sign in using username@ooici.net.

For apps, try Psi as a client.

Git and gitosis

Generate an SSH key on your laptop using
```
ssh-keygen -t dsa
```
and send the id_dsa.pub to Thomas. He'll add you to gitosis, so you can checkout and commit code. For Mac, I also recommend installing SSHKeychain if you haven't already.
Install git locally. Which requires a compiler, so install XCode if you haven't already. It's an optional install on the OSX disks.

Test the system via

git clone git@amoeba.ucsd.edu:magnet.git

EC2 and Nimbus

While we develop on laptops and amoeba, our target environment is the grid. The CPE pages have enormous detail on this, but a synopsis is that 'CPE will abstract this out for you when it's done.' Right now, CPE can move between EC2 and Nimbus already.

However, you'll need at least basic skills on EC2 until CPE gets further.

You'll need the OOI AWS keys and password - see Thomas. Put them into ~/.ec2 directory.
Install Firefox and the Elasticfox plugin. This gives you a nice GUI interface.
You'll also need the EC2 API tools and EC2 AMI tools (they're different)
If you like Eclipse, there's also an AWS plugin
For access to S3, recommend ExpanDrive which has both Mac and Windows versions. For Linux, there's also s3fs

Testing it out

Start an AMI, I recommend the m1.small with an ubuntu server AMI. Go to the 'instances' screen of Elasticfox, right-click and copy the public DNS name into the clipboard. Then you should be able to do something like this:

1
ssh -i ${HOME}/.ec2/ooikeypair.private -l root

(I recommend a shell alias)

Python development environment

Python comes with OSX and Linux, so that's sorted.
You'll want to get virtualenv (as well as fabric and pip) see this article to learn more on these tool. I also use and recommend virtualenvwrapper. That way, I can use the very clean
```
workon dx
```
or
```
workon cpe
```
Logging, logfiles and such are documented here. Python logging == good.

Code editors

At present, the group is split between vim (tutorials here and here) and Komodo. Use whatever makes you happy. Emacs works too, as do Netbeans and Eclipse.

Coding style guide

For Python, please follow PEP-8.

Static code analysis

For quick first-pass analysis, we use pyflakes which is easy to install and use:

easy_install pyflakes
cd magnet
pyflakes .

Try to always have zero pyflakes warnings.

There's also pylint, coverage/figleaf and others TBD.

Unit and integration tests

Please see this page to get started with Trial.

Fudge may be useful for mock objects.

Doxygen

(Not required) We have doxygen and related tools on Amoeba, but if you want to do faster edit/generate cycles, then download the binaries and symlink them into your path:

cd /usr/local/bin
sudo ln -s /Applications/Doxygen.app/Contents/Resources/mscgen mscgen
sudo ln -s /Applications/Doxygen.app/Contents/Resources/dot dot
sudo ln -s /Applications/Doxygen.app/Contents/Resources/doxygen doxygen
sudo ln -s /Applications/Doxygen.app/Contents/Resources/doxytag doxytag

You'll also need to download doxypy.py and save it in the path:

chmod 755 doxypy.py
sudo mv doxypy.py /usr/local/bin/doxypy

Test it out:

cd magnet
doxygen
open docs/html/index.hmtl

Contents should look just like these on amoeba

See this page for instructions on adding Doxygen to your Python code.

Packaging and distribution

The combination of distutils and PyPi makes for one-step installs with dependencies:

easy_install redis

which will download and install redis and any dependencies. For this to work automatically, you have to

Have a proper setup.py (metadata, version, etc)
Have the tarball available for download on ooici.net/amoeba
Have correct project setup (README, MANIFEST.in, license, etc)
Registered yourself with pypi

There's a good tutorial here. Note that we presently have no OOI packages on pypi until the source code license issue is settled.

Once you have everything sorted, a single command should release your code, update pypi and upload the binaries to their server:

python setup.py sdist register upload

See this page for instructions on using Fabric to automate more complex deployments.

Automated build and test

If you're working on an existing module in git, e.g. Magnet, then buildbot is already setup and you can see results at the status page

If you have a new module, please see this detailed page for how to hack on buildbot.

Sample code

As an example project, try the attribute store:

Design docs
Prototype notes (A bit dated)
Code docs
Code via gitweb

Check out the code:

git clone git@amoeba.ucsd.edu:attribute_store.git

Things to learn from it:

Namespace and modules
setup.py for distutils and setuptools (note the dependency list in particular)
Doxyfile and doxygen tags in the source
Use of Twisted - service, interface, factory, protocol and twistd (as.tac)
Message logging
Unit test suite, run with
```
trial attribute_store.test
```

Learning Twisted

Unless you've coded on it before, one major time sink is learning to write code in Twisted. Just another library, you say? Hah! Nope. It's an entire component framework, using asynchronous code to tackle the C10K problem. Expect to spend quite a while on this.

Here are some resources to help you learn Twisted.

This series on Krondo is excellent.
Rabbits and Warrens is a great intro to AMQP and why it's a win for distributed programming. It'll help you understand Twisted, too.
There's a book too, but it's not great other than as a recipe collection.
Eventually, you're going to need to read and grok the infamous finger tutorial. It's hard. Expect to do it multiple times, learning a bit more each time.
There are a number of example codes here that might help.

We also have a number of local docs to help:

Page Operations

Browse Space