Description

This is the main page for the OOI pilot period prototype project Data Exchange. This project is a collaboration between IOOS (NOAA) and the OOI.

The goal of the Data Exchange is to make oceanographic science easier by accelerating the transfer of data, providing cloud-based server-side computation and the propagation of standards for data formats and conventions. Whenever possible, we will reuse existing work, adhere rigorously to standards and produce open-source code for the community. The code should scale from a small system that can run on a laptop to a multi-continent archipelago of caches, servers, exchange points and networks.

We expect to be moving large amounts of data, and have spent time learning the tradeoffs of different encoding and transport methods. DAP, with its underlying use of XDR, is quite bandwidth efficient and widely used for remote access to large datasets. Layered on top of DAP are standards and programs that allow transformation, aggregation, computation and visualization. (Thredds, Ferret, NetCDF, COARDS, CF, GridFields, Matlab, Erddap, pydap and many others.)

Here's a diagram, taken from the 2009 kickoff meeting:

Components (briefly)

• Proxy is a lightweight HTTP proxy that takes the users' DAP requests and forwards them to the controller
• The fetcher is the mirror of the proxy, and pulls DAP data and sends it to the proxy and/or persister as directed.
• The persister and cache save and serve data, respectively.
• The controller handles coordination between all the components.
• The attribute store maintains the list of datasets and their status. (Actually, it does more than that, but from the DX point of view that's its role.)
• The management interface is a web-based app for viewing, registering and deleting datasets, and for setting caching policy.

Background and requirements

For a good introduction to some of the user requirements, technological constraints as well as OpenDAP, please see
"Collaboration tools and techniques for large model datasets" by Signell et al, Journal of Marine Systems, Jan 2008. It does a good job of explaining DAP, the alternatives, how it's used and more.

Preceding work and references

Please see also

• The page for the earlier DDN version 1 prototype, deployed in February 2009
• The poster describing this early prototype.
• The 2009 Kickoff meeting poster

Tasks and Deliverables

• See early actions and tasks
• See Data Exchange SoW. Note that due to the accelerated construction of the OOI, the plan has shifted

Development Iterations

• Iteration 1: Planning and Design
• Iteration 2: Planning and Design, Deliverables
• Iteration 3: Planning and Design, Deliverables
• Iteration 4: Planning and Design
• Construction iteration 1: (October - Dec 09) Planning and design

Topics

Source code and documentation:

• Data Exchange source code
• Data Exchange Doxygen documentation

Server-side computation:

• Ferret and FTDS, see also execution and dispatch)
• Gridfields (Installation)
• Using Ferret in Matlab and Python