Work items and their statuses from iteration three. Format is title, work description from the spreadsheet, followed but status and preliminary next-action items.

I. Models

Specification for the Attribute store

Investigation of various type representation systems, for instance to be used for the Attribute Store within message contents.

Data Exchange distributed architecture (from a recent talk):

II. Prototypes

Attribute store prototype

Required for external deliverable. To be used with data publishing. AMQP service on top of Redis. Redis should not bleed into the design.

Status

1. Claudiu's design page has the deep details and API.
2. The code is in git and includes a complete unit test suite for query/write/read/delete/query
3. I wrote up a short page on the implementation as well.
4. Doxygen-based documentations posted on amoeba

Next actions

1. Perhaps run an instance on amoeba for testing? Otherwise complete.

F-TDS Operational deployment

F-TDS in the cloud. Anonymous user access to datasets and server-side processing. We only use a single server deployment for TDS. Possible deployment: (1) Single AMI without Magnet or (2) Fully integrated into current provisioning model with monitoring etc. Variant (2) is the goal. Agreed upon goal: Single instance of TDS in an AMI sans Magnet. Carryover from iteration 2.

Status

Unchanged from iteration two.

Gridfields function prototype

To be defined in design week. Matt's view: In this it we may choose to only do the design for how to do this (not to implement) Depends on getting code from Bill Howe.

Status

Waiting for Bill's code.

DAP over AMQP

Use the DAP protocol to transport data products. Transport over AMQP. Chunk up the messages. Background: AMQP 08 has file transfer; Nanite has an example. Other choice: put into AMQP header whether it is begin, workable chunk or end of the stream. We want to introduce a stream of messages

Status

1. Code is in git
2. Documentation embedded in code via Doxygen and posted to amoeba
3. Magnet now handles binary (XDR) payloads correctly
4. Working proxy/fetcher and persister

Separation of the HTTP DAP reader from the AMQP DAP archiver

On top of a messaging system; separate these architectural entities

Status

1. Code working and in git
2. Doxygen-based documentation posted on amoeba
3. Unit test-based code

Gridfields cloud deployment

To be defined in design week. Matt's view: In this iteration we may choose to only do the design for how to do this (not to implement)

Status

1. Skeleton Java project created from Ferret IOSP source code and placed into git

Next steps

1. Rewrite skeleton code from Ferret to gridfields
2. Deploy to AMI and test

Dataset publishing

Use what was done as prototype in previous iteration. Integrate in operational prototype

Status

• Unchanged from iteration two.

Next steps

• Need to update code to reflect new fetcher/persister/controller architecture.

Ferret as a service

Split TDS from the Ferret processing. Ferret as independent distributed process. Background: Ferret is a separate process spawned for each request. TDS detects duplicates such that the same request is not processed twice. Dispatch work to achieve scale (over AMQP). Ferret as a service. Use the DAP over AMQP to transport data back to TDS server

Status

• No progress this iteration.

Thrift on AMQP

Enhance txAMQP to transport native AMQP types. Make it work with Magnet. It appears that binary floats are slated for the 0-10 AMQP spec. We will have to consider backporting this to 0.8/0.9. Or use Thrift or equivalent in the interim.

Status

• No progress this iteration - effort went into DAP over AMQP subtask, as that proved an excellent driver for the use-cases in question.