Data Exchange

Goal: we take a long view at the data exchange functionality and try to identify the actions and tasks for this effort for a 6-8 weeks period.

Current status: we have opendap data sets and thredds registration capabilities in the DDN prototype 1.

Short term objectives (till mid March)
1. address data caching problem using THREDDS technology
2. first level user interaction model for the UI - what are they working with, how are they thinking about the system, what are the best usage patterns

Actions

The following set of actions were identified during the meeting using Rich's email as starting point.

Action 1. Finalize the DDN prototype

Success criteria: Testing and sign-off on functionality by IOOS Stakeholders (Roy, Steve, Rich)
Deliverable: DDN Prototype improvements

Task 1.1 Revise the user interface and incorporate end-user perspective - come back and talk about user interface, how the user perceives the information.
Task 1.2 Strategy for a common body of information - specifically address how we view subscription and notification, data caching (we already cache metadata)
Task 1.3 Solidify data/metadata caching on the cloud so that the source is only asked during the update cycle.
Task 1.4 Setup prototype feedback review cycle and incorporate feedback into the interface.

Action 2. Deploy F-TDS at scale (cloud strategy)

Success criteria: Demonstrate existing server-side processing features of F-TDS on the cloud (e.g. compute monthly mean from daily SST data)
Deliverable: F-TDS cloud-based deployment

Task 2.1 Understand the technology
Task 2.2 Investigate THREDDS decomposition
Task 2.3 Investigate Ferret extension of THREDDS
Task 2.4 Decompose Ferret for scale

Action 3. Backend extensibility

Success criteria: Demonstrate ability to plug in server-side functionality (e.g, subset an unstructured grid dataset, such as sea surface height, based on an input lon/lat bounding box; display of Curvilinear and Unstructured Grid model results in ERDDAP) and confirm results using existing community tools.
Deliverable: Pluggable architecture for backend, server-side tools to prove its functionality

Task 3.1 Backend functionality abstractions - identify the way to represent the backend functionality and enable end-users to deploy into that strategy.
Task 3.2 Develop and test simple functions such as averaging over rectilinear and table, and over time. For unstructured data leading candidate is Bill Howe's GridFields library - others exist as well.
Task 3.3 Server side functionality architecture for regridding non-rectilinear grids to rectilinear grid - use domain expert and existing tools: Rectilinear => Rectilinear (Easy). Curvilinear => Rectilinear (Harder). Unstructured => Rectilinear (Harder).
Task 3.4 Server side tools for spatial averaging of grids. Rectilinear (Easy). Curvilinear (Harder). Unstructured (Harder).

Action 4. New features

Success criteria: Testing and sign-off on functionality by Matt
Deliverable: notes/reports/new features (case by case basis)

Task 4.1 Server side functionality, not science computational but representational: visualization pipeline for ossimplanet and google earth (ocean)
Task 4.2 Hyrax dataserver integration analysis
Task 4.3 Ability to define datasets based on ncML
Task 4.4 Investigate a model for cache coherency