Integration

Complete stable/usable integration of the django registration app with these components:

• regapp-db-agent (back-end, async database transactions)
• schedule dispatcher
• notification dispatch
• erddap crawler
• mirroring (data loader components)

Registration app related bug fixes

There are various bugs slowing the integration of the dispatcher and syndication.

Listing of things/bugs slowing/preventing integration:

• models
  • datasetId creation; must be globally unique
  • thredds catalog "group" concept; refine, names must also be unique when fully qualified and/or joined with another attribute (i.e. full catalog name and url, etc.)
  • only one user can have the ability to register a particular catalog.

• views
  • loading single datasets
  • improve loading of catalogs (same as refining of grouping concept above)
  • saving and propagating changes to a dataset XML
  • deleting; refine meaning to discern between
    • loaded/unloaded (in erddap); propagate all the way through memcached (add delete functionality to erddaps memcacheMap)
    • registered/unregistered

Mysql cluster

Mirroring

http://www.oceanobservatories.org/spaces/pages/viewpage.action?pageId=16421371

    Machine images are split into two functional components (images): Data requester/transformer and Catalog/Aggregator

  Current functions:

• ERDDAP request generator: Given a dataset, will produce a set of requests where each request represents all data in netCDF form for a single unit of the outer axis variable
• ERDDAP data requester: Takes in set of requests produced by above and writes to disk
• Request messenger: messages requests as serialized dict
• Aggregate Dataset in THREDDS: given path to nc files and datasetid, will produce an aggregated dataset entry in a thredds catalog
• Data repo actions: listens for messages containing requests, calls data requester and aggregate dataset functions

To-Do/Consider:

• Look into Pydap + NetCDF libs ability data server/retriever?
• Erddap has the ability to produce netCDF files from multiple formats (gridded/tabular) but is there a way around making RPCs to ERDDAP to retrieve data?

XMPP Messaging