Dashboard > CI Development > ... > Elaboration Coordination > Elaboration Planning Meeting
Log In   View a printable version of the current page.
CI Development
Elaboration Planning Meeting
Added by David Stuebe , last edited by John Graybeal on Mar 11, 2010  (view change)
Labels: 
(None)

A SnapShot of this page has been exported to Confluence:

copied from: http://etherpad.ooici.org/elaborationplan

Meeting Goal:
Assignments for design period - needs to provide for development of elaboration period plan 

Matt's Proposal for Planning:

Subsystem Leads:
Have tech choices to make 
Also product description for LCA to create at the subsystem level

  • Start making design descisions at the core, the places we already know, then at the interfaces once we have the topdown Product Description (PD)

Susanne and John:
Pull together top down PD

Arch team:
which components come together to meet the end to end product description

Then each team has to design to build that architecture

This is a working document of issues and inputs we have identified overall, and for each subsystem.  We will likely not discuss all these in our first call, but we will organize/prioritize  them and focus on the most pressing. Reminder that the LCO Review, Subsystem Assessment, and Jira tasks are all inputs for the issues in particular.  (Inputs is intended to capture potential new work that we need to factor in during the Design Period.)

Confluence page with some similar presentation of concerns and material:http://oceanobservatories.org/spaces/display/CIDev/Elaboration+Coordination

overall product description

  • need concrete product descriptions to tie component specifications to
  • pub-sub network represented in early prototype (toward NOAA data exchange use case)
  • prototype with MBARI/NCSA informs instrument integration
  • data store across multiple goegraphical sites, operating inside EPU

Functional components of the product that must be there
UI designer and the Arch team have core responsibility for the PD
(decide how to coordinate the PD this/next week due to unavailability of Mike, Matt, Susanne) Goal: get the PD by Monday-Tuesday next week, and till Friday look at how the components integrate.
Subsystem leads have input, not responsibility
Matt and John have responsibility to lead the PD process once we get off this call

Each subsystem needs to design core components that we know now are needed:
The set of activites that we can start on now - this week while the PD team developes the top down view.

* Top-down info for LCA:
- the DDN doc from Matt (Alx & Dorian, 2008) - use case for NOAA integration
- Commercial example: Superfeedr
- NCSA/MBARI demo prototype for integration/securiy - use case for release 1 S&A

We need a  clear articulation of what the LCA product will look like. In parallel, each team can make their choices of technologies for the components that make the integrated system. 

COI — Emilia/Claudiu/Matt

Technology choices for the elaboration period by service:
1) Resource registry. Implement the core resource registry service and enable specialization
by resource type specific services, for instance a service (resource) registry

In R1 we have to solve the multi-process, multi-site, single domain of authority data store.
Resource Registry Data Model - implemented on top of Data Store Data Model
Resource Registry: aka Data Store Redis, Cassandra, other? 'Eventual Consistency' is our target for R1
Data Store Strategy - for multi-process multi-site multi-Domain or Authority High Available Srvc
propose we use CUAHSI architecture as point of departure for these evaluations. Project the CUAHSI data model on the Redis dm and Cassandra dm to evaluate (define metrics for the evaluation). Look at how the querries work on the data models.

Should we consider using both Redis & Cassandra? Each is optimized for a different purpose. Example architecture: each worker process works on a cached data, which has a relation with the data store. Assess what is viable and what we can decide in R1.

2) Resource description. Together with the DM subsystem, define the attributes and life
cycle that each resource follows, so that resource management can occur uniformly
across the system.

3) Provide a reliable capability container for service deployment. This capability container
should provide all agent/governance/policy and identity management as necessary
for service interaction.
Capability container -Python, Twisted, txAMQP 
Messaging Srvc - Exchange Spaces, RabbitMQ AMQP broker
DIF (application networks - VLAN - Network Interface)
Whether we implement the Facility model in release 1 has still to be decided. We don't need the edge of domain of authority in release 1, but if we don't implement it in R1 we have to redo a lot of code to support it in R2, R3.

What is the connection between the networking architecture and the messaging arch?

In Elaboration phase COI has to bring up a Java capability container!
We need to support NetCDF at LCA, which will require Java capability container

We can go through LCA withouth governance/IdM fully in place, but we have to show that we have hooks in place.

Must have the OOI message header in place by LCA

4) Provide an initial resource agent implementation with integration into the capability
container and the control of policy enforcement
Policy & Governance - Jess/Pyke rules engine

CEI---Tim F/Kate K

Elaboration Phase work items:

  • EPU: built with COI capability container in Python with supporting libraries.  The Data Store will play a role in reliable data structures for tracking/accounting in the face of corruption problems.
  • EPU stress harness: components will be exercised by DM subsystem, the  work of the "harness" is to automate this exercising such that we can repeatedly stress it.  This will be an actual component of the LCA demonstration as well.
  • EPU reliability harness: During the DM stress cases, another system will be built (in Python) to corrupt/destroy parts of the running system.  Monitoring infrastructure will show that the system is still running (perhaps more slowly before fault compensation).
  • VM image authoring and browsing.  Using rBuilder.  DM subsystem (in context of CEI demo) could be the initial "real" user of this.

EPU Agent to control the state of the EPU form powered off through Activation and Scaling back to Power off

must show Nimbus cloud working on top of VMware, or leave VMware approach. Real goal is to do this on VMware. Exploration to determine this cost has to be done during this period, so that we can adjust everything in Elaboration accordingly. Evaluation/choice to be made in first 3-4 weeks of development of Elaboration phase. Not much more time to hold decision, based on active discussions with VMware. This two weeks needs to create the plan for that evaluation.
  The intent here is to make the box crisp, to surface the dependency on KVM that is not covered by libvert.

Also have to show EPU can work on 2 separate sites using 2 different strategies. Have to show applications at end of this phase working on both Amazon and one of our own clusters.

Want to see Data store as EPU on multiple sites and execution environments. 
  Does that decide between Redis and Cassandra?  No, we could implement synchronization for Redis. May be required independent of use of Cassandra, since you can't rely on the common data store (the services may differ from one to the other). In Federation will have to synchronize outside single technology domain. (Look at GOSIP architecture with Cassandra, maybe we don't have to solve that now, just set up for it in R1.)
  Don't have to consider federation across domains of authority at this point. Syncing is at a higher level.

DM---David S

Top Down View of Functionality from DDN1: (Product description)http://www.oceanobservatories.org/spaces/display/CIDev/Data+Distribution+Network

Elaboration Phase Work items

  • The service architecture of the DX must be re-factored to reflect the DM architecture and use the latest version of magnet. 
  • The resource registry service must be implemented to store and retrieve information resources in the inventory. 
  • Can the CUAHSI data model be implemented in nosql technology
  • Can the Common Data Model be implemented in nosql technology
  • The ingestion and transformation services need to be implemented to convert from DAP to the CI native encoding on ingestions and back to DAP for external presentation. We may also need to handle other encodings used in IDD.
  • Example services for at least one feature type from the Unidata Common Data Model must be implemented to provide an end to end demo by LCA. 
  • The persistent archive service must be prototyped and the integration plan specified.

Technology choices for the elaboration period:
Inventory: See COI resource Registry
Message encoding: Fudge, Google Protobuf, other?
Transformation: NetCDF Java, CDAT, Hyrax ? How do we get CDM features implemented?
Preservation: IRODS

SQL - table space
Cassandra - key space 
Redis - most flexible

How well will the projection of CUAHSI on the data store work?
How well will their queries work?

Are their deficiencies in their data model that may be a problem to solve in this technology?

We need Java CC to get NetCDF - we need it for LCA - this is an untenable dependency on a work item that has no execution plan!

Product description relationship with S&A - most affected by PD!
Three places to deliver comperable techs

S&A---Arjuna Balasuriya

decide on set of services associated with managing and data acquisition of instrument
second decision is on how we will construct interactions of capability container with instrument driver
  would be natural for team to use core of MOOSDB, treat drivers as being modules

You'll find you use data store and capability container as instrument agent 
  (ORB from Antelope, Blackboard from MOOSDB instance of data stores)

Management of instruments (commanding them) is core
same agent for other subsystems will be usable by S&A (notion of taskable resources, executable resources)
have to come out with clear articulation of the services of information, executable, and taskable resources

in LCA doesn't have to be implemented with any one of the technologies, would hope we'd focus on data store as a mechanism

state control more important than data acquisition
behavior functionality of MOOSDB is for later releases

also may find that data acquisition service interface is an attribute store interface
  information store is local, overall acquisition service is equivalent to synchronization of two independent data stores (= federation)

Elaboration Deliverables
1. Use-cases for each service in Release-1: direct access, Registering an instrument
and data processes, Data acquisition
2. Much more detailed architecture diagrams for Instrument agent, repositories, direct
access, data acquisition
3. Implementation strategies for the candidate technologies in S&A; demonstration of
some strategies (SIAM driver implemented for selected instrument; possibly PUCK
data accessed)

Only the basic functionalities will be demonstrated for LCA. Features such as scheduling
and governance will not be in place.
1. Pick most commonly used sensor (e.g CTD)
2. Simple UI (shell type) which will provide features for services such as;
2.1. Register an instrument
2.2. Register data processes
2.3. Pick an instrument from the inst. registry
2.4. Pick the data type from data process registry
2.5. Request services - Direct Access or Data Acquisition
3. Working code encapsulating above features to demonstrate on how a sensor such
as a CTD is connected to CI and operated in Direct access and data acquisition
modes, parsing data to the ingestion services in Data Management subsystem. The
demonstration will also capture hand-offs, such as from Data Acquisition (Observatory)
mode to Direct Access mode, and vice-versa.

*Instrument Agent---*Arjuna   

Will have to use the agent architecture designed by COI?

  Sensor prototype
  
*External Observatories integration---*David

  IOOS collaboration

*User interfaces/user experience---*Susanne Jul

  • behind on this
  • won't ask end users of R1 (not administrators) to download something in R1
  • operator in R1 is our own operations team, process of deployable types may address this (CEI deliverable).

Other notes

What is the road map for these 2 weeks?
     core statement of product: week 1
     presentation next Monday to this meeting

Overall product description format?

  • Michael's list is a notional start
  • Format examples can be identified
    !
Powered by Atlassian Confluence 2.7.1, the Enterprise Wiki. Bug/feature request - Atlassian news - Contact administrators