NDS Share

Owned by Kenton McHenry
Last updated: Jul 29, 2016 by James Myers
7 min read

Possible Template for NDS Share Charter and agreements with resource providers - for discussion as one possible model - one that emphasizes NDS and participant commitments of resources to users (research groups with data) and the commitments between NDS participants to support interoperability and coordination of services. Modified 6/28/16 to clarify requirements, particularly for service-oriented resources.

 

National Data Services (NDS) Share: A Collection of NDS Consortium Data Resources committed to the realization of the NDS Vision 

The National Data Service Consortium is dedicated to realizing an emerging vision for how scientists and researchers across all disciplines can find, reuse, and publish data.

Whereas a significant challenge in realizing this vision is a lack of readily-available hardware (storage) and software resources, the organizations listed below agree to cooperate to provide some initial resources (detailed below) and to support their availability on a best-effort basis. These resources are being provided for the following key reasons/ with the following key goals:

  • To demonstrate the capabilities of publication services for data that—due to their subject, scope, scale, heterogeneity, and/or complexity—are not well-served by existing publication services and existing repositories
  • To begin aggregating the demand for data publication from the long tail of underserved projects to demonstrate the nature, scale, and scope of that demand and hence better inform national-scale data services efforts
  • To provide best-effort support for the growing demand for data services from existing research projects and from proposed next-generation projects that would leverage such services (and the data they manage)– as that demand outstrips the resources of individual cyberinfrastructure development projects
  • To provide a growing corpus of real, openly available research data that can be used by cyberinfrastructure developers to help define, develop, scale, test tools and services, and assess interoperability across services
  • To assure that NDS discussions are informed by real-world examples of the full range of data products that researchers are producing and reusing.
  • To serve as an exemplar for how NDS Consortium participants can coordinate to provide rich, sustainable data services through an evolving set of software, resources, and organizational commitments 

NDS Share Resources:

NDS Share is composed of resources provided by NDS Consortium members. The resources provided range from general compute and storage capacity to software components and libraries to data-related services (for analysis, curation, publication, preservation, discovery, conversion, etc.). In its formative stages, most resources committed to NDS Share are expected to be provided on a 'best-effort' basis by contributors who are leveraging their ongoing projects and internal resources. To have resources become part of NDS Share, resource providers must meet some minimal requirements. 

For resources to be accepted into NDS Share, the following commitments are required: 

1) The data exchanged with other NDS services will be associated with a citable persistent identifier

2) The data produced through the use of the committed resources and/or shared with other NDS Share capabilities will be available under an open (Creative Commons or similar) license

3) All data and metadata generated through the use of committed resources in connection with this agreement will be made available in a way that it can be exchanged with other NDS services through a standard API and/or serialized export formats. Acceptable option(s) will be defined within the next 6 months through NDS participants, specifically through the work of interoperability-related pilot projects and the coordination activities of the NDS Technical Advisory Committee's Interoperability Task Force.

4) Resources must be provided openly to all NDS participants

 Resource providers are free to place additional requirements on the use of their specific resources within NDS Share. NDS will review the offered services and any resource-specific requirements to assure that they are consistent with minimal requirements listed here and with the stated goals of NDS Share (listed above) before accepting the offered resource into NDS Share.

The following organizations intend to provide Data Resources through NDS:

<Best-effort Compute/Storage commitment example>

<Org Name, e.g. NCSA>

<Org Name> will provide, as part of NDS Share, storage for published data, up to <size, e.g. 200TB>, and associated cloud computing resources to support access <and possibly co-located computing> that would be maintained, on a best-effort basis, for a minimum of <# years, e.g. 3? 5?> years. These resources are intended to support active US research efforts producing data with ongoing scientific value and for which alternative data repository options do not exist. Use of the <Org Name> resources in NDS Share will be managed by policies developed by <Org Name> in collaboration with NDS participants. For projects requesting less than <size, e.g. 1 TB> of storage for data publication and having no computational needs beyond those required for data access, permission to use the <Org Name> Share resources would be automatic based upon a request from the leadership of a funded research project in the US and a self-assertion of the value of the data and the appropriateness of NDS Share as a publication option at a level equivalent to the justification in an NSF Data Management Plan. For projects that require more space and/or significant cloud resources, a statement from a community(ies) of potential users attesting to the data’s value and to the need for NDS Share resources, e.g. because there is no alternative repository available and/or because co-location with the tools and services being developed in NDS Labs would add significant value, will be required and will be reviewed by <review team, e.g. an <Org Name>/NDS committee>. <Org Name> reserves the right to immediately revoke allocations and/or to block access to data publications and reclaim the storage space if it determines that continued access would be inconsistent with the stated purposes of the provided resource.

<Org Name> will work through NDS to explore opportunities to move specific publications to more appropriate, long-term repositories. As a means to extend the period during which resources will be available,  and to address oversubscription if/when that occurs, <Org Name> and NDS will explore opportunities to support expansion and extension of this storage and will, as necessary, develop policies to remove (deaccession) data, to maximize the overall scientific value of the provided resources.

<Org Name> agrees to impose minimum requirements as specified by NDS on the publication model and the software used to store and access data publications stored in this resource:

  • The data publications and derived data products created using this resource will be associated with a citable persistent identifier
  • The data published to or generated with this resource will be available under an open (Creative Commons or similar) license
  • Any software required for publication and access that will run within this resource must be compatible with the containerized deployment methods used within NDS Labs. (Note this is a resource-specific requirement)
  • The software must—either natively or through the use of an open source tool—allow programmatic access to all the data and metadata comprising the publication through a common API and/or standards-based serialized format (e.g. a file). Acceptable option(s) will be defined within the next 6 months through NDS participants, specifically through the work of interoperability-related pilot projects and the coordination activities of the NDS Technical Advisory Committee's Interoperability Task Force.

 

<Existing Repository/Project/Service example, e.g. DIBBS, DataNet, operational repositories, etc.>

<Repository/Project Name>

 

<Repository/Project Name>, supported through <funding sources> provides services to the <domain> community(ies) in the following areas: <list, e.g. data publication, data visualization, cloud computing over data, data transformation, ...>. As part of its participation in NDS, <Repository/Project Name>   intends to make it's offerings compatible with NDS Share in one or more of the following ways and to offer services as described

NDS compatibilities:

  • implementing NDS interoperability mechanisms and supporting transfer of data to/from services running in NDS Share and/or Labs
  • supporting deployment of <Repository/Project Name> services and/or compnents within other NDS Share resources, (e.g by supporting containerized deployment as being explored in the NDS Labs environment)
  • coordinating with other NDS participants to provide links between <Repository/Project Name> services and NDS capabilities via 3rd party services

Offerings through NDS:

  • <Repository/Project Name>intends to offer its current user base access to offered NDS capabilities as part of its operations
  • <Repository/Project Name>intends to offer user of other NDS Share resources access to its services, under its existing policies, as part of its operations
  • <Repository/Project Name>intends to offer research projects leveraging NDS Share capabilities the use of <Repository/Project Name>'s services on a best-effort basis leveraging <Repository/Project Name> resources <describe scope/scale>
  • <Repository/Project Name>intends to offer research projects leveraging NDS Share capabilities the use of <Repository/Project Name>'s services on a best-effort basis by supporting deployment of <Repository/Project Name> services/components within other NDS Share computational and storage resources, subject to the limits of those resources and subject to any per-person or per project limits places on those resources

 

<Repository/Project Name>  reserves the right to immediately revoke allocations and/or to block access to its data services and reclaim the storage space if it determines that continued access would be inconsistent with the stated purposes for which these services are being provided as part of NDS Share.

<Repository/Project Name> will work through NDS to explore opportunities to enhance their collective offerings. As a means to optimize the scientific value of the contributions being made, <Repository/Project Name> and NDS will explore opportunities to support expansion and extension of the provided capabilities and will, as necessary, develop policies to allocate resources and/or remove (deaccession) data as necessary .

<Repository/Project Name> agrees to impose minimum requirements as specified by NDS on the publication model and the software used to store and access data publications stored in this resource:

  • The data exchanged with other NDS services will be associated with a citable persistent identifier
  • The data produced through the use of <Repository/Project Name> capabilities and or published through <Repository/Project Name>  and/or shared with other NDS Share capabilities will be available under an open (Creative Commons or similar) license
  • All data and metadata generated through the use of <Repository/Project Name> services in connection with this agreement will be made available in a way that they can be exchanged with other NDS services through a standard API and/or serialized export formats. Acceptable option(s) will be defined within the next 6 months through NDS participants, specifically through the work of interoperability-related pilot projects and the coordination activities of the NDS Technical Advisory Committee's Interoperability Task Force.