NDS update (EC/SC) [Christine]
a. Update from the recent EC and SC meetings, status of NDS and future direction, NDS meetings
- Earthcube MOU signed, everything is now in place
- Formalizes the relationship, connection to bring projects to NDS
- Custom catalog of Earthcube building blocks
- NDSC Workshop postponed
- Ideally pushing to January/July
- Co-locating with ESIP, PEARC
- No news on Arnold foundation proposal
- Joel's project to optimally run a group in line with stakeholders
- Interested in working with NDS for platform support
- NCSA funding provided through end of year, looking toward the future
- Separation between consortium from development
- Seeking funding to support consortium activities (meeting organization etc.)
- Need to finalize charter and list consortium members on the website
Development review [Kenton]
a. Present recent work and near term goals related to NDS Workbench
b. Review current roadmap, milestones, etc. for NDS development team direction
- Current focus is on NDS Labs as we are getting good traction
- Progress since last NDS meeting
- Pilot with Carole Palmer class - hands on experience with dataverse etc. (Mostly went well)
- Support for NSF-funded CI4 bootcamp (collaboration with TERRA-REF). 2 week bootcamp for grad students getting hands on data science experience. RStudio/Jupyter.
- Would have liked to do map reduce tutorial
- Workbench generally successful. Some load issues, but worked well.
- Earthcube all hands meeting with their building blocks included
- Internal project with Einstein toolkit (cosmology/astronomy simulation software). Basically Jupyter environment.
- ThinkChicago - hackathon with city of Chicago providing UI system with access to government city data + tools
- Press release from city of Chicago
- Container analysis workshop - bring together people working on providing interactive access to research data. Cyverse, TACC, SDSC, NDS workbench, TERRA-REF, CyberGIS
- Discussing container technology, interoperability, and use cases
- Workbench vision has been evolving
- Started as platform for discovering, analyzing, etc.
- Workshop use cases are drivers at the moment
- TERRA-REF - interactive analysis for remote data
- Connection via MWBDH National bridge inventory
- Looking for a place to host that data (mongo) + sample data code to interact with DB
- Is this more suitable for share?
- ThinkChicago has some large datasets 10-100GB that workbench is not designed to run
- Workbench needs to be extended if we want to deal with the use case of real datasets rather than toy datasets
- Workbench to HPC
- Done analysis/developed code on subset of data, now ready to run large scale processing
- Most centers running singularity, many compute frameworks available
- Agave as a model for submission of batch job
- Package environment in a container, docker converted to singularity, run on HPC system
- Workbench to Cloud
- Want to export from workbench environment to run on a cloud (AWS, openstack)
- Take running config + data and generate a docker compose file
- Not considering provisioning. Rather export to a known format that could be stood up
a. Discuss recent changes to EZID and migration to Datacite.
DataDNS proposal, directions, etc.
RDA and NDS
a. Opportunities at upcoming RDA meeting related to NDS