Advancing open and reproducible water data science by integrating data analytics with an online data repository

Abstract

Scientific and management challenges in the water domain require synthesis of diverse data. Many analysis tasks are difficult because datasets are large and complex, standard formats are not always agreed upon or mapped to efficient data structures, scientists may lack training for tackling large and complex datasets, and it can be difficult to share and reproduce data science workflows. Overcoming barriers to accessing, organizing, and preparing datasets for analyses can transform how water scientists work. Building on the HydroShare repository's cyberinfrastructure, we created a Python package that automates data retrieval, organization, and curation for analysis, reducing time spent in choosing appropriate data structures and writing data ingestion code. It manages metadata and automates data loading into performant structures consistent with Python's visualization, analysis, and data science capabilities and can be used to build and share more reproducible scientific workflows in HydroShare following FAIR (Findable, Accessible, Interoperable, and Reusable) principles.