Potential of I/O Aware Workflows in Climate and Weather

The efficient, convenient, and robust execution of data-driven workflows and enhanced data  management are essential for productivity in scientific computing. In HPC, the concerns of storage  and computing are traditionally separated and optimised independently from each other and the  needs of the end-to-end user. However, in complex workflows, this is becoming problematic. These  problems are particularly acute in climate and weather workflows, which as well as becoming  increasingly complex and exploiting deep storage hierarchies, can involve multiple data centres. The key contributions of this paper are: 1) A sketch of a vision for an integrated data-driven  approach, with a discussion of the associated challenges and implications, and 2) An architecture  and roadmap consistent with this vision that would allow a seamless integration into current  climate and weather workflows as it utilises versions of existing tools (ESDM, Cylc, XIOS, and  DDN’s IME). The vision proposed here is built on the belief that workflows composed of data, computing,  and communication-intensive tasks should drive interfaces and hardware configurations to  better support the programming models. When delivered, this work will increase the opportunity  for smarter scheduling of computing by considering storage in heterogeneous storage systems.  We illustrate the performance-impact on an example workload using a model built on measured  performance data using ESDM at DKRZ.