FlowCron - Increasing access to HPC by wrapping Globus into a function-as-a-service.

Abstract

Despite significant investment in High-Performance Computing (HPC) clusters by funding councils, there are still many researchers whose workflows could not benefit from the computation speed that is provided by these clusters. Reducing barriers to entry for these researchers would accelerate their scientific throughput, since they will be able to respond to results in a timely fashion, improving either their protocols or correcting any problems that might have arisen. This improves the quality of science, and therefore the return on investment, in computationally-intensive areas such as Cryogenic Electron Microscopy (cryo-EM). This paper outlines a technique, FlowCron, for users to analyse their data on a HPC facility with minimal training, increasing accessibility. FlowCron transfers the responsibilities of installation and upkeep of data processing pipelines from users to HPC systems administrators, simplifies the set up of HPC pipelines, and makes pipelines as reliable as possible once set up. The work described here has software dependencies that are common to the majority of HPC clusters. We achieve this by linking Globus and cron to produce an open-source system that requires little administrative support but provides a very easy way of running an analysis on a HPC system. The user starts the analysis through the Globus website and, when started, the data will be encrypted, uploaded to the HPC, analysed, and returned to the originating machine, along with a record of the analysis.