SWEEP: Accelerating Scientific Research Through Scalable Serverless Workflows

Scientific and commercial applications are increasingly being executed in the cloud, but the difficulties associated with cluster management render on-demand resources inaccessible or inefficient to many users. Recently, the serverless execution model, in which the provisioning of resources is abstracted from the user, has gained prominence as an alternative to traditional cyberinfrastructure solutions. With its inherent elasticity, the serverless paradigm constitutes a promising computational model for scientific workflows, allowing domain specialists to develop and deploy workflows that are subject to varying workloads and intermittent usage without the overhead of infrastructure maintenance. We present the Serverless Workflow Enablement and Execution Platform (SWEEP), a cloud-agnostic workflow management system with a purely serverless execution model that allows users to define, run and monitor generic cloud-native workflows. We demonstrate the use of SWEEP on workflows from two disparate scientific domains and present an evaluation of performance and scaling.