Intelligently-Automated Facilities Expansion with the HEPCloud Decision Engine

Abstract

The next generation of High Energy Physics experiments are expected to generate exabytes of data—two orders of magnitude greater than the current generation. In order to reliably meet peak demands, facilities must either plan to provision enough resources to cover the forecasted need, or find ways to elastically expand their computational capabilities. Commercial cloud and allocation-based High Performance Computing (HPC) resources both have explicit and implicit costs that must be considered when deciding when to provision these resources, and to choose an appropriate scale. In order to support such provisioning in a manner consistent with organizational business rules and budget constraints, we have developed a modular intelligent decision support system (IDSS) to aid in the automatic provisioning of resources—spanning multiple cloud providers, multiple HPC centers, and grid computing federations.