SCAL-E: An Auto Scaling Agent for Optimum Big Data Load Balancing in Kubernetes Environments

During the past years, the issue of effectively balancing incoming big data streams has been under serious research. It still allows for new solutions, even if load balancing is already being addressed by multiple frameworks. This paper proposes a smart agent, named “SCAL-E“ that achieves balancing of big data loads and lives within the Kubernetes Environment. SCAL-E takes advantage of MongoDB’s scaling, replicating & sharding capabilities and decides when to increase or decrease its repository’s sub-components, based on the incoming load. This way, SCAL-E assures of proper resource allocation and gives efficiency to the jobs of big data storing & forwarding.