Special Issue on benchmarking, experimentation tools, and reproducible practices for data‐intensive systems from edge to cloud

As data analytics and machine learning increasingly permeate our cities, factories, and homes, the computing infrastructure for data-intensive systems becomes more challenging. That is, the vision of pervasive, intelligent, and cyber-physical IoT systems will not be realized with centralized cloud resources alone. Such resources are simply too far away from sensor-equipped devices and users, resulting in high latency, bandwidth bottlenecks, and unnecessary energy consumption. In addition, there are often privacy and security requirements that mandate distributed architectures. As a result, new distributed computing paradigms are emerging that promise to bring computing and storage closer to data sources and users. The emerging distributed computing environments of edge and fog computing provide additional resources within mobile networks, ISP infrastructure, and even LEO satellites. These diverse and dynamic computing environments pose significant challenges to the performance, dependability, and efficiency of data-intensive systems running on such infrastructure. At the same time, it is far less clear how to properly benchmark, evaluate, and test the behavior of systems that span IoT devices, edge nodes, and cloud resources. For example, IoT sensor data stream processing systems can be leveraged to continuously optimize the operation of urban infrastructures (such as public transportation systems, water networks, or medical infrastructures). The behavior of such systems must be thoroughly assessed before they can be deployed to edge and fog infrastructure. In addition, these systems must be evaluated reproducibly under the expected computing environment conditions, including variations of those conditions, given the inherently unsteady nature of IoT environments. In addition, there is growing concern about the energy consumption and greenhouse gas emissions of ICT (and especially distributed ML-based applications), which further warrants close examination of the behavior of new data-intensive applications. Despite significant research and development efforts to improve benchmarking, experimentation tools, and reproducible practices for data-intensive systems spanning from the edge to the cloud, more research is urgently needed. We therefore invited high-quality research papers on this topic for this special issue of Software: Practice and Experience, and we were able to select two out of four submissions for this special issue with the help of our reviewers. The first accepted paper is titled “faas-sim: A Trace-Driven Simulation Framework for Serverless Edge Computing Platforms”.1 It is co-authored by Philipp Raith, Thomas Rausch, Alireza Furutanpey, and Schahram Dustdar. The paper presents the design and implementation of a new simulation framework, “faas-sim,” for modeling and evaluating serverless software architectures spanning the edge-cloud continuum based on a scenario description, a given network topology, and workload traces. The new simulator is demonstrated by using it for performance estimation, resource planning, co-simulation, and scientific evaluation. The authors also evaluate faas-sim's network simulation and resource utilization. Furthermore, they highlight traces that come with faas-sim and provide an overview of published research that has used faas-sim. The second accepted paper is titled “Software-in-the-Loop Simulation for Developing and Testing Carbon-Aware Applications”.2 It is co-authored by Philipp Wiesner, Marvin Steinke, Henrik Nickel, Yazan Kitana, and Odej Kao. As an alternative to relying on purely simulated or purely real testbeds, the paper proposes the use of software-in-the-loop simulation and hybrid testbeds for testing carbon-aware software applications in the context of energy simulations. The paper describes the design and implementation of a prototype, “Vessim,” as well as two experiments demonstrating the capabilities and features of the novel tool. In this way, the paper shows how a message broker can reliably and realistically connect currently running applications under test to real-time simulations, while the energy demand is continuously measured or modeled. We are grateful to the Editor-in-Chief of the Journal, Dr. Rajkumar Buyya, for inviting us to organize this special issue. We are also grateful for the valuable support from the administrative office of the journal. In addition, we are very grateful for the thorough and thoughtful reviews provided by our reviewers. Finally, we appreciate the hard work and trust of the authors who submitted papers to our special issue.