Constraint-Driven Complexity-Aware Data Science Workflow for AutoBDA

Abstract

The Internet of Things, privacy, and technical constraints increase the demand for edge-based data-driven services, which is one of the major goals of Industry 4.0 and Society 5.0. Big data analysis (BDA) is the preferred approach to unleash hidden knowledge. However, BDA consumes excessive resources and time. These limitations hamper the meaningful adoption of BDA, especially the time and situation critical edge use cases, and hinder the goals of Industry 4.0 and Society 5.0. Automating the BDA process at the edge is a cognitive approach to address the aforementioned concerns. Data science workflow is an indispensable challenge for successful automation. Therefore, we conducted a systematic literature survey on data science workflow platforms as the first contribution. Moreover, we learned that the BDA workflow depends on diversified constraints and undergoes rigorous data-mining stages. These caused an increase in the solution space, dynamic constraints, complexity issues, and NP-hardness of BDA workflow. Graphplan is a heuristic AI-planning technique that can address concerns associated with BDA workflow. Therefore, as the second contribution, we adopted the graphplan to generate a workflow for edge-based BDA automation. Experiments demonstrate that the proposed method achieved our objectives.