Efficient parameter learning for Bayesian Network classifiers following the Apache Spark Dataframes paradigm

Abstract

Every year the volume of information is growing at a high rate; therefore, more modern approaches are required to deal with such issues efficiently. Distributed systems, such as Apache Spark, offer such a modern approach, resulting in more and more machine learning models, being adapted into using distributed logic. In this paper, we propose a classification model, based on Bayesian Networks (BNs), that utilizes the distributed environment of Apache Spark using the Dataframes paradigm. This model can exploit any user-provided directed acyclic graph (DAG) that portrays the dependencies between the features of a dataset to estimate the parameters of the conditional probability distributions associated with each node in the graph to make accurate predictions. Moreover, in contrast with the majority of implementations that are only able to handle discrete features, it is also capable of efficiently handling continuous features by calculating the Gaussian probability density function.