Optimization of Big Data Mining Algorithm Based on Spark Framework: Preparation of Camera-Ready Contributions to SCITEPRESS Proceedings

Abstract

Abstract: Frequent itemsets mining is the core of association rule mining data. However, with the continuous increase of data, the traditional Apriori algorithm cannot meet people's daily needs, and the algorithm efficiency is low. This paper proposes the Eclat algorithm based on the Spark framework. In view of the shortcomings of serial algorithm in processing big data, it is modified. Using the vertical structure to avoid repetitive traversal of large amounts of data, while computing based on memory can greatly reduce I/O load and reduce computing time. Combined with the pruning strategy, the calculation of irrelevant itemsets is reduced, and the parallel computing capability of the algorithm is improved. The experimental results show that the efficiency of the Eclat algorithm based on the Spark framework is far better than that of the Eclat algorithm, and it has high efficiency and good scalability when processing massive data.