A distributed architecture for rule engine to deal with big data

Abstract

Rule engine, which acknowledges facts and draws conclusions by repeatedly matching facts with rules, is a good way of knowledge representation and inference. However, because of its low computational efficiency and the limitation of single machine's capacity, it cannot deal well with big data. As traditional MapReduce architecture can only address this problem in certain conditions, we have made some improvements and therefore proposed a distributed implementation of the rule engine using MapReduce-based architecture. It is designed to deal with a large amount of data in a parallel and distributed way by using a computing cluster that consists of multiple machines, on which certain part of the Rete algorithm would be operated. In the phase of splitting rules and the Rete-net, Apriori algorithm is also improved and adopted so as to gain a better system performance. This paper not only describes details of the design and its implementation, but also shows its high performance through several experiments.