Efficient, problem tailored big data processing using framework delegation

Abstract

The rise of the Internet of Things, social networking, and embedded connectivity has led to an explosion of available data. In order to better analyze this big data, many different tools have been created that can process the data efficiently. However, the increase in the amount of tools available makes it more difficult to determine which one will provide the most efficient solution to a given big data problem. In this paper, we present a delegation system that takes various frameworks and problem parameters as input and computes the best framework to use for a specific big data problem. To evaluate our system, we used two big data processing frameworks, namely, Hadoop MapReduce and AJIRA, with problem size as an input parameter. Preliminary results show that the system is able to select the most optimal big data processing framework for a given problem 90% of the time. Moreover, the proposed delegation system introduces only an additional 1% overhead when compared to the individual framework in terms of execution time.