Rapid data stream application development framework

Abstract

Developers of data stream processing application have to write a lot of codes even for a simple functionality, and, to make it worse, tend to rewrite their codes when developing different applications. These developers also need to recompile the code even for simple changes. In this paper, we present a configurable data stream application framework, which will help developers in developing data stream applications by reducing the amount of codes written to develop a data stream processing application. In this framework, we introduce a Domain Specific Language (DSL) for defining and configuring the data stream application. Our framework provides many basic stream processing functionalities, such as passing the data from data source to processing classes, filtering, windowing the data, as well as sending the data to data collectors. Most configurations related to these functionalities can be easily changed using the DSL without the need to recompile the code. The framework was tested using two case studies, where for each of them we developed data stream applications with and without the framework. The case studies show the increased productivity in terms of the number of lines of code and the number of files written for each respective application.