MicroStream: A Distributed In-memory Caching Service For Data Production

Abstract

Data-driven innovation and optimization have become an important direction for the intelligent transformation of enterprises. Data processing tasks have been developed and orchestrated to extract data insights, creating direct or indirect data dependencies between tasks or between tasks and the presentation layer. Traditional ETL (Extract-Transformation-Load) solutions share data through persistent storage, which has certain performance bottlenecks in hybrid cloud and multisource data scenarios. In this paper, we propose MicroStream, a distributed data virtualization and caching middleware service. MicroStream shields the direct access of ETL tasks to the storage layer and converts batch access to the source database into microstream access. ETL jobs share data through the distributed in-memory caching of MicroStream. In resource-constrained scenarios, such a solution significantly improves the performance of data transformation while reducing the extra load that the transformation jobs imply on the source persistent layer. We present a detailed performance evaluation of MicroStream and show that its performance compares favorably with traditional database-oriented solutions.