Consistent Stream Processing: Doctoral Symposium

Abstract

Stream Processors (SPs) continuously transform huge volumes of input streams with a computational model that is inherently distributed, scalable, and fault-tolerant. For these reasons they are used in application environments in which almost real-time computation is of paramount importance, such as stock option analysis, fraud detection systems, monitoring, and real-time data analytics for web applications. In many applicative domains, SPs are used in conjunction with data management systems such as transactional databases and data warehouses that store intermediate or final results produced by the SPs. However, SPs have no control on the consistency guarantees of the results produced on external components. We propose a novel approach that we name consistent stream processing that integrates the external state of databases within the SP and enforces consistency guarantees both on state updates and on external querying. We extend the computational model of SPs with transactions and we provide two possible strategies to enforce their transactional properties.