An Online Approach to Determine Correlation between Data Streams

Abstract

Real time stream processing demands processed outcomes in minimal latency. Massive streams are generated in real time where linear relationship is determined using correlation. Existing approaches are used for correlating static data sets such as, Kandell, Pearson, Spearman etc. These approaches are insufficient to solve noise free online correlation. In this paper, we propose an online ordinal correlation approach having functionalities such as single pass, avoiding recalculation from scratch, removing outliers, and low memory requirements. In this approach, Compare Reduce Aggregate (CRA) algorithm is used for determining association between two feature vectors in real time using single scanning technique. Time and space complexities in CRA algorithm are measured as O(n) and O(1), respectively. This algorithm is used for reducing noise or error in a stream and used as a replacement of rank based correlation. It is recommended to have distinct elements and less variability in the streams for gaining maximum performance of this algorithm.