Harmonizing recurring patterns and non-recurring trends in traffic datasets for enhanced estimation of missing information

Abstract

Traffic datasets commonly comprise missing information due to sensor malfunctions, environmental conditions, security concerns, and technical/data quality issues. These challenges are inherent in real-world traffic data collection systems. Despite numerous imputation algorithms proposed in the literature, concerns persist about selecting a reliable algorithm that consistently performs well across diverse missing data scenarios. This is crucial for two main reasons. Firstly, real-world traffic datasets often exhibit a range of missing gaps with varying temporal durations, encompassing both short and long gaps within a single dataset. Secondly, in spatio-temporal traffic datasets, both recurring and non-recurring traffic conditions coexist. Since different data imputation principles reported in the literature suit either type of missing data (short or long gaps) and traffic conditions (recurring or non-recurring) better than others, algorithms often output sub-optimal estimates for network-wide datasets characterized by multiple types of missing gaps and traffic conditions.

To address the issue, this paper proposes a tensor decomposition algorithm named SINTD (Stochastic Informed Non-Negative Tensor Decomposition) and logically integrates it with a spline regression model in a novel data imputation framework called SPRINT (Spline-powered Informed Non-negative Tensor Decomposition). Where SINTD mines dominant patterns in the traffic datasets, effective in estimating missing gaps under recurring traffic conditions, integration of spline with tensor decomposition helps a) capturing the time-localized trends unaccounted by tensor decomposition, aiding in approximating better the non-recurring component of the traffic states, and b) complementing SINTD for improved mining of recurring patterns in the subsequent iterations of SPRINT. Although the two algorithms have distinct limitations when used separately, their harmonization allows us to effectively utilize their respective strengths and overcome individual limitations. This paper, through extensive experimentation on six traffic datasets and benchmarking against nine baseline algorithms, demonstrates the efficacy of SPRINT in consistently producing high-accuracy missing data estimates across five diverse missing data scenarios. These include a) experiments on datasets exhibiting a mix of short and long-duration missing gaps—mimicking the intricate missing data structure of real-world traffic datasets, and b) a Logan City (Australia) case study highlighting the imputation of missing data under potential non-recurring traffic conditions resulting from road incidents.