DeepQC: A deep learning system for automatic quality control of in-situ soil moisture sensor time series data

Abstract

Amidst changing climate, real-time soil moisture monitoring is vital for the development of in-season decision support tools to help farmers manage weather-related risks in agriculture. Precision Sustainable Agriculture (PSA) recently established a real-time soil moisture monitoring network across the central, Midwest, and eastern U.S., but continuous field-scale sensor observations often come with data gaps and anomalies. To maintain the data quality and continuity needed for developing decision tools, a quality control system is necessary.

The International Soil Moisture Network (ISMN) introduced the Flagit module for anomaly detection in soil moisture time series observations. However, under certain conditions, Flagit's threshold and spectral based quality control approaches may underperform in identifying anomalies. Recently, deep learning methods have been successfully applied to detect time series anomalies in time series data in various disciplines. However, their use in agriculture for anomaly detection in time series datasets has not been yet investigated. This study focuses on developing a Bi-directional Long Short-Term Memory (LSTM) model, referred to as DeepQC, to identify anomalies in soil moisture time series data. Manual flagged PSA observations were used for training, validation, and testing the model, following an 80:10:10 split. The study then compared the DeepQC and Flagit-based estimates to assess their relative performance.

Flagit correctly flagged 95.8 % of the correct observations and 50.3 % of the anomaly observations, indicating its limitations in identifying anomalies, particularly at sites consists of more than 30 % anomalies. On the other hand, the DeepQC correctly flagged 89.8 % of the correct observations and 99.5 % of the anomalies, with overall accuracy of 95.6 %, in significantly less time, demonstrating its superiority over Flagit approach. Importantly, the performance of the DeepQC remained consistent regardless of the number of anomalies in site observations. Given the promising results obtained with the DeepQC, future studies will focus on implementing and finetuning this model on national and global soil moisture networks.