Unsupervised Anomaly Detection in Spatio-Temporal Stream Network Sensor Data

IF 4.6 1区 地球科学 Q2 ENVIRONMENTAL SCIENCES Water Resources Research Pub Date : 2024-11-07 DOI:10.1029/2023wr035707
Edgar Santos-Fernandez, Jay M. Ver Hoef, Erin E. Peterson, James McGree, Cesar A. Villa, Catherine Leigh, Ryan Turner, Cameron Roberts, Kerrie Mengersen
{"title":"Unsupervised Anomaly Detection in Spatio-Temporal Stream Network Sensor Data","authors":"Edgar Santos-Fernandez, Jay M. Ver Hoef, Erin E. Peterson, James McGree, Cesar A. Villa, Catherine Leigh, Ryan Turner, Cameron Roberts, Kerrie Mengersen","doi":"10.1029/2023wr035707","DOIUrl":null,"url":null,"abstract":"The use of in-situ digital sensors for water quality monitoring is becoming increasingly common worldwide. While these sensors provide near real-time data for science, the data are prone to technical anomalies that can undermine the trustworthiness of the data and the accuracy of statistical inferences, particularly in spatial and temporal analyses. Here we propose a framework for detecting anomalies in sensor data recorded in stream networks, which takes advantage of spatial and temporal autocorrelation to improve detection rates. The proposed framework involves the implementation of effective data imputation to handle missing data, alignment of time-series to address temporal disparities, and the identification of water quality events. We explore the effectiveness of a suite of state-of-the-art statistical methods including posterior predictive distributions, finite mixtures, and Hidden Markov Models (HMM). We showcase the practical implementation of automated anomaly detection in near-real time by employing a Bayesian recursive approach. This demonstration is conducted through a comprehensive simulation study and a practical application to a substantive case study situated in the Herbert River, located in Queensland, Australia, which flows into the Great Barrier Reef. We found that methods such as posterior predictive distributions and HMM produce the best performance in detecting multiple types of anomalies. Utilizing data from multiple sensors deployed relatively near one another enhances the ability to distinguish between water quality events and technical anomalies, thereby significantly improving the accuracy of anomaly detection. Thus, uncertainty and biases in water quality reporting, interpretation, and modeling are reduced, and the effectiveness of subsequent management actions improved.","PeriodicalId":23799,"journal":{"name":"Water Resources Research","volume":null,"pages":null},"PeriodicalIF":4.6000,"publicationDate":"2024-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Water Resources Research","FirstCategoryId":"89","ListUrlMain":"https://doi.org/10.1029/2023wr035707","RegionNum":1,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENVIRONMENTAL SCIENCES","Score":null,"Total":0}
引用次数: 0

Abstract

The use of in-situ digital sensors for water quality monitoring is becoming increasingly common worldwide. While these sensors provide near real-time data for science, the data are prone to technical anomalies that can undermine the trustworthiness of the data and the accuracy of statistical inferences, particularly in spatial and temporal analyses. Here we propose a framework for detecting anomalies in sensor data recorded in stream networks, which takes advantage of spatial and temporal autocorrelation to improve detection rates. The proposed framework involves the implementation of effective data imputation to handle missing data, alignment of time-series to address temporal disparities, and the identification of water quality events. We explore the effectiveness of a suite of state-of-the-art statistical methods including posterior predictive distributions, finite mixtures, and Hidden Markov Models (HMM). We showcase the practical implementation of automated anomaly detection in near-real time by employing a Bayesian recursive approach. This demonstration is conducted through a comprehensive simulation study and a practical application to a substantive case study situated in the Herbert River, located in Queensland, Australia, which flows into the Great Barrier Reef. We found that methods such as posterior predictive distributions and HMM produce the best performance in detecting multiple types of anomalies. Utilizing data from multiple sensors deployed relatively near one another enhances the ability to distinguish between water quality events and technical anomalies, thereby significantly improving the accuracy of anomaly detection. Thus, uncertainty and biases in water quality reporting, interpretation, and modeling are reduced, and the effectiveness of subsequent management actions improved.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
时空流网络传感器数据中的无监督异常检测
在全球范围内,使用原位数字传感器进行水质监测正变得越来越普遍。虽然这些传感器能为科学研究提供近乎实时的数据,但这些数据容易出现技术异常,从而影响数据的可信度和统计推断的准确性,尤其是在空间和时间分析中。在此,我们提出了一个用于检测流网络中记录的传感器数据异常的框架,该框架利用空间和时间自相关性来提高检测率。建议的框架包括实施有效的数据估算以处理缺失数据、调整时间序列以解决时间差异问题,以及识别水质事件。我们探索了一系列最先进统计方法的有效性,包括后验预测分布、有限混合物和隐马尔可夫模型(HMM)。我们展示了通过采用贝叶斯递归方法在近乎实时的时间内实现自动异常检测的实际应用。我们通过全面的模拟研究和实际应用,对位于澳大利亚昆士兰州流入大堡礁的赫伯特河进行了案例研究。我们发现,后验预测分布和 HMM 等方法在检测多种类型的异常情况方面性能最佳。利用相对就近部署的多个传感器的数据,可以增强区分水质事件和技术异常的能力,从而显著提高异常检测的准确性。这样,水质报告、解释和建模中的不确定性和偏差就会减少,后续管理行动的有效性也会提高。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Water Resources Research
Water Resources Research 环境科学-湖沼学
CiteScore
8.80
自引率
13.00%
发文量
599
审稿时长
3.5 months
期刊介绍: Water Resources Research (WRR) is an interdisciplinary journal that focuses on hydrology and water resources. It publishes original research in the natural and social sciences of water. It emphasizes the role of water in the Earth system, including physical, chemical, biological, and ecological processes in water resources research and management, including social, policy, and public health implications. It encompasses observational, experimental, theoretical, analytical, numerical, and data-driven approaches that advance the science of water and its management. Submissions are evaluated for their novelty, accuracy, significance, and broader implications of the findings.
期刊最新文献
Efficient Model Calibration Using Submodels Unsteady Secondary Flow Structure at a Large River Confluence Can Satellite or Reanalysis Precipitation Products Depict the Location and Intensity of Rainfall at Flash Flood Scale Over the Eastern Mountainous Area of the Tibetan Plateau? Estimation of Recovery Efficiency in High-Temperature Aquifer Thermal Energy Storage Considering Buoyancy Flow A Spatially-Distributed Machine Learning Approach for Fractional Snow Covered Area Estimation
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1