{"title":"强化学习在污水处理过程控制优化中的应用:方法、挑战和前进道路","authors":"Henry C. Croll, Kaoru Ikuma, S. Ong, S. Sarkar","doi":"10.1080/10643389.2023.2183699","DOIUrl":null,"url":null,"abstract":"Abstract Wastewater treatment process control optimization is a complex task in a highly nonlinear environment. Reinforcement learning (RL) is a machine learning technique that stands out for its ability to perform better than human operators for certain high-dimensional, complex decision-making problems, making it an ideal candidate for wastewater treatment process control optimization. However, while RL control optimization strategies have shown potential to provide operational cost savings and effluent quality improvements, RL has proven slow to be adopted among environmental engineers. This review provides an overview of existing RL applications for wastewater treatment control optimization found in literature and evaluates five key challenges that must be addressed prior to widespread adoption: practical RL implementation, managing data, integrating existing process models, building trust in empirical control strategies, and bridging gaps in professional training. Finally, this review discusses potential paths forward to addressing each key challenge, including leveraging soft sensing to improve online data collection, working with process engineers to integrate RL programming with existing industry software, utilizing supervised training to build expert knowledge into the RL agent, and focusing research efforts on known scenarios such as the Benchmark Simulation Model No. 1 to build a robust database of RL agent control optimization results. GRAPHICAL ABSTRACT","PeriodicalId":10823,"journal":{"name":"Critical Reviews in Environmental Science and Technology","volume":"53 1","pages":"1775 - 1794"},"PeriodicalIF":11.4000,"publicationDate":"2023-03-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"4","resultStr":"{\"title\":\"Reinforcement learning applied to wastewater treatment process control optimization: Approaches, challenges, and path forward\",\"authors\":\"Henry C. Croll, Kaoru Ikuma, S. Ong, S. Sarkar\",\"doi\":\"10.1080/10643389.2023.2183699\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Abstract Wastewater treatment process control optimization is a complex task in a highly nonlinear environment. Reinforcement learning (RL) is a machine learning technique that stands out for its ability to perform better than human operators for certain high-dimensional, complex decision-making problems, making it an ideal candidate for wastewater treatment process control optimization. However, while RL control optimization strategies have shown potential to provide operational cost savings and effluent quality improvements, RL has proven slow to be adopted among environmental engineers. This review provides an overview of existing RL applications for wastewater treatment control optimization found in literature and evaluates five key challenges that must be addressed prior to widespread adoption: practical RL implementation, managing data, integrating existing process models, building trust in empirical control strategies, and bridging gaps in professional training. Finally, this review discusses potential paths forward to addressing each key challenge, including leveraging soft sensing to improve online data collection, working with process engineers to integrate RL programming with existing industry software, utilizing supervised training to build expert knowledge into the RL agent, and focusing research efforts on known scenarios such as the Benchmark Simulation Model No. 1 to build a robust database of RL agent control optimization results. GRAPHICAL ABSTRACT\",\"PeriodicalId\":10823,\"journal\":{\"name\":\"Critical Reviews in Environmental Science and Technology\",\"volume\":\"53 1\",\"pages\":\"1775 - 1794\"},\"PeriodicalIF\":11.4000,\"publicationDate\":\"2023-03-06\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"4\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Critical Reviews in Environmental Science and Technology\",\"FirstCategoryId\":\"93\",\"ListUrlMain\":\"https://doi.org/10.1080/10643389.2023.2183699\",\"RegionNum\":1,\"RegionCategory\":\"环境科学与生态学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"ENVIRONMENTAL SCIENCES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Critical Reviews in Environmental Science and Technology","FirstCategoryId":"93","ListUrlMain":"https://doi.org/10.1080/10643389.2023.2183699","RegionNum":1,"RegionCategory":"环境科学与生态学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENVIRONMENTAL SCIENCES","Score":null,"Total":0}
引用次数: 4
摘要
摘要污水处理过程控制优化是一个高度非线性环境下的复杂任务。强化学习(RL)是一种机器学习技术,它能够在某些高维、复杂的决策问题上比人类操作员表现得更好,是废水处理过程控制优化的理想候选者。然而,尽管RL控制优化策略已显示出节省运营成本和改善出水质量的潜力,但事实证明,RL在环境工程师中的应用进展缓慢。这篇综述概述了文献中现有的RL在废水处理控制优化中的应用,并评估了在广泛采用之前必须解决的五个关键挑战:实际的RL实施、数据管理、集成现有的过程模型、在经验控制策略中建立信任,以及弥合专业培训中的差距。最后,这篇综述讨论了解决每一个关键挑战的潜在途径,包括利用软测量改进在线数据收集,与流程工程师合作将RL编程与现有行业软件集成,利用监督培训将专家知识构建到RL代理中,并将研究重点放在已知场景上,如Benchmark Simulation Model No.1,以建立RL agent控制优化结果的鲁棒数据库。图形摘要
Reinforcement learning applied to wastewater treatment process control optimization: Approaches, challenges, and path forward
Abstract Wastewater treatment process control optimization is a complex task in a highly nonlinear environment. Reinforcement learning (RL) is a machine learning technique that stands out for its ability to perform better than human operators for certain high-dimensional, complex decision-making problems, making it an ideal candidate for wastewater treatment process control optimization. However, while RL control optimization strategies have shown potential to provide operational cost savings and effluent quality improvements, RL has proven slow to be adopted among environmental engineers. This review provides an overview of existing RL applications for wastewater treatment control optimization found in literature and evaluates five key challenges that must be addressed prior to widespread adoption: practical RL implementation, managing data, integrating existing process models, building trust in empirical control strategies, and bridging gaps in professional training. Finally, this review discusses potential paths forward to addressing each key challenge, including leveraging soft sensing to improve online data collection, working with process engineers to integrate RL programming with existing industry software, utilizing supervised training to build expert knowledge into the RL agent, and focusing research efforts on known scenarios such as the Benchmark Simulation Model No. 1 to build a robust database of RL agent control optimization results. GRAPHICAL ABSTRACT
期刊介绍:
Two of the most pressing global challenges of our era involve understanding and addressing the multitude of environmental problems we face. In order to tackle them effectively, it is essential to devise logical strategies and methods for their control. Critical Reviews in Environmental Science and Technology serves as a valuable international platform for the comprehensive assessment of current knowledge across a wide range of environmental science topics.
Environmental science is a field that encompasses the intricate and fluid interactions between various scientific disciplines. These include earth and agricultural sciences, chemistry, biology, medicine, and engineering. Furthermore, new disciplines such as environmental toxicology and risk assessment have emerged in response to the increasing complexity of environmental challenges.
The purpose of Critical Reviews in Environmental Science and Technology is to provide a space for critical analysis and evaluation of existing knowledge in environmental science. By doing so, it encourages the advancement of our understanding and the development of effective solutions. This journal plays a crucial role in fostering international cooperation and collaboration in addressing the pressing environmental issues of our time.