Dear Editor, This letter is concerned with the secure tracking control problem in the unmanned aerial vehicle (UAV) system by fixed-time convergent reinforcement learning (RL). By virtue of the zero-sum game, the false data injection (FDI) attacker and secure controller are viewed as game players. Then, the attack-defense process is recast as a min-max problem. For solving the problem and acquiring the optimal secure control policy, a single-critic RL algorithm with fixed-time convergence is presented. Meanwhile, the associated convergence and stability proofs are given. A simulation is provided to show the effectiveness of the raised method.
{"title":"Secure Tracking Control via Fixed-Time Convergent Reinforcement Learning for a UAV CPS","authors":"Zhenyu Gong;Feisheng Yang","doi":"10.1109/JAS.2023.124149","DOIUrl":"https://doi.org/10.1109/JAS.2023.124149","url":null,"abstract":"Dear Editor, This letter is concerned with the secure tracking control problem in the unmanned aerial vehicle (UAV) system by fixed-time convergent reinforcement learning (RL). By virtue of the zero-sum game, the false data injection (FDI) attacker and secure controller are viewed as game players. Then, the attack-defense process is recast as a min-max problem. For solving the problem and acquiring the optimal secure control policy, a single-critic RL algorithm with fixed-time convergence is presented. Meanwhile, the associated convergence and stability proofs are given. A simulation is provided to show the effectiveness of the raised method.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10555236","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315268","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kun Jiang;Wenzhang Liu;Yuanda Wang;Lu Dong;Changyin Sun
Efficient exploration in complex coordination tasks has been considered a challenging problem in multi-agent reinforcement learning (MARL). It is significantly more difficult for those tasks with latent variables that agents cannot directly observe. However, most of the existing latent variable discovery methods lack a clear representation of latent variables and an effective evaluation of the influence of latent variables on the agent. In this paper, we propose a new MARL algorithm based on the soft actor-critic method for complex continuous control tasks with confounders. It is called the multi-agent soft actor-critic with latent variable (MASAC-LV) algorithm, which uses variational inference theory to infer the compact latent variables representation space from a large amount of offline experience. Besides, we derive the counterfactual policy whose input has no latent variables and quantify the difference between the actual policy and the counterfactual policy via a distance function. This quantified difference is considered an intrinsic motivation that gives additional rewards based on how much the latent variable affects each agent. The proposed algorithm is evaluated on two collaboration tasks with confounders, and the experimental results demonstrate the effectiveness of MASAC-LV compared to other baseline algorithms.
{"title":"Discovering Latent Variables for the Tasks With Confounders in Multi-Agent Reinforcement Learning","authors":"Kun Jiang;Wenzhang Liu;Yuanda Wang;Lu Dong;Changyin Sun","doi":"10.1109/JAS.2024.124281","DOIUrl":"https://doi.org/10.1109/JAS.2024.124281","url":null,"abstract":"Efficient exploration in complex coordination tasks has been considered a challenging problem in multi-agent reinforcement learning (MARL). It is significantly more difficult for those tasks with latent variables that agents cannot directly observe. However, most of the existing latent variable discovery methods lack a clear representation of latent variables and an effective evaluation of the influence of latent variables on the agent. In this paper, we propose a new MARL algorithm based on the soft actor-critic method for complex continuous control tasks with confounders. It is called the multi-agent soft actor-critic with latent variable (MASAC-LV) algorithm, which uses variational inference theory to infer the compact latent variables representation space from a large amount of offline experience. Besides, we derive the counterfactual policy whose input has no latent variables and quantify the difference between the actual policy and the counterfactual policy via a distance function. This quantified difference is considered an intrinsic motivation that gives additional rewards based on how much the latent variable affects each agent. The proposed algorithm is evaluated on two collaboration tasks with confounders, and the experimental results demonstrate the effectiveness of MASAC-LV compared to other baseline algorithms.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315240","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, the optimal variational generalized Nash equilibrium (v-GNE) seeking problem in merely monotone games with linearly coupled cost functions is investigated, in which the feasible strategy domain of each agent is coupled through an affine constraint. A distributed algorithm based on the hybrid steepest descent method is first proposed to seek the optimal v-GNE. Then, an accelerated algorithm with relaxation is proposed and analyzed, which has the potential to further improve the convergence speed to the optimal v-GNE. Some sufficient conditions in both algorithms are obtained to ensure the global convergence towards the optimal v-GNE. To illustrate the performance of the algorithms, numerical simulation is conducted based on a networked Nash-Cournot game with bounded market capacities.
{"title":"Distributed Optimal Variational GNE Seeking in Merely Monotone Games","authors":"Wangli He;Yanzhen Wang","doi":"10.1109/JAS.2024.124284","DOIUrl":"https://doi.org/10.1109/JAS.2024.124284","url":null,"abstract":"In this paper, the optimal variational generalized Nash equilibrium (v-GNE) seeking problem in merely monotone games with linearly coupled cost functions is investigated, in which the feasible strategy domain of each agent is coupled through an affine constraint. A distributed algorithm based on the hybrid steepest descent method is first proposed to seek the optimal v-GNE. Then, an accelerated algorithm with relaxation is proposed and analyzed, which has the potential to further improve the convergence speed to the optimal v-GNE. Some sufficient conditions in both algorithms are obtained to ensure the global convergence towards the optimal v-GNE. To illustrate the performance of the algorithms, numerical simulation is conducted based on a networked Nash-Cournot game with bounded market capacities.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315272","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Set stabilization is one of the essential problems in engineering systems, and self-triggered control (STC) can save the storage space for interactive information, and can be successfully applied in networked control systems with limited communication resources. In this study, the set stabilization problem and STC design of Boolean control networks are investigated via the semi-tensor product technique. On the one hand, the largest control invariant subset is calculated in terms of the strongly connected components of the state transition graph, by which a graph-theoretical condition for set stabilization is derived. On the other hand, a characteristic function is exploited to determine the triggering mechanism and feasible controls. Based on this, the minimum-time and minimum-triggering open-loop, state-feedback and output-feedback STCs for set stabilization are designed, respectively. As classic applications of self-triggered set stabilization, self-triggered synchronization, self-triggered output tracking and self-triggered output regulation are discussed as well. Additionally, several practical examples are given to illustrate the effectiveness of theoretical results.
{"title":"Self-Triggered Set Stabilization of Boolean Control Networks and Its Applications","authors":"Rong Zhao;Jun-e Feng;Dawei Zhang","doi":"10.1109/JAS.2023.124050","DOIUrl":"https://doi.org/10.1109/JAS.2023.124050","url":null,"abstract":"Set stabilization is one of the essential problems in engineering systems, and self-triggered control (STC) can save the storage space for interactive information, and can be successfully applied in networked control systems with limited communication resources. In this study, the set stabilization problem and STC design of Boolean control networks are investigated via the semi-tensor product technique. On the one hand, the largest control invariant subset is calculated in terms of the strongly connected components of the state transition graph, by which a graph-theoretical condition for set stabilization is derived. On the other hand, a characteristic function is exploited to determine the triggering mechanism and feasible controls. Based on this, the minimum-time and minimum-triggering open-loop, state-feedback and output-feedback STCs for set stabilization are designed, respectively. As classic applications of self-triggered set stabilization, self-triggered synchronization, self-triggered output tracking and self-triggered output regulation are discussed as well. Additionally, several practical examples are given to illustrate the effectiveness of theoretical results.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315208","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dear Editor, This letter develops a novel method to implement event-triggered optimal control (ETOC) for discrete-time nonlinear systems using parallel control and deep reinforcement learning (DRL), referred to as Deep-ETOC. The developed Deep-ETOC method introduces the communication cost into the performance index through parallel control, so that the developed method enables control systems to learn ETOC policies directly without triggering conditions. Then, dueling double deep Q-network (D3QN) is utilized to achieve our method. In simulations, we present a preliminary comparative study of DRL and Lyapunov analysis for ETOC.
{"title":"Deep Reinforcement Learning or Lyapunov Analysis? A Preliminary Comparative Study on Event-Triggered Optimal Control","authors":"Jingwei Lu;Lefei Li;Qinglai Wei;Fei–Yue Wang","doi":"10.1109/JAS.2024.124434","DOIUrl":"https://doi.org/10.1109/JAS.2024.124434","url":null,"abstract":"Dear Editor, This letter develops a novel method to implement event-triggered optimal control (ETOC) for discrete-time nonlinear systems using parallel control and deep reinforcement learning (DRL), referred to as Deep-ETOC. The developed Deep-ETOC method introduces the communication cost into the performance index through parallel control, so that the developed method enables control systems to learn ETOC policies directly without triggering conditions. Then, dueling double deep Q-network (D3QN) is utilized to achieve our method. In simulations, we present a preliminary comparative study of DRL and Lyapunov analysis for ETOC.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10555241","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315209","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
For the first time, this article introduces a LiDAR Point Clouds Dataset of Ships composed of both collected and simulated data to address the scarcity of LiDAR data in maritime applications. The collected data are acquired using specialized maritime LiDAR sensors in both inland waterways and wide-open ocean environments. The simulated data is generated by placing a ship in the LiDAR coordinate system and scanning it with a redeveloped Blensor that emulates the operation of a LiDAR sensor equipped with various laser beams. Furthermore, we also render point clouds for foggy and rainy weather conditions. To describe a realistic shipping environment, a dynamic tail wave is modeled by iterating the wave elevation of each point in a time series. Finally, networks serving small objects are migrated to ship applications by feeding our dataset. The positive effect of simulated data is described in object detection experiments, and the negative impact of tail waves as noise is verified in single-object tracking experiments. The Dataset is available at https://github.com/zqy411470859/ship_dataset.
{"title":"A LiDAR Point Clouds Dataset of Ships in a Maritime Environment","authors":"Qiuyu Zhang;Lipeng Wang;Hao Meng;Wen Zhang;Genghua Huang","doi":"10.1109/JAS.2024.124275","DOIUrl":"https://doi.org/10.1109/JAS.2024.124275","url":null,"abstract":"For the first time, this article introduces a LiDAR Point Clouds Dataset of Ships composed of both collected and simulated data to address the scarcity of LiDAR data in maritime applications. The collected data are acquired using specialized maritime LiDAR sensors in both inland waterways and wide-open ocean environments. The simulated data is generated by placing a ship in the LiDAR coordinate system and scanning it with a redeveloped Blensor that emulates the operation of a LiDAR sensor equipped with various laser beams. Furthermore, we also render point clouds for foggy and rainy weather conditions. To describe a realistic shipping environment, a dynamic tail wave is modeled by iterating the wave elevation of each point in a time series. Finally, networks serving small objects are migrated to ship applications by feeding our dataset. The positive effect of simulated data is described in object detection experiments, and the negative impact of tail waves as noise is verified in single-object tracking experiments. The Dataset is available at https://github.com/zqy411470859/ship_dataset.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315259","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, a model predictive control (MPC) framework is proposed for finite-time stabilization of linear and nonlinear discrete-time systems subject to state and control constraints. The proposed MPC framework guarantees the finite-time convergence property by assigning the control horizon equal to the dimension of the overall system, and only penalizing the terminal cost in the optimization, where the stage costs are not penalized explicitly. A terminal inequality constraint is added to guarantee the feasibility and stability of the closed-loop system. Initial feasibility can be improved via augmentation. The finite-time convergence of the proposed MPC is proved theoretically, and is supported by simulation examples.
{"title":"Finite-Time Stabilization for Constrained Discrete-time Systems by Using Model Predictive Control","authors":"Bing Zhu;Xiaozhuoer Yuan;Li Dai;Zhiwen Qiang","doi":"10.1109/JAS.2024.124212","DOIUrl":"https://doi.org/10.1109/JAS.2024.124212","url":null,"abstract":"In this paper, a model predictive control (MPC) framework is proposed for finite-time stabilization of linear and nonlinear discrete-time systems subject to state and control constraints. The proposed MPC framework guarantees the finite-time convergence property by assigning the control horizon equal to the dimension of the overall system, and only penalizing the terminal cost in the optimization, where the stage costs are not penalized explicitly. A terminal inequality constraint is added to guarantee the feasibility and stability of the closed-loop system. Initial feasibility can be improved via augmentation. The finite-time convergence of the proposed MPC is proved theoretically, and is supported by simulation examples.","PeriodicalId":54230,"journal":{"name":"Ieee-Caa Journal of Automatica Sinica","volume":null,"pages":null},"PeriodicalIF":11.8,"publicationDate":"2024-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315160","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper revisits the problem of bumpless transfer control (BTC) for discrete-time nondeterministic switched linear systems. The general case of asynchronous switching is considered for the first time in the field of BTC for switched systems. A new approach called interpolated bumpless transfer control (IBTC) is proposed, where the bumpless transfer controllers are formulated with the combination of the two adjacent mode-dependent controller gains, and are interpolated for finite steps once the switching is detected. In contrast with the existing approaches, IBTC does not necessarily run through the full interval of subsystems, as well as possesses the time-varying controller gains (with more flexibility and less conservatism) achieved from a control synthesis allowing for the stability and other performance of the whole switched system. Sufficient conditions ensuring stability and $H_{infty}$