Structure-Aware Reinforcement Learning for Optimal Transmission Scheduling Over Packet Length-Dependent Lossy Networks

Abstract

The optimal transmission scheduling over packet length-dependent lossy networks is investigated. A set of sensors observe some linear time-invariant systems, and obtain their local state estimates using Kalman filters independently. The local state estimates are encoded in a data packet, which is sent to a remote estimator over packet length-dependent lossy networks, i.e., the packet arrival probability is exponentially decreasing with the packet length. The tradeoff arises between the packet arrival probability and the data size. To improve the remote estimation performance, one needs to design a transmission scheduling policy to determine how many as well as which sensors transmit data at each time instant, which is formulated here as a Markov decision process (MDP) model. There exists an optimal stationary policy for the MDP model, which is verified to possess a threshold structure. Based on this, a structure-aware reinforcement learning algorithm is proposed to approximate the MDP's optimal policy. Some simulation examples are given to verify the optimal policy's structure, and illustrate the performance of the proposed structure-aware reinforcement learning algorithm.