Risk-Aware Reinforcement Learning-Based Federated Learning for IoV Systems

Abstract

Federated learning (FL) that improves data privacy reduces the computational overhead for Internet of Vehicles (IoV) systems but has difficulty in defending against selfish attacks due to the restricted quality of service requirements and the high mobility of vehicles. In this paper, we design a risk-aware hierarchical reinforcement learning-based FL framework for IoV to resist selfish attacks. By designing a two-level hierarchical policy selection module that consists of two deep neural networks, this framework divides the training policy into two sub-policies, i.e., the selection of FL participants and the corresponding local training data size, which are chosen based on the previous training performance and vehicle participation performance. This framework designs a risk-aware safety guide to avoid dangerous states such as local task failure resulting from risky training policies. Specifically, the guide uses a warning signal to evaluate the short-term risk of each state-action pair, applies an R-network to estimate the long-term risks for modifying the chosen training policy, and designs a punishment function for the modified training policy to revise the immediate reward to further enhance the safe exploration. We analyze the convergence performance and computational complexity of our scheme. Experimental results on MNIST, CIFAR-10, and Stanford Cars datasets verify the effectiveness of our scheme, including the global model accuracy, training latency, detection success rate, and convergence speed compared with the benchmarks FedAvg, MFL, DQNPS, and SHRL.