HER-PT: An intelligent penetration testing framework with Hindsight Experience Replay

Abstract

Penetration testing (PT) is an active method to evaluate the security of computer systems. With the continuous expansion of the scale of the network, the difficulty of penetration testing increases sharply, and at the same time, it relies heavily on expert experience. Therefore, AI-based techniques such as Deep Reinforcement Learning (DRL) will be an effective solution to automate penetration testing and reduce labor costs. However, in the existing DRL-based PT work, the attacker has a large number of low feedback behaviors, and it is difficult to collect enough successful experiences and positive learning rewards, that is, the sparse reward problem. In addition, existing works on automatic penetration based on MSF in real environments mainly focus on single-host scenarios and have not been extended to multi-host networks. In this paper, we propose a new intelligent PT framework “HER-PT” that integrates Hindsight Experience Replay (HER) techniques into DRL-based PT models in the hope of solving sparse reward problems in reinforcement learning and applying penetration testing to real multi-host scenarios. We constructed several network scenarios, trained HER-PT model agents in the cyber attack simulator Nasim for autonomous penetration testing experiments, and tried different reinforcement learning optimization schemes. Experimental results show that HER-PT can converge within 500 episodes in a medium scenario of 16 hosts, which is about 50% faster than other models. It can still maintain a success rate of 85.76% in the medium frequency dynamic change scene. The results show that HER-PT can effectively accelerate the training of the model and shorten the training period.