Game-Theoretic Neyman-Pearson Detection to Combat Strategic Evasion

The security in networked systems depends greatly on recognizing and identifying adversarial behaviors. Traditional detection methods target specific categories of attacks and have become inadequate against increasingly stealthy and deceptive attacks that are designed to bypass detection strategically. This work proposes game-theoretical frameworks to recognize and combat such evasive attacks. We focus on extending a fundamental class of statistical-based detection methods based on Neyman-Pearson’s (NP) hypothesis testing formulation. We capture the conflicting relationship between a strategic evasive attacker and an evasion-aware NP detector. By analyzing both the equilibrium behaviors of the attacker and the NP detector, we characterize their performance using Equilibrium Receiver-Operational-Characteristic (EROC) curves. We show that the evasion-aware NP detectors outperform the non-strategic ones by allowing them to take advantage of the attacker’s messages to adaptively modify their decision rules to enhance their success rate in detecting anomalies. In addition, we extend our framework to a sequential setting where the user sends out identically distributed messages. We corroborate the analytical results with a case study of an intrusion detection evasion problem.