Early mitigation of CPU-optimized ransomware using monitoring encryption instructions

Ransomware attacks pose a significant threat to information systems. Server hosts, including cloud infrastructure as a service, are prime targets for ransomware developers. To address this, security mechanisms, such as antivirus software, have proven effective. Moreover, research on ransomware detection advocates for behavior-based finding mechanisms while ransomware is in operation. In response to evolving detections, ransomware developers are now adapting an optimized design tailored for CPU architecture (CPU-optimized ransomware). This variant can rapidly encrypt files, potentially evading detection by traditional antivirus methods that rely on fixed time intervals for file scans. In ransomware detection research, numerous files can be encrypted by CPU-optimized ransomware until malicious activity is detected. This study proposes an early mitigation mechanism named CryptoSniffer, which is designed specifically to counter CPU-optimized ransomware attacks on server hosts. CryptoSniffer focuses on the misuse of CPU architecture-specific encryption instructions for swift file encryption by CPU-optimized ransomware. This can be achieved by capturing the ciphertext in user processes and thwarting file encryption by scrutinizing the content intended for writing. To demonstrate the efficacy of CryptoSniffer, the mechanism was implemented in the latest Linux kernel, and its security and performance were systematically evaluated. The experimental results demonstrate that CryptoSniffer successfully prevents real-world CPU-optimized ransomware, and the performance overhead is well-suited for practical applications.