High-speed detection of unsolicited bulk emails

Abstract

We propose a Progressive Email Classifier (PEC) for high-speed classification of message patterns that are commonly associated with unsolicited bulk email (UNBE). PEC is designed to operate at the network access point, the ingress between the Internet Service Provider (ISP) and the enterprise network; so that a surge of UNBE containing fresh patterns can be detected before they spread into the enterprise network. A real-time scoreboard keeps track of detected feature instances (FI) based on a scoring and aging engine, until they are considered either from valid or UNBE sources. A FI of a valid email is discarded, but an anomalous one is passed to a blacklist to control (e.g., block or defer) subsequent emails containing the FI. The anomaly detector of PEC can be used at different protocol layers. To gain some insights on the performance of PEC, we implemented PEC and integrated it with the sendmail daemon to detect anomalous URL links from email streams. Arbitrarily chosen on-line texts and URL links extracted from a corpus of spamming-phishing emails were used to compose testing emails. Experimental results on a Xeon based server show that PEC can handle 1.2M score/age updates, parse 0.9M URL links (of average size 30 bytes) for hashing and matching, and parsing of 25,000 email bodies of average size 1.5kB per second. The lossy detection system can be easily scaled by progressive selection of detection features and detection thresholds. It can be used alone or as an early screening tool for an existing infrastructure to defeat major UNBE flooding.