Automatic Calibration Using Receiver Operating Characteristics Curves

Application-level filters, such as e-mail and VoIP spam filters, that analyze dynamic behavior changes are replacing static signature-recognition filters. These application-level filters learn behavior and use that knowledge to filter unwanted requests. Because behavior of a service request's participating entities changes rapidly, filters must adapt quickly by using end user's preferences about receiving that service request message. Many adaptive filters learn from the participating entities' behavior; however, none configure themselves automatically to an end user's changing tolerance levels. Also, filter administrators cannot manually change the threshold for each service request in real time. Traditional adaptive filters fail when administrators must optimize multiple filter thresholds manually and often. Thus, to improve a filter's learning, we must automate its threshold-update process. We propose an automatic threshold-calibration mechanism using Receiver Operating Characteristics (ROC) curves that updates the threshold based on an end user's feedback. To demonstrate the mechanism's real-time applicability, we integrated it in a Voice over IP (VoIP) spam filter that analyzes incoming Spam over IP Telephony (SPIT) calls. Using this mechanism, we observed good improvement in the VoIP spam filter's accuracy. Further, computing and updating the optimum threshold in realtime does not impede the filter's temporal performance because we update thresholds after each call's completion. Because we reach an optimum threshold for any initial setting, this mechanism works efficiently when we cannot predict end-user behavior. Furthermore, automatic calibration proves efficient when using multiple threshold values.