Benchmarking Label Dynamics of VirusTotal Engines

VirusTotal is the largest online anti-malware scanning service. It is widely used by security researchers for labeling malware data or serving as a comparison baseline. However, several important challenges of using VirusTotal are left unaddressed (e.g., whether VirusTotal labels are already stable, when VirusTotal labels can be trusted), severely harming the correctness of research projects depending on VirusTotal. In this paper, we present VTSet, which contains daily VirusTotal labels on more than 14,000 files over one year. VTSet can be used to build and evaluate various tools to tackle the existing challenges and facilitate the usage of VirusTotal. Besides the data, VTSet also provides a demonstration tool to display many measurement results and a query tool to ease the access of its data. A video demonstration of VTSet is located at the following link: https://youtu.be/aSVaUGHxFi4.