Towards recognition of open-set speech forgery algorithms by using prototype learning

Abstract

Recent advances in machine learning have made forged video and audio more convincing. This poses a threat to the security of individuals, societies and nations. To address this threat, the ASVspoof initiative was conceived to spearhead research on Automatic Speaker Verification (ASV) for anti-spoofing. Currently, most research on ASVspoof has focused on detecting whether speech has been tampered with. However, little attention has been paid to the recognition of speech forgery algorithms. Moreover, in the real world, new forgery algorithms keep emerging, making it difficult to adapt forgery algorithm recognition models trained under closed-set conditions to realistic open-set scenarios. Therefore, we propose a method based on prototype learning and adaptive thresholding for recognizing speech forgery algorithms in open-set. The method uses manifold mixup and dummy prototypes to simulate and recognize unknown speech forgery algorithms. Prototype classification improves the ability to recognize speech forgery algorithms with high similarity. At the same time, it has the advantage of preventing catastrophic forgetting and facilitates subsequent incremental training using samples of newly recognized forgery algorithms. Thus, our method increases the number of recognized categories for forgery algorithms. Experimental results show that our method is effective. The codes are available at https://github.com/multimedia-security/open-set-recognization.