Improving the Quality of Textual Adversarial Examples with Dynamic N-gram Based Attack

Abstract

Natural language models have been widely used for their impressive performance in various tasks, while their poor robustness also puts critical applications at high risk. These models are vulnerable to adversarial examples, which contain imperceptible noise that leads the model to wrong predictions. To ensure such malicious examples are imperceptible to humans, various word-level attack methods have been proposed. Previous works on word-level attacks attempt to generate adversarial examples by substituting words in sentences. They utilize different candidate substitution selection methods and substitution strategies to improve attack effectiveness and the quality of generated examples. However, previous works are all unigram-based attack methods, which ignore the connection between words. The unigram nature of these methods downgrades fluency, increases grammatical errors, and biases the semantics of adversarial examples, making adversarial examples easier to be detected by humans. In this paper, to improve the quality of textual adversarial examples and makes the adversarial example more imperceptible to human, we propose a black-box word-level attack method called Dynamic N-Gram Based Attack (DyGram). DyGram tokenizes the entire sentence into multiple n-gram units, rather than individual words as in previous works, and substitutes words in a sentence in descending order of n-gram unit importance. Extensive experiments demonstrate that DyGram achieves higher attack success rates than previous attack methods and improves the quality of generated adversarial examples in terms of the number of perturbed words, perplexity, grammatical correctness, and semantic similarity.