Robustifying vision transformer for image forgery localization with multi-exit architectures

Abstract

The proliferation of image manipulation tools has led to an increase in the number of manipulated images being disseminated online, posing risks like the propagation of fake news and telecom fraud. Thus, there is an increasing demand for precise, generic, and robust methods for detecting and locating manipulated images. In this paper, we propose a simple and clean model, named MEAFormer, for image forgery localization that does not heavily rely on pre-trained models. MEAFormer comprises three main components: an encoder network, a neck network, and a decoder network. Specifically, the transformer-based encoder network extracts hierarchical feature representations from the input image, providing rich contextual information in each layer. The neck network, incorporating our proposed cross-layer feature aggregation (CFA), aggregates these hierarchical features. To achieve better spatial feature co-occurrence, instead of using noise or edge artifacts, we introduce a multi-scale graph reasoning (MGR) module within the decoder network via bipartite graphs over the encoder and decoder features in a multi-scale fashion. The cross-level enhancement (CLE) further performs adjacent-level feature fusion to amplify the regions of interest in aggregated manipulation features. Finally, the multi-exit architecture (MEA) guides the model to learn fine-grained features and segment out the manipulated region. Extensive experiments across diverse and challenging datasets conclusively establish the superiority of MEAFormer over existing state-of-the-art methods, excelling in accuracy, generalization, and robustness.