Zero-Shot Learning based on Vision Transformer

Abstract

Zero-Shot Learning (ZSL) simulates human’s transfer learning mechanism, which can recognize samples or categories that have not appeared during the training phase. However, the current ZSL still has a domain shift issue. To solved the domain shift issue, we propose a new ZSL method that combines Vision Transformer (ViT) and the encoder-decoder mechanism. This method refers to ViT’s Multi-Head Self-Attention (MSA) to extract more detailed visual features. The encoder-decoder mechanism can make the semantic information extracted from the image features accurately express its visual features and enhance recognition accuracy. We implemented it on three data sets of CUB, SUN and AWA2, and the experimental results proved that the method suggested in this study performs better than the current available methods. It shows that our new method is an effective ZSL method.