ViT-LR: Pushing the Envelope for Transformer-Based on-Device Embedded Continual Learning

Abstract

State-of-the-Art Edge Artificial Intelligence (AI) is currently mostly targeted at a train-then-deploy paradigm: edge devices are exclusively responsible for inference, whereas training is delegated to data centers, leading to high energy and CO2 impact. On-Device Continual Learning could help in making Edge AI more sustainable by specializing AI models directly on-field. We deploy a continual image recognition model on a Jetson Xavier NX embedded system, and experimentally investigate how Attention influences performance and its viability as a Continual Learning backbone, analyzing the redundancy of its components to prune and further improve our solution efficiency. We achieve up to 83.81% accuracy on the Core50's new instances and classes scenario, starting from a pre-trained tiny Vision Transformer, surpassing AR1 *free with Latent Replay, and reach performance comparable and superior to the SoA without relying on growing Replay Examples.