Analysis of Layer Efficiency and Layer Reduction on Pre-Trained Deep Learning Models

Abstract

Recent technologies in the deep learning area enable many industries and practitioners fastening the development processes of their products. However, deep learning still encounters some potential issues like overfitting and huge size. The huge size greatly constrains performance and portability of the deep learning model in embedded devices with limited environments. Due to the paradigm of it mixed with the meaning of “deep” layers, many researchers tend to derive the pre-trained model into building deeper layers to solve their problems without knowing whether they are actually needed or not. To address these issues, we exploit the activation and gradient output and weight in each layer of the pre-trained models to measure its efficiencies. By exploiting them, we estimate the efficiencies using our measurements and compare it with the manual layer reduction to validate the most relevant method. We also use the method for continuous layer reductions for validation. With this approach, we save up to 12x and 26x of the time of one manual layer reduction and re-training on VGG-16 and custom AlexNet respectively.