A Scoring Method for Interpretability of Concepts in Convolutional Neural Networks

In this paper, we propose a scoring algorithm for measuring the interpretability of CNN models by focusing on the feature extraction operation at the convolutional layers. The proposed approach is based on the principal of concept analysis, for a predefined list of concepts. A map of the network is created based on its responsiveness against each concept. Once this map is ready, various images can be applied as inputs and they are matched with the concepts whose hidden nodes are highly activated. Finally, the evaluation algorithm kicks in to use these descriptions during the final prediction and provides human-understandable explanations.