Inter-model interpretability: Self-supervised models as a case study

Since early machine learning models, metrics such as accuracy and precision have been the de facto way to evaluate and compare trained models. However, a single metric number does not fully capture model similarities and differences, especially in the computer vision domain. A model with high accuracy on a certain dataset might provide a lower accuracy on another dataset without further insights. To address this problem, we build on a recent interpretability technique called Dissect to introduce inter-model interpretability, which determines how models relate or complement each other based on the visual concepts they have learned (such as objects and materials). Toward this goal, we project 13 top-performing self-supervised models into a Learned Concepts Embedding (LCE) space that reveals proximities among models from the perspective of learned concepts. We further crossed this information with the performance of these models on four computer vision tasks and 15 datasets. The experiment allowed us to categorize the models into three categories and revealed the type of visual concepts different tasks required for the first time. This is a step forward for designing cross-task learning algorithms.