Analysis of Evaluation Metrics with the Distance between Positive Pairs and Negative Pairs in Deep Metric Learning

Abstract

Deep metric learning (DML) acquires embeddings via deep learning, where distances among samples of the same class are shorter than those of different classes. The previous DML studies proposed new metrics to overcome the issues of general metrics, but they have the following two problems; one is that they consider only a small portion of the whole distribution of the data, and the other is that their scores cannot be directly compared among methods when the number of classes is different. To analyze these issues, we consider the histograms of the inner products between arbitrary positive pairs and those of negative pairs. We can evaluate the entire distribution by measuring the distance between the two histograms. By normalizing the histograms by their areas, we can also cancel the effect of the number of classes. In experiments, visualizations of the histograms revealed that the embeddings of the existing DML methods do not generalize well to the validation set. We also confirmed that the evaluation of the distance between the positive and negative histograms is less affected by the variation in the number of classes compared with Recall@1 and MAP@R.