Estimating Contribution of Training Datasets using Shapley Values in Data-scale for Visual Recognition

Abstract

In this paper, we propose a method to measure contributions of multiple datasets i.e. how much a specific dataset contributes to improve accuracy of the model. Our method is based on shapley value, of which purpose is to measure contribution by difference of the accuracy of the models. Unlike previous method, our method first converts the accuracy to data-scale measurements using fitted log curve. We calculate contributions in a fair way that each trials are evaluated not by its improvements of accuracy, but by the number of data needed to make the improvements. Our method can avoid overestimation of contributions in small data cases. To evaluate the proposed method, we trained models for Person Re-Identification tasks with combinations of datasets, and calculated contributions of each datasets. Results show that the proposed metrics can effectively reduce the over-estimations in small data cases, while the contributions maintain good properties such as local accuracy and additive law derived from shapley value definition. We also proposed normalization of shapley values in data-scale by its actual number of instances, which indicates intrinsic importance of a dataset per instance.