Leveraging Partial Model Extractions using Uncertainty Quantification

Companies deploy deep learning models in the cloud and offer black-box access to them as a pay as you go service. It has been shown that with enough queries those models can be extracted. This paper presents a new cloning scheme using uncertainty quantification, enabling the adversary to leverage partial model extractions. First, a relatively small number of queries is spent to extract part of the target’s model. Second, for every query directed at the adversary, the uncertainty of the output of the extracted model is computed; when below a given threshold, the adversary will return the output. Otherwise, the query is delegated to the target’s model and its output returned. In this way the adversary is able to monetize knowledge that has successfully been extracted. We propose methods to determine thresholds such that the accuracy of the new scheme is close to the target network’s accuracy. The new scheme has been implemented, and experiments were conducted on the Caltech-256 and indoor datasets using multiple uncertainty quantification methods. The results show that the rate of delegation decreases logarithmically with the initial number of queries spent on extraction. Compared to conventional cloning techniques, the main advantages of the new scheme are that the total costs in terms of queries to the target model can be lower while achieving the same accuracy, and that the accuracy of the new scheme can be arbitrarily close to the target model’s accuracy by selecting a suitable value of the threshold.