Towards sharper excess risk bounds for differentially private pairwise learning

Pairwise learning is a vital part of machine learning. It depends on pairs of training instances, and is naturally fit for modeling relationships between samples. However, as a data driven paradigm, it faces huge privacy issues. Differential privacy (DP) is a useful tool to protect the privacy of machine learning, but corresponding excess population risk bounds are loose in existing DP pairwise learning analysis. In this paper, we propose a gradient perturbation algorithm for pairwise learning to get better risk bounds under Polyak–Łojasiewicz condition, including both convex and non-convex cases. Specifically, for the theoretical risk bound in expectation, previous best results are of rates $O\left(\frac{p}{n^2{ϵ}^2}+\frac{1}{n}\right)$ and $O\left(\frac{\sqrt{p}}{nϵ}+\frac{1}{\sqrt{n}}\right)$ under strongly convex condition and convex conditions, respectively. In this paper, we use the on-average stability and achieve an $O\left(min\left\{\frac{\sqrt{p}}{n^{1.5}ϵ}+\frac{p^{1.5}}{n^{2.5}{ϵ}^3},\frac{p}{n^2{ϵ}^2}+\frac{1}{n}\right\}\right)$ bound, significantly improving previous bounds. For the high probability risk bound, previous best results are analyzed by the uniform stability, and $O\left({\beta }_n^U+\frac{\sqrt{p}}{\sqrt{n}ϵ}\right)$ excess population risk bounds are achieved under strongly convex or convex conditions, where ${\beta }_n^U$ is the traditional pairwise uniform stability parameter, it is large since it considers the worst case of the loss sensitivity. In this paper, we propose the pairwise locally elastic stability and improve the high probability bound to $O\left(\frac{{\beta }_E}{\sqrt{n}}+\frac{\sqrt{p}}{\sqrt{n}ϵ}\right)$, in which the pairwise locally elastic stability parameter ${\beta }_E\ll {\beta }_n^U$ because it considers the average sensitivity of the pairwise loss function.