Optimally Tuning Finite-Difference Estimators

We consider stochastic gradient estimation when only noisy function evaluations are available. Central finite-difference scheme is a common method in this setting, which involves generating samples under perturbed inputs. Though it is widely known how to select the perturbation size to achieve the optimal order of the error, exactly achieving the optimal first-order error, which we call asymptotic optimality, is considered much more challenging and not attempted in practice. In this paper, we provide evidence that designing asymptotically optimal estimator is practically possible. In particular, we propose a new two-stage scheme that first estimates the required parameter in the perturbation size, followed by running finite-difference based on the estimated parameter in the first stage. Both theory and numerical experiments demonstrate the optimality of the proposed estimator and the robustness over conventional finite-difference schemes based on ad hoc tuning.