Profit-Driven Experimental Design

Abstract

From intense competition to the recent pandemic, companies currently face considerable volatility in the business environment. For companies that design experiments to identify parameters of interest and make subsequent policy decisions based on these parameters, the cost of such experimentation has become increasingly comparable to the economic gains obtained, as the insights offered by an experiment can be short-lived due to changing market conditions. In this paper, we develop a general framework to quantify the total expected profit from both the experimental and postexperimental stages given an experimental strategy. The proposed framework is constructed using the asymptotic properties of the underlying parameter estimates as a channel to connect the profits from the two stages. Exploiting this framework, we calculate the difference in the total expected profits between any two experimental strategies, as well as the lower and upper bounds. Furthermore, we derive the actual and the bounds of the optimal sample size that maximizes the total expected profit. The profit and sample size bounds are independent of the ground-truth parameter value and can be calculated before conducting experiments to support experimental planning. In particular, our results demonstrate that when the postexperiment profit can be expressed as the sum of profits from N homogeneous units, the optimal sample size is on the order of O(\sqrt{N}). Finally, we showcase how our framework can be applied to different business setups, such as the demand-learning newsvendor problem and the pricing problem.