Hysteresis-based optimization of data transfer throughput

Abstract

The achievable throughput for a data transfer can be determined by a variety of factors such as network bandwidth, round trip time, background traffic, dataset size, and end-system configuration. For the best-effort optimization of the transfer throughput, three application-layer transfer parameters -- pipelining, parallelism and concurrency -- have been actively used in the literature. However, it is highly challenging to identify the best combination of these parameter settings for a specific data transfer request. In this paper, we analyze historical data consisting of 70 Million file transfers; apply data mining techniques to extract the hidden relations among the parameters and the optimal throughput; and propose a novel approach based on hysteresis to predict the optimal parameter settings.