Large transfers for data analytics on shared wide-area networks

One part of large-scale data analytics is the problem of transferring the data across wide-area networks (WANs). Often, the data must be gathered (e.g., from remote sites), processed, possibly transferred (e.g., for further processing), and then possibly disseminated. If the data-transfer stages are bottlenecks, the overall data analytics pipeline will be affected. Although a variety of tools and protocols have been developed for large data transfers on WANs, most of the related work has been in the context of dedicated or non-shared networks. However, in practice, most networks are likely to be shared. We consider and evaluate the problem of large data transfers on shared networks and large round-trip-times (RTT) as are found on many WANs. Using a variety of synthetic background network traffic (e.g., uniform, TCP, UDP, square waveform, bursty), we compare the performance of well-known protocols (e.g., GridFTP, UDT). On our emulated WAN network, both GridFTP and UDT perform well in all-TCP situations, but UDT performs better when UDP-based background traffic is prominent.