DryadOpt: Branch-and-Bound on Distributed Data-Parallel Execution Engines

Abstract

We introduce Dryad Opt, a library that enables massively parallel and distributed execution of optimization algorithms for solving hard problems. Dryad Opt performs an exhaustive search of the solution space using branch-and-bound, by recursively splitting the original problem into many simpler sub problems. It uses both parallelism (at the core level) and distributed execution (at the machine level). Dryad Opt provides a simple yet powerful interface to its users, who only need to implement sequential code to process individual sub problems (either by solving them in full or generating new sub problems). The parallelism and distribution are handled automatically by Dryad Opt, and are invisible to the user. The distinctive feature of our system is that it is implemented on top of Dryad LINQ, a distributed data-parallel execution engine similar to Hadoop and Map-Reduce. Despite the fact that these engines offer a constrained application model, with restricted communication patterns, our experiments show that careful design choices allow Dryad Opt to scale linearly with the number of machines, with very little overhead.