Parallel Out-of-Core Matlab for Extreme Virtual Memory

Abstract

Summary form only given. Large data sets that cannot fit in memory can be addressed with out-of-core methods, which use memory as a "window " to view a section of the data stored on disk at a time. The parallel Matlab for eXtreme virtual memory (pMatlab XVM) library adds out-of-core extensions to the parallel Matlab (pMatlab) library. We have applied pMatlab XVM to the DARPA high productivity computing systems' HPCchallenge FFT benchmark. The benchmark was run using several different implementations: C+MPI, pMatlab, pMatlab hand coded for out-of-core and pMatlab XVM. These experiments found 1) the performance of the C+MPI and pMatlab versions were comparable; 2) the out-of-core versions deliver 80% of the performance of the in-core versions; 3) the out-of-core versions were able to perform a 1 terabyte (64 billion point) FFT and 4) the pMatlab XVM program was smaller, easier to implement and verify, and more efficient than its hand coded equivalent. We are transitioning this technology to several DoD signal processing applications and plan to apply pMatlab XVM to the full HPCchallenge benchmark suite. Using next generation hardware, problems sizes a factor of 100 to 1000 times larger should be feasible