Architecture for an Offline Parallel Debugger

This paper provides and overview of the {\it GDBase} framework for offline parallel debuggers. The framework was designed to become the basis of debugging tools which scale successfully on systems with tens to hundreds of thousands of cores. With several systems coming online at more than 50,000 cores in the past year, debuggers which can run at these scales are now required. The proposed framework offers two features not found in current generation debugging tools: the ability to debug "offline'', and a central database to act as a repository of debugging information. These two features enable the GDBase debugger to offer several advantages. The debugger can be used in conjunction with modern batch systems with low overhead, with user interaction taking place after the parallel system resources are freed. The use of a database and a simple API allows for multiple interfaces and data mining tools to be implemented to provide novel ways of viewing and analyzing debugging data. The database also enables cross-run analysis, and the combination of debugging, performance, and system health information. Evidence is provided of the scalability of the framework, as well as output from several simple analysis tools that have been implemented.