Troubleshooting distributed data analytics systems

Abstract

Data analytics applications are deployed on large-scale distributed systems. In order to ensure high performance, troubleshooting for such applications and underlying systems is critical. In this thesis, we focus on efficient log analysis for troubleshooting distributed data analytics systems. We made the following contributions. 1) We designed a tool that collects logs and resource metrics of distributed data analytics systems to facilitate troubleshooting processes. 2) We designed a log analysis tool that is able to extract semantic meaning from logs and automatically report potential anomalies by leveraging natural language processing approaches.