On the Effectiveness of Machine Learning Experiment Management Tools

Abstract

Machine learning experiment management tools support developers and data scientists on planning, tracking, and retrieving machine-learning experiments and assets when building intelligent software systems. Among others, they allow tracing back system behavior to experiment runs, for instance, when model performance drifts. Unfortunately, despite a surge of these tools, they are not well integrated with traditional software engineering tooling, and no hard empirical data exists on their effectiveness and value for users. We present a short research agenda and early results towards unified and effective software engineering and experiment management software.