Sentiment Analysis for Software Engineering: How Far Can We Go?

Abstract

Sentiment analysis has been applied to various software engineering (SE) tasks, such as evaluating app reviews or analyzing developers' emotions in commit messages. Studies indicate that sentiment analysis tools provide unreliable results when used out-of-the-box, since they are not designed to process SE datasets. The silver bullet for a successful application of sentiment analysis tools to SE datasets might be their customization to the specific usage context. We describe our experience in building a software library recommender exploiting crowdsourced opinions mined from Stack Overflow (e.g., what is the sentiment of developers about the usability of a library). To reach our goal, we retrained—on a set of 40k manually labeled sentences/words extracted from Stack Overflow—a state-of-the-art sentiment analysis tool exploiting deep learning. Despite such an effort- and time-consuming training process, the results were negative. We changed our focus and performed a thorough investigation of the accuracy of these tools on a variety of SE datasets. Our results should warn the research community about the strong limitations of current sentiment analysis tools.