How Should Developers Respond to App Reviews? Features Predicting the Success of Developer Responses

Context: The Google Play Store allows app developers to respond to user reviews. Existing research shows that response strategies vary considerably. In addition, while responding to reviews can lead to several types of favorable outcomes, not every response leads to success, which we define as increased user ratings. Aims: This work has two objectives. The first is to investigate the potential to predict early whether a developer response to a review is likely to be successful. The second is to pinpoint how developers can increase the chance of their responses to achieve success. Method: We track changes in user reviews of the 1,600 top free apps over a ten-week period, and find that in 11,034 out of 228,274 one- to four-star reviews, the ratings increase after a response. We extract three groups of features, namely time, presentation and tone, from the responses given to these reviews. We apply the extreme gradient boosting (XGBoost) algorithm to model the success of developer responses using these features. We employ model interpretation techniques to derive insights from the model. Results: Our model can achieve an AUC of 0.69, thus demonstrating that feature engineering and machine learning have the potential to enable developers to estimate the probability of success of their responses at composition time. We learn from it that the ratio between the length of the review and response, the textual similarity between the review and response, and the timeliness and the politeness of the response have the highest predictive power for distinguishing successful and unsuccessful developer responses. Conclusions: Based on our findings, we provide recommendations that developers can follow to increase the chance of success of their responses. Tools may also leverage our findings to support developers in writing more effective responses to reviews on the app store.