M. Golzadeh, Alexandre Decan, Natarajan Chidambaram
Development bots are often used to automate a wide variety of repetitive tasks in collaborative software development. Such bots are commonly among the most active project contributors in terms of commit activity. As such, tools that analyse contributor activity (e.g., for recognizing and giving credit to project members for their contributions) need to take into account the bots and exclude their activity. While there are a few techniques to detect bots in software repositories, these techniques are not perfect and may miss some bots or may wrongly identify some human accounts as bots. In this paper, we present an exploratory study on the accuracy of bot detection techniques on a set of 540 accounts from 27 GitHub projects. We show that none of the bot detection techniques are accurate enough to detect bots among the 20 most active contributors of each project. We show that combining these techniques drastically increases the accuracy and recall of bot detection. We also highlight the importance of considering bots when attributing contributions to humans, since bots are prevalent among the top contributors and responsible for large proportions of commits.
{"title":"On the Accuracy of Bot Detection Techniques","authors":"M. Golzadeh, Alexandre Decan, Natarajan Chidambaram","doi":"10.1145/3528228.3528406","DOIUrl":"https://doi.org/10.1145/3528228.3528406","url":null,"abstract":"Development bots are often used to automate a wide variety of repetitive tasks in collaborative software development. Such bots are commonly among the most active project contributors in terms of commit activity. As such, tools that analyse contributor activity (e.g., for recognizing and giving credit to project members for their contributions) need to take into account the bots and exclude their activity. While there are a few techniques to detect bots in software repositories, these techniques are not perfect and may miss some bots or may wrongly identify some human accounts as bots. In this paper, we present an exploratory study on the accuracy of bot detection techniques on a set of 540 accounts from 27 GitHub projects. We show that none of the bot detection techniques are accurate enough to detect bots among the 20 most active contributors of each project. We show that combining these techniques drastically increases the accuracy and recall of bot detection. We also highlight the importance of considering bots when attributing contributions to humans, since bots are prevalent among the top contributors and responsible for large proportions of commits.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128502154","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Juan Carlos Farah, Basile Spaenlehauer, Xinyang Lu, Sandy Ingram, D. Gillet
The widespread use of bots to support software development makes social coding platforms such as GitHub a particularly rich source of data for the study of human-bot interaction. Software development bots are used to automate repetitive tasks, interacting with their human counterparts via comments posted on the various discussion interfaces available on such platforms. One type of interaction supported by GitHub involves reacting to comments using predefined emoji. To investigate how users react to bot comments, we conducted an observational study comprising 54 million GitHub comments, with a particular focus on comments that elicited the laugh reaction. The results from our analysis suggest that some reaction types are not equally distributed across human and bot comments and that a bot's design and purpose influence the types of reactions it receives. Furthermore, while the laugh reaction is not exclusively used to express laughter, it can be used to convey humor when a bot behaves unexpectedly. These insights could inform the way bots are designed and help developers equip them with the ability to recognize and recover from unanticipated situations. In turn, bots could better support the communication, collaboration, and productivity of teams using social coding platforms.
{"title":"An Exploratory Study of Reactions to Bot Comments on GitHub","authors":"Juan Carlos Farah, Basile Spaenlehauer, Xinyang Lu, Sandy Ingram, D. Gillet","doi":"10.1145/3528228.3528409","DOIUrl":"https://doi.org/10.1145/3528228.3528409","url":null,"abstract":"The widespread use of bots to support software development makes social coding platforms such as GitHub a particularly rich source of data for the study of human-bot interaction. Software development bots are used to automate repetitive tasks, interacting with their human counterparts via comments posted on the various discussion interfaces available on such platforms. One type of interaction supported by GitHub involves reacting to comments using predefined emoji. To investigate how users react to bot comments, we conducted an observational study comprising 54 million GitHub comments, with a particular focus on comments that elicited the laugh reaction. The results from our analysis suggest that some reaction types are not equally distributed across human and bot comments and that a bot's design and purpose influence the types of reactions it receives. Furthermore, while the laugh reaction is not exclusively used to express laughter, it can be used to convey humor when a bot behaves unexpectedly. These insights could inform the way bots are designed and help developers equip them with the ability to recognize and recover from unanticipated situations. In turn, bots could better support the communication, collaboration, and productivity of teams using social coding platforms.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128864031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
GitBots are bots in Git repositories to automate repetitive tasks that occur in software development, testing and maintenance. Git-Bots are expected to perform the repetitive tasks that are normally done by humans, such as feedback on issue reports and answers to questions. However, studies on GitBots for labeling issue reports fall short of replacing developers' labeling tasks. Developers still manually attach labels to issues. In this paper, we introduce an issue labeling bot classifying issue reports into custom labels that developers define by themselves so that our bot could attach labels in a similar way to human behavior.
{"title":"Classifying Issues into Custom Labels in GitBot","authors":"Doje Park, Heetae Cho, Seonah Lee","doi":"10.1145/3528228.3528404","DOIUrl":"https://doi.org/10.1145/3528228.3528404","url":null,"abstract":"GitBots are bots in Git repositories to automate repetitive tasks that occur in software development, testing and maintenance. Git-Bots are expected to perform the repetitive tasks that are normally done by humans, such as feedback on issue reports and answers to questions. However, studies on GitBots for labeling issue reports fall short of replacing developers' labeling tasks. Developers still manually attach labels to issues. In this paper, we introduce an issue labeling bot classifying issue reports into custom labels that developers define by themselves so that our bot could attach labels in a similar way to human behavior.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"2010 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123920981","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hamid Mohayeji, Felipe Ebert, Eric Arts, Eleni Constantinou, Alexander Serebrenik
Bots support different software maintenance and evolution activities, such as code review or executing tests. Recently, several bots have been proposed to help developers to keep track of postponed activities, expressed by means of TODO comments: e.g., TODO Bot automatically creates a GITHuB issue when a TODO comment is added to a repository, increasing visibility of TODO comments. In this work, we perform a preliminary evaluation of the impact of the TODO Bot on software development practice. We conjecture that the introduction of the TODO Bot would facilitate keeping track of the TODO comments, and hence encourage developers to use more TODO comments in their code changes. To evaluate this conjecture, we analyze all the 2,208 repositories which have at least one GITHuB issue created by the TODO Bot. Firstly, we investigate to what extent the bot is being used and describe the repositories using the bot. We observe that the majority (54%) of the repositories which adopted the TODO Bot are new, i.e., were created within less than one month of first issue created by the bot, and from those, more than 60% have the issue created within three days. We observe a statistically significant increase in the number of the TODO comments after the adoption of the bot, however with a small effect size. Our results suggest that the adoption of the TODO Bot encourages developers to introduce TODO comments rendering the postponed decisions more visible. Nevertheless, it does not speed up the process of addressing TODO comments or corresponding GITHuB issues.
{"title":"On the Adoption of a TODO Bot on GITHuB: A Preliminary Study","authors":"Hamid Mohayeji, Felipe Ebert, Eric Arts, Eleni Constantinou, Alexander Serebrenik","doi":"10.1145/3528228.3528408","DOIUrl":"https://doi.org/10.1145/3528228.3528408","url":null,"abstract":"Bots support different software maintenance and evolution activities, such as code review or executing tests. Recently, several bots have been proposed to help developers to keep track of postponed activities, expressed by means of TODO comments: e.g., TODO Bot automatically creates a GITHuB issue when a TODO comment is added to a repository, increasing visibility of TODO comments. In this work, we perform a preliminary evaluation of the impact of the TODO Bot on software development practice. We conjecture that the introduction of the TODO Bot would facilitate keeping track of the TODO comments, and hence encourage developers to use more TODO comments in their code changes. To evaluate this conjecture, we analyze all the 2,208 repositories which have at least one GITHuB issue created by the TODO Bot. Firstly, we investigate to what extent the bot is being used and describe the repositories using the bot. We observe that the majority (54%) of the repositories which adopted the TODO Bot are new, i.e., were created within less than one month of first issue created by the bot, and from those, more than 60% have the issue created within three days. We observe a statistically significant increase in the number of the TODO comments after the adoption of the bot, however with a small effect size. Our results suggest that the adoption of the TODO Bot encourages developers to introduce TODO comments rendering the postponed decisions more visible. Nevertheless, it does not speed up the process of addressing TODO comments or corresponding GITHuB issues.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"34 45","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120966203","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Model-driven methods are gaining momentum in the industry to develop software intensive systems. To be effective in quality and efficient in productivity, they require a strong toolchain with seamless automation. The DevOps approach can help reach this by unifying software development and operations with a strong focus on automation and monitoring. The aim of this short paper is to review automation tasks that are specific to a model-driven context and to classify them according to a typical DevOps lifecycle covering design, code, testing, deployment and runtime activities. Tasks are identified based on different industry use cases experienced in our research centre or reported in the literature. Some challenges are identified and discussed, especially related to the use of bots in a model-driven context.
{"title":"Survey of Automation Practices in Model-Driven Development and Operations","authors":"C. Ponsard, Valéry Ramon","doi":"10.1145/3528228.3528405","DOIUrl":"https://doi.org/10.1145/3528228.3528405","url":null,"abstract":"Model-driven methods are gaining momentum in the industry to develop software intensive systems. To be effective in quality and efficient in productivity, they require a strong toolchain with seamless automation. The DevOps approach can help reach this by unifying software development and operations with a strong focus on automation and monitoring. The aim of this short paper is to review automation tasks that are specific to a model-driven context and to classify them according to a typical DevOps lifecycle covering design, code, testing, deployment and runtime activities. Tasks are identified based on different industry use cases experienced in our research centre or reported in the literature. Some challenges are identified and discussed, especially related to the use of bots in a model-driven context.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"105 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127412598","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Software startups develop innovative, software-intensive product and services. This context leads to uncertainty regarding the software they are building. Experimentation, a process of testing hypotheses about the product, helps these companies to reduce uncertainty through different evidence-based approaches. The first step in experimentation is to identify the hypotheses to be tested. HyMap is a technique where a facilitator helps a software startup founder to draw a cognitive map representing her understanding of the context and, based on that, create hypotheses about the software to be built. In this paper, we present the Digital Mentor, an working-in-progress conversational bot to help creating a HyMap without the need of a human facilitator. We report the proposed solution consisting of a web application with the backend of a natural language understanding system, the current state of development, the challenges we faced so far and the next steps we plan to move forward.
{"title":"Digital Mentor: towards a conversational bot to identify hypotheses for software startups","authors":"Jorge Melegati, Xiaofeng Wang","doi":"10.1145/3528228.3528407","DOIUrl":"https://doi.org/10.1145/3528228.3528407","url":null,"abstract":"Software startups develop innovative, software-intensive product and services. This context leads to uncertainty regarding the software they are building. Experimentation, a process of testing hypotheses about the product, helps these companies to reduce uncertainty through different evidence-based approaches. The first step in experimentation is to identify the hypotheses to be tested. HyMap is a technique where a facilitator helps a software startup founder to draw a cognitive map representing her understanding of the context and, based on that, create hypotheses about the software to be built. In this paper, we present the Digital Mentor, an working-in-progress conversational bot to help creating a HyMap without the need of a human facilitator. We report the proposed solution consisting of a web application with the backend of a natural language understanding system, the current state of development, the challenges we faced so far and the next steps we plan to move forward.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122295153","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Natarajan Chidambaram, Alexandre Decan, M. Golzadeh
Contemporary social coding platforms such as GitHub facilitate collaborative distributed software development. Developers engaged in these platforms often use machine accounts (bots) for automating effort-intensive or repetitive activities. Determining whether a contributor corresponds to a bot or a human account is important in socio-technical studies, for example to assess the positive and negative impact of using bots, analyse the evolution of bots and their usage, identify top human contributors, and so on. BoDeGHa is one of the bot detection tools that have been proposed in the literature. It relies on comment activity within a single repository to predict whether an account is driven by a bot or by a human. This paper presents preliminary results on how the effectiveness of BoDeGHa can be improved by combining the predictions obtained from many repositories at once. We found that doing this not only increases the number of cases for which a prediction can be made, but that many diverging predictions can be fixed this way. These promising, albeit preliminary, results suggest that the “wisdom of the crowd” principle can improve the effectiveness of bot detection tools.
{"title":"Leveraging Predictions From Multiple Repositories to Improve Bot Detection","authors":"Natarajan Chidambaram, Alexandre Decan, M. Golzadeh","doi":"10.1145/3528228.3528403","DOIUrl":"https://doi.org/10.1145/3528228.3528403","url":null,"abstract":"Contemporary social coding platforms such as GitHub facilitate collaborative distributed software development. Developers engaged in these platforms often use machine accounts (bots) for automating effort-intensive or repetitive activities. Determining whether a contributor corresponds to a bot or a human account is important in socio-technical studies, for example to assess the positive and negative impact of using bots, analyse the evolution of bots and their usage, identify top human contributors, and so on. BoDeGHa is one of the bot detection tools that have been proposed in the literature. It relies on comment activity within a single repository to predict whether an account is driven by a bot or by a human. This paper presents preliminary results on how the effectiveness of BoDeGHa can be improved by combining the predictions obtained from many repositories at once. We found that doing this not only increases the number of cases for which a prediction can be made, but that many diverging predictions can be fixed this way. These promising, albeit preliminary, results suggest that the “wisdom of the crowd” principle can improve the effectiveness of bot detection tools.","PeriodicalId":431263,"journal":{"name":"2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE)","volume":"98 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121473399","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：