Michael Sildatke, H. Karwanni, B. Kraft, Oliver Schmidts, Albert Zündorf
In collaborative research projects, both researchers and practitioners work together solving business-critical challenges. These projects often deal with ETL processes, in which humans extract information from non-machine-readable documents by hand. AI-based machine learning models can help to solve this problem. Since machine learning approaches are not deterministic, their quality of output may decrease over time. This fact leads to an overall quality loss of the application which embeds machine learning models. Hence, the software qualities in development and production may differ. Machine learning models are black boxes. That makes practitioners skeptical and increases the inhibition threshold for early productive use of research prototypes. Continuous monitoring of software quality in production offers an early response capability on quality loss and encourages the use of machine learning approaches. Furthermore, experts have to ensure that they integrate possible new inputs into the model training as quickly as possible. In this paper, we introduce an architecture pattern with a reference implementation that extends the concept of Metrics Driven Research Collaboration with an automated software quality monitoring in productive use and a possibility to auto-generate new test data coming from processed documents in production. Through automated monitoring of the software quality and auto-generated test data, this approach ensures that the software quality meets and keeps requested thresholds in productive use, even during further continuous deployment and changing input data.
在协作研究项目中，研究人员和实践者共同解决关键业务挑战。这些项目通常处理ETL过程，在这个过程中，人们手工从非机器可读的文档中提取信息。基于人工智能的机器学习模型可以帮助解决这个问题。由于机器学习方法不确定，它们的输出质量可能会随着时间的推移而下降。这一事实导致嵌入机器学习模型的应用程序的整体质量下降。因此，开发和生产中的软件质量可能不同。机器学习模型是黑盒子。这使得从业者持怀疑态度，并增加了早期有效使用研究原型的抑制阈值。持续监控产品中的软件质量提供了对质量损失的早期响应能力，并鼓励使用机器学习方法。此外，专家必须确保他们尽可能快地将可能的新输入整合到模型训练中。在本文中，我们介绍了一个架构模式，该模式带有一个参考实现，它扩展了Metrics Driven Research Collaboration的概念，在生产中使用自动化的软件质量监控，并可能自动生成来自生产中处理过的文档的新测试数据。通过对软件质量和自动生成的测试数据的自动监控，这种方法确保软件质量在生产使用中满足并保持所要求的阈值，即使在进一步的连续部署和更改输入数据期间也是如此。
{"title":"Automated Software Quality Monitoring in Research Collaboration Projects","authors":"Michael Sildatke, H. Karwanni, B. Kraft, Oliver Schmidts, Albert Zündorf","doi":"10.1145/3387940.3391478","DOIUrl":"https://doi.org/10.1145/3387940.3391478","url":null,"abstract":"In collaborative research projects, both researchers and practitioners work together solving business-critical challenges. These projects often deal with ETL processes, in which humans extract information from non-machine-readable documents by hand. AI-based machine learning models can help to solve this problem. Since machine learning approaches are not deterministic, their quality of output may decrease over time. This fact leads to an overall quality loss of the application which embeds machine learning models. Hence, the software qualities in development and production may differ. Machine learning models are black boxes. That makes practitioners skeptical and increases the inhibition threshold for early productive use of research prototypes. Continuous monitoring of software quality in production offers an early response capability on quality loss and encourages the use of machine learning approaches. Furthermore, experts have to ensure that they integrate possible new inputs into the model training as quickly as possible. In this paper, we introduce an architecture pattern with a reference implementation that extends the concept of Metrics Driven Research Collaboration with an automated software quality monitoring in productive use and a possibility to auto-generate new test data coming from processed documents in production. Through automated monitoring of the software quality and auto-generated test data, this approach ensures that the software quality meets and keeps requested thresholds in productive use, even during further continuous deployment and changing input data.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115447497","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Genetic improvement has proved to be a successful technique in optimising various software properties, such as bug fixing, runtime improvement etc. It uses automated search to find improved program variants. Usually the evaluation of each mutated program involves running a test suite, and then calculating the fitness based on Boolean test case results. This, however, creates plateaus in the fitness landscape that are hard for search to efficiently traverse. Therefore, we propose to consider a more fine-grained fitness function that takes the output of test case assertions into account.
{"title":"Refining Fitness Functions in Test-Based Program Repair","authors":"J. Petke, Aymeric Blot","doi":"10.1145/3387940.3392180","DOIUrl":"https://doi.org/10.1145/3387940.3392180","url":null,"abstract":"Genetic improvement has proved to be a successful technique in optimising various software properties, such as bug fixing, runtime improvement etc. It uses automated search to find improved program variants. Usually the evaluation of each mutated program involves running a test suite, and then calculating the fitness based on Boolean test case results. This, however, creates plateaus in the fitness landscape that are hard for search to efficiently traverse. Therefore, we propose to consider a more fine-grained fitness function that takes the output of test case assertions into account.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123078760","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ryoko Izuta, S. Matsumoto, Yoshiki Higo, S. Kusumoto
This paper proposes a concept of introducing Git repository to record a history of program evolution via automated program repair techniques. In contrast to the general usage of Git by actual developers, a Git repository is generated by an APR system. This paper presents that it is feasible to store the history of program repair efficiently and comprehensively by using Git. Moreover, the proposed concept allows to share the details of an APR execution and to compare various APR executions.
{"title":"Program Repairing History as Git Repository","authors":"Ryoko Izuta, S. Matsumoto, Yoshiki Higo, S. Kusumoto","doi":"10.1145/3387940.3392178","DOIUrl":"https://doi.org/10.1145/3387940.3392178","url":null,"abstract":"This paper proposes a concept of introducing Git repository to record a history of program evolution via automated program repair techniques. In contrast to the general usage of Git by actual developers, a Git repository is generated by an APR system. This paper presents that it is feasible to store the history of program repair efficiently and comprehensively by using Git. Moreover, the proposed concept allows to share the details of an APR execution and to compare various APR executions.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122568835","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Per Lenberg, R. Feldt, Lars Göran Wallgren Tengberg, Lucas Gren
We are becoming increasingly dependent on software systems also for highly critical tasks in society. To minimize the risk of failures, regulatory institutions define standards that software organizations must meet. However, the quality of the safety-critical software is, ultimately, determined by the software engineers' behavior. Even though previous studies have recognized the significance of such behavioral aspects, research that studies them is limited. The aim of this initial study was, therefore, to identify how and in what way, behavioral aspects affect the quality of safety-critical software. Thematic analysis of interviews with six software engineers identified four themes linking developer behavior to safety. Our analysis suggests that developing safety-critical systems imposes stress on software engineers and that to reduce such pressure it is critical to enhance organizational trust. It also indicates that the agile way-of-working has the potential to improve safety by facilitating the sharing of domain knowledge. Our findings provide directions for future studies into these important aspects and can be of wider relevance, in particular for the development of secure software, but potentially also for general software engineering.
{"title":"Behavioral Aspects of Safety-Critical Software Development","authors":"Per Lenberg, R. Feldt, Lars Göran Wallgren Tengberg, Lucas Gren","doi":"10.1145/3387940.3392227","DOIUrl":"https://doi.org/10.1145/3387940.3392227","url":null,"abstract":"We are becoming increasingly dependent on software systems also for highly critical tasks in society. To minimize the risk of failures, regulatory institutions define standards that software organizations must meet. However, the quality of the safety-critical software is, ultimately, determined by the software engineers' behavior. Even though previous studies have recognized the significance of such behavioral aspects, research that studies them is limited. The aim of this initial study was, therefore, to identify how and in what way, behavioral aspects affect the quality of safety-critical software. Thematic analysis of interviews with six software engineers identified four themes linking developer behavior to safety. Our analysis suggests that developing safety-critical systems imposes stress on software engineers and that to reduce such pressure it is critical to enhance organizational trust. It also indicates that the agile way-of-working has the potential to improve safety by facilitating the sharing of domain knowledge. Our findings provide directions for future studies into these important aspects and can be of wider relevance, in particular for the development of secure software, but potentially also for general software engineering.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129407632","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We set down the principles behind a modeling language for quantum software. We present a minimal set of extensions to the well-known Unified Modeling Language (UML) that allows it to effectively model quantum software. These extensions are separate and independent of UML as a whole. As such they can be used to extend any other software modeling language, or as a basis for a completely new language. We argue that these extensions are both necessary and sufficient to model, abstractly, any piece of quantum software. Finally, we provide a small set of examples that showcase the effectiveness of the extension set.
{"title":"Towards a Quantum Software Modeling Language","authors":"Carlos A. Pérez-Delgado, H. G. Pérez-González","doi":"10.1145/3387940.3392183","DOIUrl":"https://doi.org/10.1145/3387940.3392183","url":null,"abstract":"We set down the principles behind a modeling language for quantum software. We present a minimal set of extensions to the well-known Unified Modeling Language (UML) that allows it to effectively model quantum software. These extensions are separate and independent of UML as a whole. As such they can be used to extend any other software modeling language, or as a basis for a completely new language. We argue that these extensions are both necessary and sufficient to model, abstractly, any piece of quantum software. Finally, we provide a small set of examples that showcase the effectiveness of the extension set.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128858207","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Angela Mayhua-Quispe, Franci Suni Lopez, Maria Fernanda Granda, Nelly Condori-Fernández
The present live study is proposed with the objective of investigating the influence of negative emotions (i.e., stress) in the efficiency for verifying conceptual models. To conduct this study, we use a Model-driven Testing tool, named CoSTest, and our own version of stress detector within a competition setting. The experiment design, overview of the empirical procedure, instrumentation and potential threats are presented in the proposal.
{"title":"How Do Negative Emotions Influence on the Conceptual Models Verification?: A live study proposal","authors":"Angela Mayhua-Quispe, Franci Suni Lopez, Maria Fernanda Granda, Nelly Condori-Fernández","doi":"10.1145/3387940.3392090","DOIUrl":"https://doi.org/10.1145/3387940.3392090","url":null,"abstract":"The present live study is proposed with the objective of investigating the influence of negative emotions (i.e., stress) in the efficiency for verifying conceptual models. To conduct this study, we use a Model-driven Testing tool, named CoSTest, and our own version of stress detector within a competition setting. The experiment design, overview of the empirical procedure, instrumentation and potential threats are presented in the proposal.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125301018","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
L. Canonico, Vimal Vakeel, James Dominic, Paige Rodeghero
Chaos Engineering refers to the practice of introducing faults in a system and observe the extent to which the system remains fault tolerant. However, is randomization the best approach to expose faults within a system? We aim to answer this question by introducing Chaos into different software architecture patterns and demonstrate how a back-end system can be made fault tolerant through artificial intelligence (AT). This paper discusses what aspects of AI would be used to make a system more resilient to perturbations and the results of these findings against existing chaos engineering approaches.
{"title":"Human-AI Partnerships for Chaos Engineering","authors":"L. Canonico, Vimal Vakeel, James Dominic, Paige Rodeghero","doi":"10.1145/3387940.3391493","DOIUrl":"https://doi.org/10.1145/3387940.3391493","url":null,"abstract":"Chaos Engineering refers to the practice of introducing faults in a system and observe the extent to which the system remains fault tolerant. However, is randomization the best approach to expose faults within a system? We aim to answer this question by introducing Chaos into different software architecture patterns and demonstrate how a back-end system can be made fault tolerant through artificial intelligence (AT). This paper discusses what aspects of AI would be used to make a system more resilient to perturbations and the results of these findings against existing chaos engineering approaches.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130276809","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Cyberattacks on the critical infrastructure is a growing concern for businesses, national authorities and public in general. The increasing complexity and connectivity of the critical infrastructure systems have made them susceptible to cyberattacks. The traditional notion of safety systems being isolated is no longer applicable, as we have seen ample examples on how these systems can be exploited through gaps in e.g. supply chain, physical security, insiders. This places greater importance on how the staff belonging to owners and operators of these critical infrastructure, e.g. operators, IT/security personnel, system engineers, management, are prepared to handle cyberattacks. This paper presents our ongoing research on investigating the preparedness of organisations to handle cybersecurity incidents and providing holistic solutions to improve cybersecurity posture. We present one experiment that has been conducted using our cybersecurity centre and man-machine laboratory to study how operators and security team of a power plant will handle a cyberattack. We highlight the main observations made through this experiment.
{"title":"What happens in a control room during a cybersecurity attack?: Preliminary observations from a pilot study","authors":"E. Nystad, Vikash Katta, J. Simensen","doi":"10.1145/3387940.3391454","DOIUrl":"https://doi.org/10.1145/3387940.3391454","url":null,"abstract":"Cyberattacks on the critical infrastructure is a growing concern for businesses, national authorities and public in general. The increasing complexity and connectivity of the critical infrastructure systems have made them susceptible to cyberattacks. The traditional notion of safety systems being isolated is no longer applicable, as we have seen ample examples on how these systems can be exploited through gaps in e.g. supply chain, physical security, insiders. This places greater importance on how the staff belonging to owners and operators of these critical infrastructure, e.g. operators, IT/security personnel, system engineers, management, are prepared to handle cyberattacks. This paper presents our ongoing research on investigating the preparedness of organisations to handle cybersecurity incidents and providing holistic solutions to improve cybersecurity posture. We present one experiment that has been conducted using our cybersecurity centre and man-machine laboratory to study how operators and security team of a power plant will handle a cyberattack. We highlight the main observations made through this experiment.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127930569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present the design of MTKeras, a generic metamorphic testing framework for machine learning, and demonstrate its effectiveness through case studies in image classification and sentiment analysis.
{"title":"A Testing Tool for Machine Learning Applications","authors":"Yelin Liu, Yang Liu, T. Chen, Z. Zhou","doi":"10.1145/3387940.3392694","DOIUrl":"https://doi.org/10.1145/3387940.3392694","url":null,"abstract":"We present the design of MTKeras, a generic metamorphic testing framework for machine learning, and demonstrate its effectiveness through case studies in image classification and sentiment analysis.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126644313","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Darryl C. Jarman, Riley Smith, Owen Johnston, D. Towey, Z. Zhou
In its simplest form, software testing consists of creating test cases from a defined input space, running them in the system-under-test (SUT), and evaluating the outputs with a mechanism for determining success or failure (i.e. an oracle). Metamorphic testing (MT) provides powerful concepts for alleviating the problem of a lack of oracles. To increase the adoption of MT among industry practitioners, approaches and tools that lower the effort to identify potential metamorphic relations (MRs) are very much in demand. As such, we propose a learning-based approach to MR discovery and exploration using concepts of metamorphic testing, association rule learning, and combinatorial testing. The results have implications for numerous applications including software testing and program comprehension, among others. These implications set a strong foundation for a future, extensible metamorphic exploration framework.
{"title":"ARCAMETES","authors":"Darryl C. Jarman, Riley Smith, Owen Johnston, D. Towey, Z. Zhou","doi":"10.1145/3387940.3391482","DOIUrl":"https://doi.org/10.1145/3387940.3391482","url":null,"abstract":"In its simplest form, software testing consists of creating test cases from a defined input space, running them in the system-under-test (SUT), and evaluating the outputs with a mechanism for determining success or failure (i.e. an oracle). Metamorphic testing (MT) provides powerful concepts for alleviating the problem of a lack of oracles. To increase the adoption of MT among industry practitioners, approaches and tools that lower the effort to identify potential metamorphic relations (MRs) are very much in demand. As such, we propose a learning-based approach to MR discovery and exploration using concepts of metamorphic testing, association rule learning, and combinatorial testing. The results have implications for numerous applications including software testing and program comprehension, among others. These implications set a strong foundation for a future, extensible metamorphic exploration framework.","PeriodicalId":309659,"journal":{"name":"Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127135470","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：