AI Driven Methodology for Anomaly Detection in Apache Spark Streaming Systems

Ahmad Alnafessah, G. Casale
{"title":"AI Driven Methodology for Anomaly Detection in Apache Spark Streaming Systems","authors":"Ahmad Alnafessah, G. Casale","doi":"10.1109/ICCAIS48893.2020.9096667","DOIUrl":null,"url":null,"abstract":"Cloud computing, Artificial Intelligence, and Big Data technologies have recently become one of the most impactful forms of technology innovation. It is common to have multiple users share the same computing resources. This practice noticeably leads to performance anomalies. For instance, some applications can feature variability in processing time due to interference from other applications, or software contention from the other users, which may lead to unexpectedly long execution time and be considered anomalous. There is an urgent need for an automated effective performance anomaly detection method that can be used within the production environment for the streaming system to avoid any late detection of unexpected system failures. To address this challenge, we introduce a new black-box training workload configuration optimization with a neural network driven methodology to identify anomalous performance in an in-memory Spark streaming Big Data platform. The proposed methodology effectively uses Bayesian Optimization to find the ideal training dataset size and Spark streaming workload configuration parameters to train the anomaly detection model. The proposed model is validated on the Apache Spark streaming system. The results demonstrate that the proposed solution succeeds and accurately detects many types of performance anomalies. In addition, the training time for the machine learning model is reduced by more than 50%, which offers a fast anomaly detection deployment for system developers to utilize more efficient monitoring solutions.","PeriodicalId":422184,"journal":{"name":"2020 3rd International Conference on Computer Applications & Information Security (ICCAIS)","volume":"28 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2020-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2020 3rd International Conference on Computer Applications & Information Security (ICCAIS)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICCAIS48893.2020.9096667","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1

Abstract

Cloud computing, Artificial Intelligence, and Big Data technologies have recently become one of the most impactful forms of technology innovation. It is common to have multiple users share the same computing resources. This practice noticeably leads to performance anomalies. For instance, some applications can feature variability in processing time due to interference from other applications, or software contention from the other users, which may lead to unexpectedly long execution time and be considered anomalous. There is an urgent need for an automated effective performance anomaly detection method that can be used within the production environment for the streaming system to avoid any late detection of unexpected system failures. To address this challenge, we introduce a new black-box training workload configuration optimization with a neural network driven methodology to identify anomalous performance in an in-memory Spark streaming Big Data platform. The proposed methodology effectively uses Bayesian Optimization to find the ideal training dataset size and Spark streaming workload configuration parameters to train the anomaly detection model. The proposed model is validated on the Apache Spark streaming system. The results demonstrate that the proposed solution succeeds and accurately detects many types of performance anomalies. In addition, the training time for the machine learning model is reduced by more than 50%, which offers a fast anomaly detection deployment for system developers to utilize more efficient monitoring solutions.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
人工智能驱动的方法异常检测在Apache Spark流系统
云计算、人工智能和大数据技术最近已经成为最具影响力的技术创新形式之一。让多个用户共享相同的计算资源是很常见的。这种做法明显会导致性能异常。例如,由于其他应用程序的干扰或来自其他用户的软件争用,某些应用程序的处理时间可能具有可变性,这可能导致意外的长执行时间并被认为是异常的。目前迫切需要一种自动化的、有效的性能异常检测方法,该方法可以在流系统的生产环境中使用,以避免任何意外系统故障的后期检测。为了应对这一挑战,我们引入了一种新的黑盒训练负载配置优化方法,该方法采用神经网络驱动的方法来识别内存Spark流大数据平台中的异常性能。该方法有效地利用贝叶斯优化找到理想的训练数据集大小和Spark流工作负载配置参数来训练异常检测模型。在Apache Spark流系统上对该模型进行了验证。结果表明,该方法是成功的,能够准确地检测到多种类型的性能异常。此外,机器学习模型的训练时间减少了50%以上,这为系统开发人员提供了一个快速的异常检测部署,以利用更有效的监控解决方案。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
ICCAIS 2020 Copyright Page The Best-Worst Method for Resource Allocation and Task Scheduling in Cloud Computing A Recommender System for Linear Satellite TV: Is It Possible? Proactive Priority Based Response to Road Flooding using AHP: A Case Study in Dammam Data and Location Privacy Issues in IoT Applications
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1