Workflows provide an easy to use programming model for the construction of complex services that are (recursively) composed of simpler services. When it comes to high performance workflow execution, the distribution (outscaling) of the constituent services of the workflow across an environment of computational nodes is a key concept and also a very straightforward advantage of the workflow paradigm. However, scalable workflow execution cannot only be provided by the distribution of services but also necessitates novel architectures for the workflow engine in charge of service orchestration. Even though workflow orchestration is commonly provided by centralized solutions, these architectures imply performance bottlenecks and single points of failure. Hence, the workflow engine has to be distributed as well, by efficiently replicating workflow metadata across several nodes in a network. A particular challenge stems from the requirement of providing scalable workflow execution that is at the same time also reliable. In this paper, we present OSIRIS-SR, a decentralized middleware for the distributed execution of workflows. It has particularly been designed to jointly provide a high degree of scalability and reliability. OSIRIS-SR locally leverages the concurrent and redundant Actor model for workflow processing, whereas globally OSIRIS-SR runs a number of scalable system services for the management of workflow metadata, with the Safety Ring being the most prominent one. The Safety Ring service features a self-healing node overlay for the purpose of active workflow instance supervision that serves at the same time as a scalable and reliable metadata storage. We discuss in detail the Safety Ring architecture and the mechanics behind the scalable and reliable workflow management in OSIRIS-SR. The evaluation results of OSIRIS-SR show that support for reliable workflow execution does not significantly impact the system's scalability characteristics.
{"title":"OSIRIS-SR: a scalable yet reliable distributed workflow execution engine","authors":"Nenad Stojnic, H. Schuldt","doi":"10.1145/2499896.2499899","DOIUrl":"https://doi.org/10.1145/2499896.2499899","url":null,"abstract":"Workflows provide an easy to use programming model for the construction of complex services that are (recursively) composed of simpler services. When it comes to high performance workflow execution, the distribution (outscaling) of the constituent services of the workflow across an environment of computational nodes is a key concept and also a very straightforward advantage of the workflow paradigm. However, scalable workflow execution cannot only be provided by the distribution of services but also necessitates novel architectures for the workflow engine in charge of service orchestration. Even though workflow orchestration is commonly provided by centralized solutions, these architectures imply performance bottlenecks and single points of failure. Hence, the workflow engine has to be distributed as well, by efficiently replicating workflow metadata across several nodes in a network. A particular challenge stems from the requirement of providing scalable workflow execution that is at the same time also reliable. In this paper, we present OSIRIS-SR, a decentralized middleware for the distributed execution of workflows. It has particularly been designed to jointly provide a high degree of scalability and reliability. OSIRIS-SR locally leverages the concurrent and redundant Actor model for workflow processing, whereas globally OSIRIS-SR runs a number of scalable system services for the management of workflow metadata, with the Safety Ring being the most prominent one. The Safety Ring service features a self-healing node overlay for the purpose of active workflow instance supervision that serves at the same time as a scalable and reliable metadata storage. We discuss in detail the Safety Ring architecture and the mechanics behind the scalable and reliable workflow management in OSIRIS-SR. The evaluation results of OSIRIS-SR show that support for reliable workflow execution does not significantly impact the system's scalability characteristics.","PeriodicalId":198333,"journal":{"name":"SWEET '13","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2013-06-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123500054","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
P. Neophytou, Panos K. Chrysanthis, Alexandros Labrinidis
Traditional workflow management or enactment systems (WfMS) and workflow design processes view the workflow as a one-time interaction with the various data sources, i.e., when a workflow is invoked, its steps are executed once and in-order. The fundamental underlying assumption has been that data sources are passive and all interactions are structured along the request/reply (query) model. Hence, traditional WfMS cannot effectively support business or scientific monitoring applications that require the processing of data streams such as those generated nowadays by sensing devices as well as mobile and web applications. Our hypothesis is that WfMS, both in the scientific and business domains, can be extended to support data stream semantics to enable monitoring applications. This includes the ability to apply flexible bounds on unbounded data streams and the ability to facilitate on-the-fly processing of bounded bundles of data (window semantics). In our previous work we have developed and implemented a Continuous Workflow Model that supports our hypothesis. This implementation of a CONtinuous workFLow ExeCution Engine (CONFLuEnCE) led to the realization that different applications have different performance requirements and hence an integrated workflow scheduling framework is essential. Such a framework is the main contribution of this paper. In particular, we designed and implemented STAFiLOS, a STreAm FLOw Scheduling for Continuous Workflows framework within CONFLuEnCE and evaluated STAFiLOS based on the Linear Road Benchmark.
{"title":"A continuous workflow scheduling framework","authors":"P. Neophytou, Panos K. Chrysanthis, Alexandros Labrinidis","doi":"10.1145/2499896.2499898","DOIUrl":"https://doi.org/10.1145/2499896.2499898","url":null,"abstract":"Traditional workflow management or enactment systems (WfMS) and workflow design processes view the workflow as a one-time interaction with the various data sources, i.e., when a workflow is invoked, its steps are executed once and in-order. The fundamental underlying assumption has been that data sources are passive and all interactions are structured along the request/reply (query) model. Hence, traditional WfMS cannot effectively support business or scientific monitoring applications that require the processing of data streams such as those generated nowadays by sensing devices as well as mobile and web applications.\u0000 Our hypothesis is that WfMS, both in the scientific and business domains, can be extended to support data stream semantics to enable monitoring applications. This includes the ability to apply flexible bounds on unbounded data streams and the ability to facilitate on-the-fly processing of bounded bundles of data (window semantics). In our previous work we have developed and implemented a Continuous Workflow Model that supports our hypothesis. This implementation of a CONtinuous workFLow ExeCution Engine (CONFLuEnCE) led to the realization that different applications have different performance requirements and hence an integrated workflow scheduling framework is essential. Such a framework is the main contribution of this paper. In particular, we designed and implemented STAFiLOS, a STreAm FLOw Scheduling for Continuous Workflows framework within CONFLuEnCE and evaluated STAFiLOS based on the Linear Road Benchmark.","PeriodicalId":198333,"journal":{"name":"SWEET '13","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2013-06-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132645276","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Mattoso, Kary A. C. S. Ocaña, Felipe Horta, Jonas Dias, Eduardo S. Ogasawara, V. S. Sousa, Daniel de Oliveira, F. Costa, Igor Araújo
In 2006 a group of leading researchers was gathered to discuss several challenges to scientific workflow supporting technologies and many of which still remain open challenges, such as the steering of workflows by users. Due to big data and long lasting workflows, many users demand steering features such as real-time monitoring, analysis and specially execution interference. The workflow execution should respond dynamically to such interference in the execution, to support the experimentation process in high performance computing. This paper revisits the issues in the user steering and dynamic workflows, presenting the state-of-the-art in it, and the open challenges. Our goal is to discuss research issues related to scientists' steering and present some ideas on how these demands may be supported in current scientific workflow technologies.
{"title":"User-steering of HPC workflows: state-of-the-art and future directions","authors":"M. Mattoso, Kary A. C. S. Ocaña, Felipe Horta, Jonas Dias, Eduardo S. Ogasawara, V. S. Sousa, Daniel de Oliveira, F. Costa, Igor Araújo","doi":"10.1145/2499896.2499900","DOIUrl":"https://doi.org/10.1145/2499896.2499900","url":null,"abstract":"In 2006 a group of leading researchers was gathered to discuss several challenges to scientific workflow supporting technologies and many of which still remain open challenges, such as the steering of workflows by users. Due to big data and long lasting workflows, many users demand steering features such as real-time monitoring, analysis and specially execution interference. The workflow execution should respond dynamically to such interference in the execution, to support the experimentation process in high performance computing. This paper revisits the issues in the user steering and dynamic workflows, presenting the state-of-the-art in it, and the open challenges. Our goal is to discuss research issues related to scientists' steering and present some ideas on how these demands may be supported in current scientific workflow technologies.","PeriodicalId":198333,"journal":{"name":"SWEET '13","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2013-06-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130718752","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：