首页 > 最新文献

Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems最新文献

英文 中文
Maximizing Determinism in Stream Processing Under Latency Constraints 延迟约束下流处理的确定性最大化
Nikos Zacheilas, V. Kalogeraki, Y. Nikolakopoulos, Vincenzo Gulisano, M. Papatriantafilou, P. Tsigas
The problem of coping with the demands of determinism and meeting latency constraints is challenging in distributed data stream processing systems that have to process high volume data streams that arrive from different unsynchronized input sources. In order to deterministically process the streaming data, they need mechanisms that synchronize the order in which tuples are processed by the operators. On the other hand, achieving real-time response in such a system requires careful tradeoff between determinism and low latency performance. We build on a recently proposed approach to handle data exchange and synchronization in stream processing, namely ScaleGate, which comes with guarantees for determinism and an efficient lock-free implementation, enabling high scalability. Considering the challenge and trade-offs implied by real-time constraints, we propose a system which comprises (a) a novel data structure called Slack-ScaleGate (SSG), along with its algorithmic implementation; SSG enables us to guarantee the deterministic processing of tuples as long as they are able to meet their latency constraints, and (b) a method to dynamically tune the maximum amount of time that a tuple can wait in the SSG data-structure, relaxing the determinism guarantees when needed, in order to satisfy the latency constraints. Our detailed experimental evaluation using a traffic monitoring application deployed in the city of Dublin, illustrates the working and benefits of our approach.
分布式数据流处理系统必须处理来自不同的非同步输入源的大容量数据流,因此处理确定性需求和满足延迟约束的问题具有挑战性。为了确定地处理流数据,他们需要同步操作符处理元组的顺序的机制。另一方面,在这样的系统中实现实时响应需要在确定性和低延迟性能之间进行仔细的权衡。我们基于最近提出的一种方法来处理流处理中的数据交换和同步,即ScaleGate,它具有确定性的保证和高效的无锁实现,从而实现高可扩展性。考虑到实时约束所隐含的挑战和权衡,我们提出了一个系统,它包括(a)一种称为松弛-尺度门(SSG)的新型数据结构及其算法实现;SSG使我们能够保证元组的确定性处理,只要它们能够满足其延迟约束,并且(b)一种动态调整元组在SSG数据结构中可以等待的最大时间的方法,在需要时放松确定性保证,以满足延迟约束。我们使用部署在都柏林市的交通监控应用程序进行了详细的实验评估,说明了我们的方法的工作和好处。
{"title":"Maximizing Determinism in Stream Processing Under Latency Constraints","authors":"Nikos Zacheilas, V. Kalogeraki, Y. Nikolakopoulos, Vincenzo Gulisano, M. Papatriantafilou, P. Tsigas","doi":"10.1145/3093742.3093921","DOIUrl":"https://doi.org/10.1145/3093742.3093921","url":null,"abstract":"The problem of coping with the demands of determinism and meeting latency constraints is challenging in distributed data stream processing systems that have to process high volume data streams that arrive from different unsynchronized input sources. In order to deterministically process the streaming data, they need mechanisms that synchronize the order in which tuples are processed by the operators. On the other hand, achieving real-time response in such a system requires careful tradeoff between determinism and low latency performance. We build on a recently proposed approach to handle data exchange and synchronization in stream processing, namely ScaleGate, which comes with guarantees for determinism and an efficient lock-free implementation, enabling high scalability. Considering the challenge and trade-offs implied by real-time constraints, we propose a system which comprises (a) a novel data structure called Slack-ScaleGate (SSG), along with its algorithmic implementation; SSG enables us to guarantee the deterministic processing of tuples as long as they are able to meet their latency constraints, and (b) a method to dynamically tune the maximum amount of time that a tuple can wait in the SSG data-structure, relaxing the determinism guarantees when needed, in order to satisfy the latency constraints. Our detailed experimental evaluation using a traffic monitoring application deployed in the city of Dublin, illustrates the working and benefits of our approach.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124085094","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 35
Hardware Accelerated Application Integration Processing: Industry Paper 硬件加速应用集成处理:工业论文
Daniel Ritter, Jonas Dann, Norman May, S. Rinderle-Ma
The growing number of (cloud) applications and devices massively increases the communication rate and volume pushing integration systems to their (throughput) limits. While the usage of modern hardware like Field Programmable Gate Arrays (FPGAs) led to low latency when employed for query and event processing, application integration adds yet unexplored processing opportunities. In this industry paper, we explore how to program integration semantics (e. g., message routing and transformation) in form of Enterprise Integration Patterns (EIP) on top of an FPGA, thus complementing the existing research on FPGA data processing. We focus on message routing, re-define the EIP for stream processing and propose modular hardware implementations as templates that are synthesized to circuits. For our real-world "connected car" scenario (i. e., composed patterns), we discuss common and new optimizations especially relevant for hardware integration processes. Our experimental evaluation shows competitive throughput compared to modern general-purpose CPUs and discusses the results.
越来越多的(云)应用程序和设备极大地提高了通信速率和容量,将集成系统推向了它们的(吞吐量)极限。虽然使用现场可编程门阵列(fpga)等现代硬件可以降低查询和事件处理的延迟,但应用程序集成增加了尚未开发的处理机会。在这篇行业论文中,我们探索了如何在FPGA之上以企业集成模式(EIP)的形式编程集成语义(如消息路由和转换),从而补充了现有的FPGA数据处理研究。我们专注于消息路由,重新定义流处理的EIP,并提出模块化硬件实现作为合成电路的模板。对于我们现实世界的“联网汽车”场景(即组合模式),我们讨论了与硬件集成过程特别相关的常见和新的优化。我们的实验评估显示了与现代通用cpu相比具有竞争力的吞吐量,并讨论了结果。
{"title":"Hardware Accelerated Application Integration Processing: Industry Paper","authors":"Daniel Ritter, Jonas Dann, Norman May, S. Rinderle-Ma","doi":"10.1145/3093742.3093911","DOIUrl":"https://doi.org/10.1145/3093742.3093911","url":null,"abstract":"The growing number of (cloud) applications and devices massively increases the communication rate and volume pushing integration systems to their (throughput) limits. While the usage of modern hardware like Field Programmable Gate Arrays (FPGAs) led to low latency when employed for query and event processing, application integration adds yet unexplored processing opportunities. In this industry paper, we explore how to program integration semantics (e. g., message routing and transformation) in form of Enterprise Integration Patterns (EIP) on top of an FPGA, thus complementing the existing research on FPGA data processing. We focus on message routing, re-define the EIP for stream processing and propose modular hardware implementations as templates that are synthesized to circuits. For our real-world \"connected car\" scenario (i. e., composed patterns), we discuss common and new optimizations especially relevant for hardware integration processes. Our experimental evaluation shows competitive throughput compared to modern general-purpose CPUs and discusses the results.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115596417","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 12
Building Connected Car Applications on Top of the World-Wide Streams Platform: Demo 基于全球流平台构建联网汽车应用:演示
W. V. Raemdonck, T. V. Cutsem, K. S. Esmaili, M. Cortes, P. Dobbelaere, L. Hoste, Eline Philips, M. Roelands, L. Trappeniers
The connected car is likely to play a fundamental role in the foreseeable Internet of Things. The connectivity aspect in combination with the available data (e.g. from GPS, on-board diagnostics, road sensors) and video (e.g. from dashcams and traffic cameras) streams enable a range of new applications, e.g., accident avoidance, online route planning, energy optimization, etc. These applications, however, come with an additional set of requirements which are not accommodated by the state-of-the-art stream processing platforms. We have built World-Wide Streams (WWS), a novel stream processing platform that has been explicitly designed with those requirements in mind. In this demo presentation, we will show a number of connected car scenarios that we have built on top of WWS.
联网汽车很可能在可预见的物联网中发挥基础性作用。连接方面与可用数据(例如来自GPS、车载诊断、道路传感器)和视频(例如来自行车记录仪和交通摄像头)流相结合,可以实现一系列新应用,例如事故避免、在线路线规划、能源优化等。然而,这些应用程序带来了一组额外的要求,这些要求是最先进的流处理平台所不能满足的。我们已经建立了全球流(WWS),这是一个新颖的流处理平台,在设计时明确考虑了这些需求。在这个演示中,我们将展示一些基于WWS构建的联网汽车场景。
{"title":"Building Connected Car Applications on Top of the World-Wide Streams Platform: Demo","authors":"W. V. Raemdonck, T. V. Cutsem, K. S. Esmaili, M. Cortes, P. Dobbelaere, L. Hoste, Eline Philips, M. Roelands, L. Trappeniers","doi":"10.1145/3093742.3095088","DOIUrl":"https://doi.org/10.1145/3093742.3095088","url":null,"abstract":"The connected car is likely to play a fundamental role in the foreseeable Internet of Things. The connectivity aspect in combination with the available data (e.g. from GPS, on-board diagnostics, road sensors) and video (e.g. from dashcams and traffic cameras) streams enable a range of new applications, e.g., accident avoidance, online route planning, energy optimization, etc. These applications, however, come with an additional set of requirements which are not accommodated by the state-of-the-art stream processing platforms. We have built World-Wide Streams (WWS), a novel stream processing platform that has been explicitly designed with those requirements in mind. In this demo presentation, we will show a number of connected car scenarios that we have built on top of WWS.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127994953","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 5
REMI, Reusable Elements for Multi-Level Information Availability: Demo REMI,多级信息可用性的可重用元素:演示
A. Gal, Nicolo Rivetti, Arik Senderovich, D. Gunopulos, I. Katakis, N. Panagiotou, V. Kalogeraki
Applications targeting Smart Cities tackle common challenges, however solutions are seldom portable from one city to another due to the heterogeneity of city ecosystems. A major obstacle involves the differences in the levels of available information. In this demonstration we present REMI, a reusable elements framework to handle varying degrees of information availability by design from two complementary angles, namely graceful degradation (GRADE) and data enrichment (DARE). In a nutshell, we develop reusable machine learning black boxes for mining and aggregating streaming data, either to infer missing data from available data, or to adapt expected accuracy based on data availability. We illustrate the proposed approach using tram data from the city of Warsaw.
针对智慧城市的应用解决了共同的挑战,但由于城市生态系统的异质性,解决方案很少从一个城市移植到另一个城市。一个主要的障碍是现有资料水平的不同。在这个演示中,我们介绍了REMI,一个可重用元素框架,通过设计从两个互补的角度处理不同程度的信息可用性,即优雅退化(GRADE)和数据丰富(DARE)。简而言之,我们开发了可重用的机器学习黑箱,用于挖掘和聚合流数据,要么从可用数据中推断缺失的数据,要么根据数据可用性调整预期的准确性。我们使用来自华沙的有轨电车数据来说明所提出的方法。
{"title":"REMI, Reusable Elements for Multi-Level Information Availability: Demo","authors":"A. Gal, Nicolo Rivetti, Arik Senderovich, D. Gunopulos, I. Katakis, N. Panagiotou, V. Kalogeraki","doi":"10.1145/3093742.3095091","DOIUrl":"https://doi.org/10.1145/3093742.3095091","url":null,"abstract":"Applications targeting Smart Cities tackle common challenges, however solutions are seldom portable from one city to another due to the heterogeneity of city ecosystems. A major obstacle involves the differences in the levels of available information. In this demonstration we present REMI, a reusable elements framework to handle varying degrees of information availability by design from two complementary angles, namely graceful degradation (GRADE) and data enrichment (DARE). In a nutshell, we develop reusable machine learning black boxes for mining and aggregating streaming data, either to infer missing data from available data, or to adapt expected accuracy based on data availability. We illustrate the proposed approach using tram data from the city of Warsaw.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126425612","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Cascading Data Corruption: About Dependencies in Cyber-Physical Systems: Poster 级联数据损坏:关于网络物理系统中的依赖关系:海报
Stefan Gries, M. Hesenius, V. Gruhn
CPS are interconnected systems that observe and manipulate real objects and processes. They allow dynamic extension and show emergent behavior which leads to dynamic decision-making processes that can change at runtime. They cannot always be easily understood because of the high number of components involved. If an error occurs in such a process, it is difficult to comprehend which component involved in the decision process is responsible for that error. The decision therefore has a high degree of dependency on the nodes involved in the process. Therefore, errors are not easily traceable to their original source. In this paper, we present the idea of dependency trees, which should help to identify error sources in the event of a fault.
CPS是相互连接的系统,可以观察和操作真实的对象和过程。它们允许动态扩展并显示导致动态决策过程的紧急行为,这些决策过程可以在运行时更改。它们并不总是容易理解,因为涉及的组件数量很多。如果在这样的过程中出现错误,很难理解决策过程中涉及的哪个组件对该错误负责。因此,决策高度依赖于流程中涉及的节点。因此,错误不容易追溯到其原始来源。在本文中,我们提出了依赖树的思想,它应该有助于在发生故障时识别错误源。
{"title":"Cascading Data Corruption: About Dependencies in Cyber-Physical Systems: Poster","authors":"Stefan Gries, M. Hesenius, V. Gruhn","doi":"10.1145/3093742.3095092","DOIUrl":"https://doi.org/10.1145/3093742.3095092","url":null,"abstract":"CPS are interconnected systems that observe and manipulate real objects and processes. They allow dynamic extension and show emergent behavior which leads to dynamic decision-making processes that can change at runtime. They cannot always be easily understood because of the high number of components involved. If an error occurs in such a process, it is difficult to comprehend which component involved in the decision process is responsible for that error. The decision therefore has a high degree of dependency on the nodes involved in the process. Therefore, errors are not easily traceable to their original source. In this paper, we present the idea of dependency trees, which should help to identify error sources in the event of a fault.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128046304","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 5
Consistent Stream Processing: Doctoral Symposium 一致性流处理:博士研讨会
Lorenzo Affetti
Stream Processors (SPs) continuously transform huge volumes of input streams with a computational model that is inherently distributed, scalable, and fault-tolerant. For these reasons they are used in application environments in which almost real-time computation is of paramount importance, such as stock option analysis, fraud detection systems, monitoring, and real-time data analytics for web applications. In many applicative domains, SPs are used in conjunction with data management systems such as transactional databases and data warehouses that store intermediate or final results produced by the SPs. However, SPs have no control on the consistency guarantees of the results produced on external components. We propose a novel approach that we name consistent stream processing that integrates the external state of databases within the SP and enforces consistency guarantees both on state updates and on external querying. We extend the computational model of SPs with transactions and we provide two possible strategies to enforce their transactional properties.
流处理器(SPs)使用固有的分布式、可扩展和容错的计算模型不断地转换大量的输入流。由于这些原因,它们被用于几乎实时计算至关重要的应用程序环境中,例如股票期权分析、欺诈检测系统、监控和web应用程序的实时数据分析。在许多应用领域中,sp与数据管理系统(如存储sp产生的中间或最终结果的事务数据库和数据仓库)一起使用。然而,sp无法控制外部组件上产生的结果的一致性保证。我们提出了一种新的方法,我们将其命名为一致流处理,它将数据库的外部状态集成到SP中,并在状态更新和外部查询上强制一致性保证。我们用事务扩展了sp的计算模型,并提供了两种可能的策略来执行它们的事务属性。
{"title":"Consistent Stream Processing: Doctoral Symposium","authors":"Lorenzo Affetti","doi":"10.1145/3093742.3093900","DOIUrl":"https://doi.org/10.1145/3093742.3093900","url":null,"abstract":"Stream Processors (SPs) continuously transform huge volumes of input streams with a computational model that is inherently distributed, scalable, and fault-tolerant. For these reasons they are used in application environments in which almost real-time computation is of paramount importance, such as stock option analysis, fraud detection systems, monitoring, and real-time data analytics for web applications. In many applicative domains, SPs are used in conjunction with data management systems such as transactional databases and data warehouses that store intermediate or final results produced by the SPs. However, SPs have no control on the consistency guarantees of the results produced on external components. We propose a novel approach that we name consistent stream processing that integrates the external state of databases within the SP and enforces consistency guarantees both on state updates and on external querying. We extend the computational model of SPs with transactions and we provide two possible strategies to enforce their transactional properties.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131934885","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 2
Data Streaming and its Application to Stream Processing: Tutorial 数据流及其在流处理中的应用教程
Leonardo Querzoni, Nicolo Rivetti
In this tutorial paper we present the results of recent research findings in the area of data streaming applied to stream processing systems. In particular, we introduce the data streaming model, detailing the main algorithmic results in this research field. We then move to detail how such algorithms can be applied to modern distributed stream processing systems to improve their efficiency. Finally we outline several open research directions in this field.
在这篇教程中,我们介绍了应用于流处理系统的数据流领域的最新研究成果。特别介绍了数据流模型,详细介绍了该研究领域的主要算法成果。然后,我们将详细介绍如何将这些算法应用于现代分布式流处理系统以提高其效率。最后提出了该领域的几个开放研究方向。
{"title":"Data Streaming and its Application to Stream Processing: Tutorial","authors":"Leonardo Querzoni, Nicolo Rivetti","doi":"10.1145/3093742.3095108","DOIUrl":"https://doi.org/10.1145/3093742.3095108","url":null,"abstract":"In this tutorial paper we present the results of recent research findings in the area of data streaming applied to stream processing systems. In particular, we introduce the data streaming model, detailing the main algorithmic results in this research field. We then move to detail how such algorithms can be applied to modern distributed stream processing systems to improve their efficiency. Finally we outline several open research directions in this field.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116973893","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
Challenges to Achieving High Availability at Scale 实现大规模高可用性的挑战
W. Shulte
Facebook is a social network that connects more than 1.8 billion people. To serve these many users requires infrastructure which is composed of thousands of interdependent systems that span geographically distributed data centers. But what is the guiding principle for building and operating these systems? For Facebook’s infrastructure teams the answer is: Systems must always be available and never lose data. This talk will explore this quest. We will focus on three aspects. Availability and consistency. What form of consistency do Facebook’s systems guarantee? Strong consistency makes understanding easy but has latency penalties, weak consistency is fast but difficult to reason for developers and users. We describe our usage of eventual consistency and delve into how Facebook constructs its caching and replicated storage systems to minimize the duration for achieving consistency. We share empirical data that measures the effectiveness of our design. Availability and correctness. With network partitions, relaxed forms of consistency, and software bugs, how do we guarantee a consistent state? We present two systems to find and repair structural errors in Facebook’s social graph, one batch and one real-time. Availability and scale. Sharding is one of the standard answers to operate at scale. But how can we develop one system that can shard storage as well as compute? We will introduce a new Sharding-as-a-Service component. We will show and evaluate how its design and service policies control for latency, failure tolerance and operationally efficiency. 1998 ACM Subject Classification Computer; C 1.4 Distributed Architectures; C.2.4 Distributed Systems; C.4 Fault Tolerance, Reliability, Availability and Serviceability; D 1.3 Distributed Programming; D 4.7 Distributed Systems; E 1 Distributed Data Structures
Facebook是一个连接超过18亿人的社交网络。为了服务这么多用户,需要由数千个相互依赖的系统组成的基础设施,这些系统跨越地理分布的数据中心。但是,构建和操作这些系统的指导原则是什么呢?对于Facebook的基础设施团队来说,答案是:系统必须始终可用,永远不会丢失数据。这次演讲将探讨这个问题。我们将重点从三个方面着手。可用性和一致性。Facebook的系统保证了什么形式的一致性?强一致性使理解变得容易,但有延迟的代价,弱一致性很快,但对开发人员和用户来说很难理解。我们描述了最终一致性的使用,并深入研究了Facebook如何构建其缓存和复制存储系统,以最大限度地减少实现一致性的持续时间。我们分享经验数据来衡量我们设计的有效性。可用性和正确性。对于网络分区、松散形式的一致性和软件bug,我们如何保证一致的状态?我们提出了两个系统来发现和修复Facebook社交图谱中的结构性错误,一个是批量的,一个是实时的。可用性和规模。分片是大规模操作的标准解决方案之一。但是,我们如何开发一个既能共享存储又能共享计算的系统呢?我们将引入一个新的分片即服务组件。我们将展示并评估其设计和服务策略如何控制延迟、容错和操作效率。1998 ACM主题分类计算机;C 1.4分布式体系结构;C.2.4分布式系统;C.4容错、可靠性、可用性和可服务性;1.3分布式编程;d4.7分布式系统;e1分布式数据结构
{"title":"Challenges to Achieving High Availability at Scale","authors":"W. Shulte","doi":"10.1145/3093742.3097270","DOIUrl":"https://doi.org/10.1145/3093742.3097270","url":null,"abstract":"Facebook is a social network that connects more than 1.8 billion people. To serve these many users requires infrastructure which is composed of thousands of interdependent systems that span geographically distributed data centers. But what is the guiding principle for building and operating these systems? For Facebook’s infrastructure teams the answer is: Systems must always be available and never lose data. This talk will explore this quest. We will focus on three aspects. Availability and consistency. What form of consistency do Facebook’s systems guarantee? Strong consistency makes understanding easy but has latency penalties, weak consistency is fast but difficult to reason for developers and users. We describe our usage of eventual consistency and delve into how Facebook constructs its caching and replicated storage systems to minimize the duration for achieving consistency. We share empirical data that measures the effectiveness of our design. Availability and correctness. With network partitions, relaxed forms of consistency, and software bugs, how do we guarantee a consistent state? We present two systems to find and repair structural errors in Facebook’s social graph, one batch and one real-time. Availability and scale. Sharding is one of the standard answers to operate at scale. But how can we develop one system that can shard storage as well as compute? We will introduce a new Sharding-as-a-Service component. We will show and evaluate how its design and service policies control for latency, failure tolerance and operationally efficiency. 1998 ACM Subject Classification Computer; C 1.4 Distributed Architectures; C.2.4 Distributed Systems; C.4 Fault Tolerance, Reliability, Availability and Serviceability; D 1.3 Distributed Programming; D 4.7 Distributed Systems; E 1 Distributed Data Structures","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115845317","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Human Body Related Event Processing: Tutorial 人体相关事件处理:教程
O. Etzion
Current developments both in the area of sensors and actuators that can be planted within the human body, along with the advancements in the areas of big data analysis, and autonomous intelligent systems serves as a major revolution the approach to healthcare. Furthermore, it has a potential to make a major revolution in life as we know it by providing the gate to singularity that brings with it eternal (or at least very long) life. This is part of a general trend of autonomous systems that brings hope for the next level in the human evolution along with paradoxes and dilemmas. The tutorial concentrates upon the description of the current development that enable event processing within the human body, the notion of body area networks in general, and applications of this area to medical and non-medical applications. Some predictions about future trends in this area, especially the vision of singularity are discussed, and some philosophical observations about the implications are also discussed in detail
目前在可植入人体的传感器和执行器领域的发展,以及大数据分析和自主智能系统领域的进步,都是医疗保健方法的重大革命。此外,正如我们所知,它有可能通过提供通往奇点的大门,给生命带来永恒(或至少是非常长的)生命,从而给生命带来重大革命。这是自主系统总体趋势的一部分,它为人类进化的下一个阶段带来了希望,同时也带来了悖论和困境。本教程将重点介绍使人体内事件处理成为可能的当前发展、一般身体区域网络的概念,以及该领域在医疗和非医疗应用中的应用。对这一领域的未来趋势进行了一些预测,特别是对奇点的看法进行了讨论,并对其含义进行了详细的哲学观察
{"title":"Human Body Related Event Processing: Tutorial","authors":"O. Etzion","doi":"10.1145/3093742.3095109","DOIUrl":"https://doi.org/10.1145/3093742.3095109","url":null,"abstract":"Current developments both in the area of sensors and actuators that can be planted within the human body, along with the advancements in the areas of big data analysis, and autonomous intelligent systems serves as a major revolution the approach to healthcare. Furthermore, it has a potential to make a major revolution in life as we know it by providing the gate to singularity that brings with it eternal (or at least very long) life. This is part of a general trend of autonomous systems that brings hope for the next level in the human evolution along with paradoxes and dilemmas. The tutorial concentrates upon the description of the current development that enable event processing within the human body, the notion of body area networks in general, and applications of this area to medical and non-medical applications. Some predictions about future trends in this area, especially the vision of singularity are discussed, and some philosophical observations about the implications are also discussed in detail","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114997609","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
SecureStreams: A Reactive Middleware Framework for Secure Data Stream Processing SecureStreams:用于安全数据流处理的响应式中间件框架
Aurélien Havet, Rafael Pires, P. Felber, Marcelo Pasin, Romain Rouvoy, V. Schiavoni
The growing adoption of distributed data processing frameworks in a wide diversity of application domains challenges end-to-end integration of properties like security, in particular when considering deployments in the context of large-scale clusters or multi-tenant Cloud infrastructures. This paper therefore introduces SecureStreams, a reactive middleware framework to deploy and process secure streams at scale. Its design combines the high-level reactive dataflow programming paradigm with Intel®'s low-level software guard extensions (SGX) in order to guarantee privacy and integrity of the processed data. The experimental results of SecureStreams are promising: while offering a fluent scripting language based on Lua, our middleware delivers high processing throughput, thus enabling developers to implement secure processing pipelines in just few lines of code.
分布式数据处理框架在各种应用程序领域的日益普及,对安全性等属性的端到端集成提出了挑战,特别是在考虑大规模集群或多租户云基础设施环境中的部署时。因此,本文介绍了SecureStreams,这是一个响应式中间件框架,用于大规模部署和处理安全流。它的设计结合了高层次的反应数据流编程范式与英特尔®的低级软件保护扩展(SGX),以保证隐私和完整性的处理数据。SecureStreams的实验结果很有希望:在提供基于Lua的流畅脚本语言的同时,我们的中间件提供了高处理吞吐量,从而使开发人员能够在几行代码中实现安全的处理管道。
{"title":"SecureStreams: A Reactive Middleware Framework for Secure Data Stream Processing","authors":"Aurélien Havet, Rafael Pires, P. Felber, Marcelo Pasin, Romain Rouvoy, V. Schiavoni","doi":"10.1145/3093742.3093927","DOIUrl":"https://doi.org/10.1145/3093742.3093927","url":null,"abstract":"The growing adoption of distributed data processing frameworks in a wide diversity of application domains challenges end-to-end integration of properties like security, in particular when considering deployments in the context of large-scale clusters or multi-tenant Cloud infrastructures. This paper therefore introduces SecureStreams, a reactive middleware framework to deploy and process secure streams at scale. Its design combines the high-level reactive dataflow programming paradigm with Intel®'s low-level software guard extensions (SGX) in order to guarantee privacy and integrity of the processed data. The experimental results of SecureStreams are promising: while offering a fluent scripting language based on Lua, our middleware delivers high processing throughput, thus enabling developers to implement secure processing pipelines in just few lines of code.","PeriodicalId":325666,"journal":{"name":"Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124317525","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 30
期刊
Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1