首页 > 最新文献

Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond最新文献

英文 中文
Cross-system NoSQL data transformations with NotaQL 使用NotaQL进行跨系统NoSQL数据转换
Johannes Schildgen, Thomas Lottermann, S. Deßloch
The rising adoption of NoSQL technology in enterprises causes a heterogeneous landscape of different data stores. Different stores provide distinct advantages and disadvantages, making it necessary for enterprises to facilitate multiple systems for specific purposes. This resulting polyglot persistence is difficult to handle for developers since some data needs to be replicated and aggregated between different and within the same stores. Currently, there are no uniform tools to perform these data transformations since all stores feature different APIs and data models. In this paper, we present the transformation language NotaQL that allows cross-system data transformations. These transformations are output-oriented, meaning that the structure of a transformation script is similar to that of the output. Besides, we provide an aggregation-centric approach, which makes aggregation operations as easy as possible.
企业中越来越多地采用NoSQL技术导致了不同数据存储的异构环境。不同的商店有不同的优点和缺点,这使得企业有必要为特定的目的提供多个系统。对于开发人员来说,这种多语言持久性很难处理,因为需要在不同存储和相同存储之间复制和聚合一些数据。目前,没有统一的工具来执行这些数据转换,因为所有存储都具有不同的api和数据模型。在本文中,我们介绍了允许跨系统数据转换的转换语言NotaQL。这些转换是面向输出的,这意味着转换脚本的结构与输出的结构相似。此外,我们还提供了以聚合为中心的方法,使聚合操作尽可能简单。
{"title":"Cross-system NoSQL data transformations with NotaQL","authors":"Johannes Schildgen, Thomas Lottermann, S. Deßloch","doi":"10.1145/2926534.2926535","DOIUrl":"https://doi.org/10.1145/2926534.2926535","url":null,"abstract":"The rising adoption of NoSQL technology in enterprises causes a heterogeneous landscape of different data stores. Different stores provide distinct advantages and disadvantages, making it necessary for enterprises to facilitate multiple systems for specific purposes. This resulting polyglot persistence is difficult to handle for developers since some data needs to be replicated and aggregated between different and within the same stores. Currently, there are no uniform tools to perform these data transformations since all stores feature different APIs and data models. In this paper, we present the transformation language NotaQL that allows cross-system data transformations. These transformations are output-oriented, meaning that the structure of a transformation script is similar to that of the output. Besides, we provide an aggregation-centric approach, which makes aggregation operations as easy as possible.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132641970","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 12
Deterministic load balancing for parallel joins 并行连接的确定性负载平衡
Paraschos Koutris, Nivetha Singara Vadivelu
We study the problem of distributing the tuples of a relation to a number of processors organized in an r-dimensional hypercube, which is an important task for parallel join processing. In contrast to previous work, which proposed randomized algorithms for the task, we ask here the question of how to construct efficient deterministic distribution strategies that can optimally load balance the input relation. We first present some general lower bounds on the load for any dimension; these bounds depend not only on the size of the relation, but also on the maximum frequency of each value in the relation. We then construct an algorithm for the case of 1 dimension that is optimal within a constant factor, and an algorithm for the case of 2 dimensions that is optimal within a polylogarithmic factor. Our 2-dimensional algorithm is based on an interesting connection with the vector load balancing problem, a well-studied problem that generalizes classic load balancing.
研究了将关系元组分配给组织在r维超立方体中的多个处理器的问题,这是并行连接处理的一个重要问题。相比以前的工作,提出了随机算法的任务,我们问的问题是如何构造有效的确定性分销策略可以优化输入的负载平衡关系。我们首先给出了任意维度载荷的一般下界;这些边界不仅取决于关系的大小,还取决于关系中每个值的最大频率。然后,我们构建了一个在常数因子内最优的一维情况下的算法,以及一个在多对数因子内最优的二维情况下的算法。我们的二维算法基于与矢量负载平衡问题的一个有趣的联系,矢量负载平衡问题是一个得到充分研究的问题,它概括了经典的负载平衡。
{"title":"Deterministic load balancing for parallel joins","authors":"Paraschos Koutris, Nivetha Singara Vadivelu","doi":"10.1145/2926534.2926536","DOIUrl":"https://doi.org/10.1145/2926534.2926536","url":null,"abstract":"We study the problem of distributing the tuples of a relation to a number of processors organized in an r-dimensional hypercube, which is an important task for parallel join processing. In contrast to previous work, which proposed randomized algorithms for the task, we ask here the question of how to construct efficient deterministic distribution strategies that can optimally load balance the input relation. We first present some general lower bounds on the load for any dimension; these bounds depend not only on the size of the relation, but also on the maximum frequency of each value in the relation. We then construct an algorithm for the case of 1 dimension that is optimal within a constant factor, and an algorithm for the case of 2 dimensions that is optimal within a polylogarithmic factor. Our 2-dimensional algorithm is based on an interesting connection with the vector load balancing problem, a well-studied problem that generalizes classic load balancing.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127910723","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Bridging the gap: towards optimization across linear and relational algebra 弥合差距:朝着优化跨越线性和关系代数
Andreas Kunft, Alexander B. Alexandrov, Asterios Katsifodimos, V. Markl
Advanced data analysis typically requires some form of pre-processing in order to extract and transform data before processing it with machine learning and statistical analysis techniques. Pre-processing pipelines are naturally expressed in dataflow APIs (e.g., MapReduce, Flink, etc.), while machine learning is expressed in linear algebra with iterations. Programmers therefore perform end-to-end data analysis utilizing multiple programming paradigms and systems. This impedance mismatch not only hinders productivity but also prevents optimization opportunities, such as sharing of physical data layouts (e.g., partitioning) and data structures among different parts of a data analysis program. The goal of this work is twofold. First, it aims to alleviate the impedance mismatch by allowing programmers to author complete end-to-end programs in one engine-independent language that is automatically parallelized. Second, it aims to enable joint optimizations over both relational and linear algebra. To achieve this goal, we present the design of Lara, a deeply embedded language in Scala which enables authoring scalable programs using two abstract data types (DataBag and Matrix) and control flow constructs. Programs written in Lara are compiled to an intermediate representation (IR) which enables optimizations across linear and relational algebra. The IR is finally used to compile code for different execution engines.
高级数据分析通常需要某种形式的预处理,以便在使用机器学习和统计分析技术处理数据之前提取和转换数据。预处理管道用数据流api(如MapReduce、Flink等)自然表达,而机器学习用迭代的线性代数表达。因此,程序员利用多种编程范例和系统执行端到端的数据分析。这种阻抗不匹配不仅阻碍了生产力,而且还阻碍了优化机会,例如在数据分析程序的不同部分之间共享物理数据布局(例如,分区)和数据结构。这项工作的目标是双重的。首先,它的目标是通过允许程序员用一种独立于引擎的自动并行语言编写完整的端到端程序来减轻阻抗不匹配。其次,它旨在实现关系代数和线性代数的联合优化。为了实现这一目标,我们介绍了Lara的设计,这是Scala中的一种深度嵌入式语言,可以使用两种抽象数据类型(DataBag和Matrix)和控制流构造来创作可扩展的程序。用Lara编写的程序被编译成一种中间表示(IR),它可以跨线性和关系代数进行优化。最后,IR用于编译不同执行引擎的代码。
{"title":"Bridging the gap: towards optimization across linear and relational algebra","authors":"Andreas Kunft, Alexander B. Alexandrov, Asterios Katsifodimos, V. Markl","doi":"10.1145/2926534.2926540","DOIUrl":"https://doi.org/10.1145/2926534.2926540","url":null,"abstract":"Advanced data analysis typically requires some form of pre-processing in order to extract and transform data before processing it with machine learning and statistical analysis techniques. Pre-processing pipelines are naturally expressed in dataflow APIs (e.g., MapReduce, Flink, etc.), while machine learning is expressed in linear algebra with iterations. Programmers therefore perform end-to-end data analysis utilizing multiple programming paradigms and systems. This impedance mismatch not only hinders productivity but also prevents optimization opportunities, such as sharing of physical data layouts (e.g., partitioning) and data structures among different parts of a data analysis program. The goal of this work is twofold. First, it aims to alleviate the impedance mismatch by allowing programmers to author complete end-to-end programs in one engine-independent language that is automatically parallelized. Second, it aims to enable joint optimizations over both relational and linear algebra. To achieve this goal, we present the design of Lara, a deeply embedded language in Scala which enables authoring scalable programs using two abstract data types (DataBag and Matrix) and control flow constructs. Programs written in Lara are compiled to an intermediate representation (IR) which enables optimizations across linear and relational algebra. The IR is finally used to compile code for different execution engines.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127134090","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 37
On exploring efficient shuffle design for in-memory MapReduce 基于内存MapReduce的高效shuffle设计研究
Harunobu Daikoku, H. Kawashima, O. Tatebe
MapReduce is commonly used as a way of big data analysis in many fields. Shuffling, the inter-node data exchange phase of MapReduce, has been reported as the major bottleneck of the framework. Acceleration of shuffling has been studied in literature, and we raise two questions in this paper. The first question pertains to the effect of Remote Direct Memory Access (RDMA) on the performance of shuffling. RDMA enables one machine to read and write data on the local memory of another and has been known to be an efficient data transfer mechanism. Does the pure use of RDMA affect the performance of shuffling? The second question is the data transfer algorithm to use. There are two types of shuffling algorithms for the conventional MapReduce implementations: Fully-Connected and more sophisticated algorithms such as Pairwise. Does the data transfer algorithm affect the performance of shuffling? To answer these questions, we designed and implemented yet another MapReduce system from scratch in C/C++ to gain the maximum performance and to reserve design flexibility. For the first question, we compared RDMA shuffling based on rsocket with the one based on IPoIB. The results of experiments with GroupBy showed that RDMA accelerates map+shuffle phase by around 50%. For the second question, we first compared our in-memory system with Apache Spark to investigate whether our system performed more efficiently than the existing system. Our system demonstrated performance improvement by a factor of 3.04 on Word Count, and by a factor of 2.64 on BiGram Count as compared to Spark. Then, we compared the two data exchange algorithms, Fully-Connected and Pairwise. The results of experiments with BiGram Count showed that Fully-Connected without RDMA was 13% more efficient than Pairwise with RDMA. We conclude that it is necessary to overlap map and shuffle phases to gain performance improvement. The reason of the relatively small percentage of improvement can be attributed to the time-consuming insertions of key-value pairs into the hash-map in the map phase.
MapReduce在很多领域都是常用的大数据分析方法。据报道,MapReduce节点间数据交换阶段的变换是该框架的主要瓶颈。文献对洗牌的加速进行了研究,本文提出了两个问题。第一个问题是关于远程直接内存访问(RDMA)对变换性能的影响。RDMA使一台机器能够在另一台机器的本地内存上读写数据,并且已经被认为是一种有效的数据传输机制。单纯使用RDMA会影响变换的性能吗?第二个问题是要使用的数据传输算法。对于传统的MapReduce实现,有两种类型的变换算法:完全连接和更复杂的算法,如Pairwise。数据传输算法是否影响洗牌的性能?为了回答这些问题,我们用C/ c++从头开始设计并实现了另一个MapReduce系统,以获得最大的性能并保留设计灵活性。对于第一个问题,我们比较了基于rsocket的RDMA变换和基于IPoIB的RDMA变换。GroupBy的实验结果表明,RDMA将map+shuffle阶段加速了约50%。对于第二个问题,我们首先将我们的内存系统与Apache Spark进行比较,以调查我们的系统是否比现有系统执行得更有效。与Spark相比,我们的系统在Word Count上的性能提高了3.04倍,在BiGram Count上的性能提高了2.64倍。然后,我们比较了两种数据交换算法,Fully-Connected和Pairwise。使用BiGram Count进行的实验结果表明,不使用RDMA的full - connected比使用RDMA的Pairwise效率高13%。我们得出结论,有必要重叠映射和shuffle阶段以获得性能改进。改进百分比相对较小的原因可以归结为在映射阶段将键值对插入到哈希映射中非常耗时。
{"title":"On exploring efficient shuffle design for in-memory MapReduce","authors":"Harunobu Daikoku, H. Kawashima, O. Tatebe","doi":"10.1145/2926534.2926538","DOIUrl":"https://doi.org/10.1145/2926534.2926538","url":null,"abstract":"MapReduce is commonly used as a way of big data analysis in many fields. Shuffling, the inter-node data exchange phase of MapReduce, has been reported as the major bottleneck of the framework. Acceleration of shuffling has been studied in literature, and we raise two questions in this paper. The first question pertains to the effect of Remote Direct Memory Access (RDMA) on the performance of shuffling. RDMA enables one machine to read and write data on the local memory of another and has been known to be an efficient data transfer mechanism. Does the pure use of RDMA affect the performance of shuffling? The second question is the data transfer algorithm to use. There are two types of shuffling algorithms for the conventional MapReduce implementations: Fully-Connected and more sophisticated algorithms such as Pairwise. Does the data transfer algorithm affect the performance of shuffling? To answer these questions, we designed and implemented yet another MapReduce system from scratch in C/C++ to gain the maximum performance and to reserve design flexibility. For the first question, we compared RDMA shuffling based on rsocket with the one based on IPoIB. The results of experiments with GroupBy showed that RDMA accelerates map+shuffle phase by around 50%. For the second question, we first compared our in-memory system with Apache Spark to investigate whether our system performed more efficiently than the existing system. Our system demonstrated performance improvement by a factor of 3.04 on Word Count, and by a factor of 2.64 on BiGram Count as compared to Spark. Then, we compared the two data exchange algorithms, Fully-Connected and Pairwise. The results of experiments with BiGram Count showed that Fully-Connected without RDMA was 13% more efficient than Pairwise with RDMA. We conclude that it is necessary to overlap map and shuffle phases to gain performance improvement. The reason of the relatively small percentage of improvement can be attributed to the time-consuming insertions of key-value pairs into the hash-map in the map phase.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124850305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
Faucet: a user-level, modular technique for flow control in dataflow engines 水龙头:数据流引擎中用于流量控制的用户级模块化技术
Andrea Lattuada, Frank McSherry, Zaheer Chothia
This document presents Faucet, a modular flow control approach for distributed data-parallel dataflow engines with support for arbitrary (cyclic) topologies. When compared to existing backpressure techniques Faucet has the following differentiating characteristics: (i) the implementation only relies on existing progress information exposed by the system and does not require changes to the underlying dataflow system, (ii) it can be applied selectively to certain parts of the dataflow graph, and (iii) it is designed to support a wide variety of use cases, topologies and workloads. We demonstrate Faucet on an example computation for efficiently determining a cyclic join of relations, whose variability in rates of produced and consumed tuples challenges the flow control techniques employed by systems like Storm, Heron, and Spark. Our implementation, prototyped in Timely Dataflow, introduces flow control at critical locations in the computation, keeping the computation stable and resource-bound while introducing at most 20% runtime overhead over an unconstrained implementation. Our experience is that the information Timely Dataflow provides to user logic is sufficient for a variety of flow control and scheduling tasks, and merits further investigation.
本文档介绍了Faucet,一种支持任意(循环)拓扑的分布式数据并行数据流引擎的模块化流控制方法。与现有的背压技术相比,Faucet具有以下区别特征:(i)实现仅依赖于系统暴露的现有进度信息,不需要更改底层数据流系统,(ii)它可以选择性地应用于数据流图的某些部分,(iii)它旨在支持各种各样的用例,拓扑结构和工作负载。我们通过一个示例计算来演示Faucet,以有效地确定关系的循环连接,其产生和消耗元组的速率的可变性挑战了诸如Storm, Heron和Spark等系统所采用的流控制技术。我们的实现,在及时数据流中原型化,在计算中的关键位置引入了流控制,保持计算稳定和资源约束,同时引入了比无约束实现最多20%的运行时开销。我们的经验是,及时数据流提供给用户逻辑的信息对于各种流控制和调度任务来说是足够的,值得进一步研究。
{"title":"Faucet: a user-level, modular technique for flow control in dataflow engines","authors":"Andrea Lattuada, Frank McSherry, Zaheer Chothia","doi":"10.1145/2926534.2926544","DOIUrl":"https://doi.org/10.1145/2926534.2926544","url":null,"abstract":"This document presents Faucet, a modular flow control approach for distributed data-parallel dataflow engines with support for arbitrary (cyclic) topologies. When compared to existing backpressure techniques Faucet has the following differentiating characteristics: (i) the implementation only relies on existing progress information exposed by the system and does not require changes to the underlying dataflow system, (ii) it can be applied selectively to certain parts of the dataflow graph, and (iii) it is designed to support a wide variety of use cases, topologies and workloads. We demonstrate Faucet on an example computation for efficiently determining a cyclic join of relations, whose variability in rates of produced and consumed tuples challenges the flow control techniques employed by systems like Storm, Heron, and Spark. Our implementation, prototyped in Timely Dataflow, introduces flow control at critical locations in the computation, keeping the computation stable and resource-bound while introducing at most 20% runtime overhead over an unconstrained implementation. Our experience is that the information Timely Dataflow provides to user logic is sufficient for a variety of flow control and scheduling tasks, and merits further investigation.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125641213","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 12
Tight bounds on one- and two-pass MapReduce algorithms for matrix multiplication 矩阵乘法的一遍和两遍MapReduce算法的紧密边界
Prakash V. Ramanan, A. Nagar
We study one- and two-pass mapReduce algorithms for multiplying two matrices. First, consider one-pass algorithms. In the literature, there is a tight bound for the tradeoff between communication cost and parallelism. It measures communication cost using the replication rate r, and measures parallelism by reducer size q. It gives a tight bound on qr for multiplying dense square matrices. We extend it in two different ways: First, to sparse rectangular matrices; second, to a different measure of parallelism, namely, reducer workload w. We present tight bounds on qr and wr2, for multiplying sparse rectangular matrices. We also show that the lower bound on qr follows from the lower bound on wr2; so, the lower bound on wr2 is stronger. Next, consider two-pass algorithms. It has been shown that, for a given reducer size, the two-pass algorithm has less communication cost than the one-pass algorithm. We present tight bounds on qfrfrs and wfr2frs, for multiplying dense rectangular matrices; the subscripts f and s correspond to the first and second pass, respectively. Also, using our bound on qfrfrs, we present a tight bound on the total communication cost as a function of qf. Our lower bounds hold for the class of two-pass algorithms that perform all the real number multiplications in the first pass.
我们研究了两个矩阵相乘的一次和两次mapReduce算法。首先,考虑一遍算法。在文献中,对于通信成本和并行性之间的权衡有一个严格的界限。它使用复制速率r来衡量通信成本,并通过减速机大小q来衡量并行度。它给出了稠密方阵乘法的严密边界qr。我们以两种不同的方式将其扩展:首先,扩展到稀疏矩形矩阵;其次,对于不同的并行度度量,即减速器工作负载w。我们给出了qr和wr2的紧界,用于稀疏矩形矩阵的乘法。我们还证明了qr的下界是从wr2的下界推导出来的;所以,wr2的下界更强。接下来,考虑两步算法。研究表明,对于给定的减速机大小,两遍算法比一遍算法具有更小的通信开销。对于密集矩形矩阵的乘法,我们给出了qfrfrs和wfr2frs的紧界;下标f和s分别对应于第一次和第二次传递。此外,利用qfrfrs的界,我们给出了总通信成本作为qf函数的紧界。我们的下界适用于在第一次执行所有实数乘法的两遍算法。
{"title":"Tight bounds on one- and two-pass MapReduce algorithms for matrix multiplication","authors":"Prakash V. Ramanan, A. Nagar","doi":"10.1145/2926534.2926542","DOIUrl":"https://doi.org/10.1145/2926534.2926542","url":null,"abstract":"We study one- and two-pass mapReduce algorithms for multiplying two matrices. First, consider one-pass algorithms. In the literature, there is a tight bound for the tradeoff between communication cost and parallelism. It measures communication cost using the replication rate r, and measures parallelism by reducer size q. It gives a tight bound on qr for multiplying dense square matrices. We extend it in two different ways: First, to sparse rectangular matrices; second, to a different measure of parallelism, namely, reducer workload w. We present tight bounds on qr and wr2, for multiplying sparse rectangular matrices. We also show that the lower bound on qr follows from the lower bound on wr2; so, the lower bound on wr2 is stronger. Next, consider two-pass algorithms. It has been shown that, for a given reducer size, the two-pass algorithm has less communication cost than the one-pass algorithm. We present tight bounds on qfrfrs and wfr2frs, for multiplying dense rectangular matrices; the subscripts f and s correspond to the first and second pass, respectively. Also, using our bound on qfrfrs, we present a tight bound on the total communication cost as a function of qf. Our lower bounds hold for the class of two-pass algorithms that perform all the real number multiplications in the first pass.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121537439","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
DFA minimization in map-reduce map-reduce中的DFA最小化
G. Grahne, Shahab Harrafi, Iraj Hedayati, A. Moallemi
We describe Map-Reduce implementations of two of the most prominent DFA minimization methods, namely Moore's and Hopcroft's algorithms. Our analysis shows that the one based on Hopcroft's algorithm is more efficient, both in terms of running time and communication cost. This is validated by our extensive experiments on various types of DFA's, with up to 217 states. It also turns out that both algorithms are sensitive to skewed input, the Hopcroft's algorithm being intrinsically so.
我们描述了两种最突出的DFA最小化方法的Map-Reduce实现,即摩尔算法和霍普克罗夫特算法。我们的分析表明,基于Hopcroft算法的算法在运行时间和通信成本方面都更加高效。我们对多达217种状态的各种DFA进行了广泛的实验,验证了这一点。结果还表明,这两种算法对倾斜输入都很敏感,Hopcroft算法本质上就是这样。
{"title":"DFA minimization in map-reduce","authors":"G. Grahne, Shahab Harrafi, Iraj Hedayati, A. Moallemi","doi":"10.1145/2926534.2926537","DOIUrl":"https://doi.org/10.1145/2926534.2926537","url":null,"abstract":"We describe Map-Reduce implementations of two of the most prominent DFA minimization methods, namely Moore's and Hopcroft's algorithms. Our analysis shows that the one based on Hopcroft's algorithm is more efficient, both in terms of running time and communication cost. This is validated by our extensive experiments on various types of DFA's, with up to 217 states. It also turns out that both algorithms are sensitive to skewed input, the Hopcroft's algorithm being intrinsically so.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132351969","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 7
Toward elastic memory management for cloud data analytics 面向云数据分析的弹性内存管理
Jingjing Wang, M. Balazinska
We present several key elements towards elastic memory management in modern big data systems. The goal of our approach is to avoid out-of-memory failures without over-provisioning but also to avoid garbage-collection overheads when possible.
我们提出了在现代大数据系统中实现弹性内存管理的几个关键要素。我们方法的目标是在不过度配置的情况下避免内存不足故障,同时尽可能避免垃圾收集开销。
{"title":"Toward elastic memory management for cloud data analytics","authors":"Jingjing Wang, M. Balazinska","doi":"10.1145/2926534.2926541","DOIUrl":"https://doi.org/10.1145/2926534.2926541","url":null,"abstract":"We present several key elements towards elastic memory management in modern big data systems. The goal of our approach is to avoid out-of-memory failures without over-provisioning but also to avoid garbage-collection overheads when possible.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126956376","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 10
Model-centric computation abstractions in machine learning applications 机器学习应用中以模型为中心的计算抽象
Bingjing Zhang, Bo Peng, J. Qiu
We categorize parallel machine learning applications into four types of computation models and propose a new set of model-centric computation abstractions. This work sets up parallel machine learning as a combination of training data-centric and model parameter-centric processing. The analysis uses Latent Dirichlet Allocation (LDA) as an example, and experimental results show that an efficient parallel model update pipeline can achieve similar or higher model convergence speed compared with other work.
我们将并行机器学习应用分为四种类型的计算模型,并提出了一套新的以模型为中心的计算抽象。这项工作将并行机器学习建立为以训练数据为中心和以模型参数为中心的处理的结合。以潜在狄利克雷分配(Latent Dirichlet Allocation, LDA)为例进行分析,实验结果表明,一种高效的并行模型更新管道与其他方法相比,可以达到相似或更高的模型收敛速度。
{"title":"Model-centric computation abstractions in machine learning applications","authors":"Bingjing Zhang, Bo Peng, J. Qiu","doi":"10.1145/2926534.2926539","DOIUrl":"https://doi.org/10.1145/2926534.2926539","url":null,"abstract":"We categorize parallel machine learning applications into four types of computation models and propose a new set of model-centric computation abstractions. This work sets up parallel machine learning as a combination of training data-centric and model parameter-centric processing. The analysis uses Latent Dirichlet Allocation (LDA) as an example, and experimental results show that an efficient parallel model update pipeline can achieve similar or higher model convergence speed compared with other work.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127899899","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 7
Some pairs problems 一些配对问题
J. Ullman, Jonathan Ullman
A common form of MapReduce application involves discovering relationships between certain pairs of inputs. Similarity joins serve as a good example of this type of problem, which we call a "some-pairs" problem. In the framework of [4], algorithms are measured by the tradeoff between reducer size (maximum number of inputs a reducer can handle) and the replication rate (average number of reducers to which an input must be sent. There are two obvious approaches to solving some-pairs problems in general. We show that no general-purpose MapReduce algorithm can beat both of these two algorithms in the worst case. We then explore a recursive algorithm for solving some-pairs problems and heuristics for beating the lower bound on common instances of the some-pairs class of problems.
MapReduce应用程序的一种常见形式涉及发现某些输入对之间的关系。相似性连接是这类问题的一个很好的例子,我们称之为“某些对”问题。在[4]的框架中,算法是通过reducer大小(一个reducer可以处理的最大输入数量)和复制率(必须发送输入的reducer的平均数量)之间的权衡来衡量的。一般来说,有两种明显的方法来解决某些配对问题。我们表明,在最坏的情况下,没有通用MapReduce算法可以同时击败这两种算法。然后,我们探讨了解决若干对问题的递归算法和在若干对类问题的常见实例上击败下界的启发式算法。
{"title":"Some pairs problems","authors":"J. Ullman, Jonathan Ullman","doi":"10.1145/2926534.2926543","DOIUrl":"https://doi.org/10.1145/2926534.2926543","url":null,"abstract":"A common form of MapReduce application involves discovering relationships between certain pairs of inputs. Similarity joins serve as a good example of this type of problem, which we call a \"some-pairs\" problem. In the framework of [4], algorithms are measured by the tradeoff between reducer size (maximum number of inputs a reducer can handle) and the replication rate (average number of reducers to which an input must be sent. There are two obvious approaches to solving some-pairs problems in general. We show that no general-purpose MapReduce algorithm can beat both of these two algorithms in the worst case. We then explore a recursive algorithm for solving some-pairs problems and heuristics for beating the lower bound on common instances of the some-pairs class of problems.","PeriodicalId":393776,"journal":{"name":"Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2016-02-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122331611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
期刊
Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1