In this paper, we investigate the possibilities offered by the Hadoop eco-system for searching time series in an electric power company (Top-K or range-queries based on a similarity measure). There has been much work done on speeding up the search of time series in a large dataset, mainly by designing efficient indexing techniques preceded by reduction techniques. In this paper, we do not follow these approaches but focus on using the brutal force of distributed computations in the Hadoop environment. We propose an implementation of time series search functions in Hadoop and describe experiments on a large database of electric power consumption curves (35M customers observed during 1 month at a 30' sampling rate). We also show that this architecture supports easily the computation of several distances for the same query with a small response time overhead: this is very useful in practice when the end-user does not know very well which distance to use.
{"title":"Searching time series with Hadoop in an electric power company","authors":"Alice Berard, G. Hébrail","doi":"10.1145/2501221.2501224","DOIUrl":"https://doi.org/10.1145/2501221.2501224","url":null,"abstract":"In this paper, we investigate the possibilities offered by the Hadoop eco-system for searching time series in an electric power company (Top-K or range-queries based on a similarity measure). There has been much work done on speeding up the search of time series in a large dataset, mainly by designing efficient indexing techniques preceded by reduction techniques. In this paper, we do not follow these approaches but focus on using the brutal force of distributed computations in the Hadoop environment. We propose an implementation of time series search functions in Hadoop and describe experiments on a large database of electric power consumption curves (35M customers observed during 1 month at a 30' sampling rate). We also show that this architecture supports easily the computation of several distances for the same query with a small response time overhead: this is very useful in practice when the end-user does not know very well which distance to use.","PeriodicalId":441216,"journal":{"name":"BigMine '13","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-08-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134628951","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marcelo Mendoza, Bárbara Poblete, Felipe Bravo-Marquez, Daniel Gayo-Avello
Usually time series are controlled by generative processes which display changes over time. On many occasions, two or more generative processes may switch forcing the abrupt replacement of a fitted time series model by another one. We claim that the incorporation of past data can be useful in the presence of concept shift. We believe that history tends to repeat itself and from time to time, it is desirable to discard recent data reusing old past data to perform model fitting and forecasting. We address this challenge by introducing an ensemble method that deals with long-memory time series. Our method starts by segmenting historical time series data to identify data segments which present model consistency. Then, we project the time series by using data segments which are close to current data. By using a dynamic time warping alignment function, we try to anticipate concept shifts, looking for similarities between current data and the prequel of a past shift. We evaluate our proposal on non-stationary and non-linear time series. To achieve this we perform forecasting accuracy testing against well known state-of-the-art methods such as neural networks and threshold auto regressive models. Our results show that the proposed method anticipates many concept shifts.
{"title":"Long-memory time series ensembles for concept shift detection","authors":"Marcelo Mendoza, Bárbara Poblete, Felipe Bravo-Marquez, Daniel Gayo-Avello","doi":"10.1145/2501221.2501225","DOIUrl":"https://doi.org/10.1145/2501221.2501225","url":null,"abstract":"Usually time series are controlled by generative processes which display changes over time. On many occasions, two or more generative processes may switch forcing the abrupt replacement of a fitted time series model by another one. We claim that the incorporation of past data can be useful in the presence of concept shift. We believe that history tends to repeat itself and from time to time, it is desirable to discard recent data reusing old past data to perform model fitting and forecasting. We address this challenge by introducing an ensemble method that deals with long-memory time series. Our method starts by segmenting historical time series data to identify data segments which present model consistency. Then, we project the time series by using data segments which are close to current data. By using a dynamic time warping alignment function, we try to anticipate concept shifts, looking for similarities between current data and the prequel of a past shift. We evaluate our proposal on non-stationary and non-linear time series. To achieve this we perform forecasting accuracy testing against well known state-of-the-art methods such as neural networks and threshold auto regressive models. Our results show that the proposed method anticipates many concept shifts.","PeriodicalId":441216,"journal":{"name":"BigMine '13","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-08-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127266144","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Chen Jin, Qiang Fu, Huahua Wang, Ankit Agrawal, W. Hendrix, W. Liao, Md. Mostofa Ali Patwary, A. Banerjee, A. Choudhary
Several important combinatorial optimization problems can be formulated as maximum a posteriori (MAP) inference in discrete graphical models. We adopt the recently proposed parallel MAP inference algorithm Bethe-ADMM and implement it using message passing interface (MPI) to fully utilize the computing power provided by the modern supercomputers with thousands of cores. The empirical results show that our parallel implementation scales almost linearly even with thousands of cores.
{"title":"Solving combinatorial optimization problems using relaxed linear programming: a high performance computing perspective","authors":"Chen Jin, Qiang Fu, Huahua Wang, Ankit Agrawal, W. Hendrix, W. Liao, Md. Mostofa Ali Patwary, A. Banerjee, A. Choudhary","doi":"10.1145/2501221.2501227","DOIUrl":"https://doi.org/10.1145/2501221.2501227","url":null,"abstract":"Several important combinatorial optimization problems can be formulated as maximum a posteriori (MAP) inference in discrete graphical models. We adopt the recently proposed parallel MAP inference algorithm Bethe-ADMM and implement it using message passing interface (MPI) to fully utilize the computing power provided by the modern supercomputers with thousands of cores. The empirical results show that our parallel implementation scales almost linearly even with thousands of cores.","PeriodicalId":441216,"journal":{"name":"BigMine '13","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-08-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127241749","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jonathan W. Berry, M. Oster, C. Phillips, S. Plimpton, Timothy M. Shead
We present an algorithm to maintain the connected components of a graph that arrives as an infinite stream of edges. We formalize the algorithm on X-stream, a new parallel theoretical computational model for infinite streams. Connectivity-related queries, including component spanning trees, are supported with some latency, returning the state of the graph at the time of the query. Because an infinite stream may eventually exceed the storage limits of any number of finite-memory processors, we assume an aging command or daemon where "uninteresting" edges are removed when the system nears capacity. Following an aging command the system will block queries until its data structures are repaired, but edges will continue to be accepted from the stream, never dropped. The algorithm will not fail unless a model-specific constant fraction of the aggregate memory across all processors is full. In normal operation, it will not fail unless aggregate memory is completely full. Unlike previous theoretical streaming models designed for finite graphs that assume a single shared memory machine or require arbitrary-size intemediate files, X-stream distributes a graph over a ring network of finite-memory processors. Though the model is synchronous and reminiscent of systolic algorithms, our implementation uses an asynchronous message-passing system. We argue the correctness of our X-stream connected components algorithm, and give preliminary experimental results on synthetic and real graph streams.
{"title":"Maintaining connected components for infinite graph streams","authors":"Jonathan W. Berry, M. Oster, C. Phillips, S. Plimpton, Timothy M. Shead","doi":"10.1145/2501221.2501234","DOIUrl":"https://doi.org/10.1145/2501221.2501234","url":null,"abstract":"We present an algorithm to maintain the connected components of a graph that arrives as an infinite stream of edges. We formalize the algorithm on X-stream, a new parallel theoretical computational model for infinite streams. Connectivity-related queries, including component spanning trees, are supported with some latency, returning the state of the graph at the time of the query. Because an infinite stream may eventually exceed the storage limits of any number of finite-memory processors, we assume an aging command or daemon where \"uninteresting\" edges are removed when the system nears capacity. Following an aging command the system will block queries until its data structures are repaired, but edges will continue to be accepted from the stream, never dropped. The algorithm will not fail unless a model-specific constant fraction of the aggregate memory across all processors is full. In normal operation, it will not fail unless aggregate memory is completely full.\u0000 Unlike previous theoretical streaming models designed for finite graphs that assume a single shared memory machine or require arbitrary-size intemediate files, X-stream distributes a graph over a ring network of finite-memory processors. Though the model is synchronous and reminiscent of systolic algorithms, our implementation uses an asynchronous message-passing system. We argue the correctness of our X-stream connected components algorithm, and give preliminary experimental results on synthetic and real graph streams.","PeriodicalId":441216,"journal":{"name":"BigMine '13","volume":"107 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-08-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115119569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：