A. D. Jaggard, Swara Kopparty, V. Ramachandran, R. Wright
We present a framework for the design and analysis of probing methods to monitor network performance, an important technique for collecting measurements in tasks such as fault detection. We use this framework to study the interaction among numerous, possibly conflicting, optimization goals in the design of a probing algorithm. We present a rigorous definition of a probing-algorithm design problem that can apply broadly to network-measurement scenarios. We also present several metrics relevant to the analysis of probing algorithms, including probing frequency and network coverage, communication and computational overhead, and the amount of algorithm state required. We show inherent tradeoffs among optimization goals and give hardness results for achieving some combinations of optimization goals. We also consider the possibility of developing approximation algorithms for achieving some of the goals and describe a randomized approach as an alternative, evaluating it using our framework. Our work aids future development of low-overhead probing techniques and introduces principles from IP-based networking to theoretically grounded approaches for concurrent path-selection problems.
{"title":"The design space of probing algorithms for network-performance measurement","authors":"A. D. Jaggard, Swara Kopparty, V. Ramachandran, R. Wright","doi":"10.1145/2465529.2465765","DOIUrl":"https://doi.org/10.1145/2465529.2465765","url":null,"abstract":"We present a framework for the design and analysis of probing methods to monitor network performance, an important technique for collecting measurements in tasks such as fault detection. We use this framework to study the interaction among numerous, possibly conflicting, optimization goals in the design of a probing algorithm. We present a rigorous definition of a probing-algorithm design problem that can apply broadly to network-measurement scenarios. We also present several metrics relevant to the analysis of probing algorithms, including probing frequency and network coverage, communication and computational overhead, and the amount of algorithm state required. We show inherent tradeoffs among optimization goals and give hardness results for achieving some combinations of optimization goals. We also consider the possibility of developing approximation algorithms for achieving some of the goals and describe a randomized approach as an alternative, evaluating it using our framework. Our work aids future development of low-overhead probing techniques and introduces principles from IP-based networking to theoretically grounded approaches for concurrent path-selection problems.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"38 2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126604690","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We study a generalization of the classic paging problem where memory capacity can vary over time - a property of many modern computing realities, from cloud computing to multi-core and energy-optimized processors. We show that good performance in the "classic" case provides no performance guarantees when memory capacity fluctuates: roughly speaking, moving from static to dynamic capacity can mean the difference between optimality within a factor 2 in space, time and energy, and suboptimality by an arbitrarily large factor. Surprisingly, several classic paging algorithms still perform remarkably well, maintaining that factor 2 optimality even if faced with adversarial capacity fluctuations - without taking those fluctuations into explicit account!
{"title":"Elastic paging","authors":"Enoch Peserico","doi":"10.1145/2465529.2479781","DOIUrl":"https://doi.org/10.1145/2465529.2479781","url":null,"abstract":"We study a generalization of the classic paging problem where memory capacity can vary over time - a property of many modern computing realities, from cloud computing to multi-core and energy-optimized processors. We show that good performance in the \"classic\" case provides no performance guarantees when memory capacity fluctuates: roughly speaking, moving from static to dynamic capacity can mean the difference between optimality within a factor 2 in space, time and energy, and suboptimality by an arbitrarily large factor. Surprisingly, several classic paging algorithms still perform remarkably well, maintaining that factor 2 optimality even if faced with adversarial capacity fluctuations - without taking those fluctuations into explicit account!","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"245 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114829171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We explore the achievable delay performance in wireless random-access networks. While relatively simple and inherently distributed in nature, suitably designed backlog-based random-access schemes provide the striking capability to match the optimal throughput performance of centralized scheduling mechanisms. The specific type of activation rules for which throughput optimality has been established, may however yield excessive backlogs and delays. Motivated by that issue, we examine whether the poor delay performance is inherent to the basic operation of these schemes, or caused by the specific kind of activation rules. We derive delay lower bounds for backlog-based activation rules, which offer fundamental insight in the cause of the excessive delays. For fixed activation rates we obtain lower bounds indicating that delays and mixing times can grow dramatically with the load in certain topologies as well.
{"title":"Delays and mixing times in random-access networks","authors":"N. Bouman, S. Borst, J. V. Leeuwaarden","doi":"10.1145/2465529.2465759","DOIUrl":"https://doi.org/10.1145/2465529.2465759","url":null,"abstract":"We explore the achievable delay performance in wireless random-access networks. While relatively simple and inherently distributed in nature, suitably designed backlog-based random-access schemes provide the striking capability to match the optimal throughput performance of centralized scheduling mechanisms. The specific type of activation rules for which throughput optimality has been established, may however yield excessive backlogs and delays. Motivated by that issue, we examine whether the poor delay performance is inherent to the basic operation of these schemes, or caused by the specific kind of activation rules. We derive delay lower bounds for backlog-based activation rules, which offer fundamental insight in the cause of the excessive delays. For fixed activation rates we obtain lower bounds indicating that delays and mixing times can grow dramatically with the load in certain topologies as well.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117118017","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Multi-path TCP (MP-TCP) has the potential to greatly improve application performance by using multiple paths transparently. We propose a fluid model for a large class of MP-TCP algorithms and identify design criteria that guarantee the existence, uniqueness, and stability of system equilibrium. We characterize algorithm parameters for TCP-friendliness and prove an inevitable tradeoff between responsiveness and friendliness. We discuss the implications of these properties on the behavior of existing algorithms and motivate a new design that generalizes existing algorithms. We use ns2 simulations to evaluate the proposed algorithm and illustrate its superior overall performance.
{"title":"Multipath TCP algorithms: theory and design","authors":"Qiuyu Peng, A. Elwalid, S. Low","doi":"10.1145/2465529.2466585","DOIUrl":"https://doi.org/10.1145/2465529.2466585","url":null,"abstract":"Multi-path TCP (MP-TCP) has the potential to greatly improve application performance by using multiple paths transparently. We propose a fluid model for a large class of MP-TCP algorithms and identify design criteria that guarantee the existence, uniqueness, and stability of system equilibrium. We characterize algorithm parameters for TCP-friendliness and prove an inevitable tradeoff between responsiveness and friendliness. We discuss the implications of these properties on the behavior of existing algorithms and motivate a new design that generalizes existing algorithms. We use ns2 simulations to evaluate the proposed algorithm and illustrate its superior overall performance.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132021462","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Zhibin Yu, L. Eeckhout, Nilanjan Goswami, Tao Li, L. John, Hai Jin, Chengzhong Xu
Recently, graphics processing units (GPUs) have opened up new opportunities for speeding up general-purpose parallel applications due to their massive computational power and up to hundreds of thousands of threads enabled by programming models such as CUDA. However, due to the serial nature of existing micro-architecture simulators, these massively parallel architectures and workloads need to be simulated sequentially. As a result, simulating GPGPU architectures with typical benchmarks and input data sets is extremely time-consuming. This paper addresses the GPGPU architecture simulation challenge by generating miniature, yet representative GPGPU kernels. We first summarize the static characteristics of an existing GPGPU kernel in a profile, and analyze its dynamic behavior using the novel concept of the divergence flow statistics graph (DFSG). We subsequently use a GPGPU kernel synthesizing framework to generate a miniature proxy of the original kernel, which can reduce simulation time significantly. The key idea is to reduce the number of simulated instructions by decreasing per-thread iteration counts of loops. Our experimental results show that our approach can accelerate GPGPU architecture simulation by a factor of 88X on average and up to 589X with an average IPC relative error of 5.6%.
{"title":"Accelerating GPGPU architecture simulation","authors":"Zhibin Yu, L. Eeckhout, Nilanjan Goswami, Tao Li, L. John, Hai Jin, Chengzhong Xu","doi":"10.1145/2465529.2465540","DOIUrl":"https://doi.org/10.1145/2465529.2465540","url":null,"abstract":"Recently, graphics processing units (GPUs) have opened up new opportunities for speeding up general-purpose parallel applications due to their massive computational power and up to hundreds of thousands of threads enabled by programming models such as CUDA. However, due to the serial nature of existing micro-architecture simulators, these massively parallel architectures and workloads need to be simulated sequentially. As a result, simulating GPGPU architectures with typical benchmarks and input data sets is extremely time-consuming. This paper addresses the GPGPU architecture simulation challenge by generating miniature, yet representative GPGPU kernels. We first summarize the static characteristics of an existing GPGPU kernel in a profile, and analyze its dynamic behavior using the novel concept of the divergence flow statistics graph (DFSG). We subsequently use a GPGPU kernel synthesizing framework to generate a miniature proxy of the original kernel, which can reduce simulation time significantly. The key idea is to reduce the number of simulated instructions by decreasing per-thread iteration counts of loops. Our experimental results show that our approach can accelerate GPGPU architecture simulation by a factor of 88X on average and up to 589X with an average IPC relative error of 5.6%.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"73 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127271583","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Much attention has been given recently to a set of promising non-volatile memory technologies, such as PCM, STT-MRAM, and ReRAM. These, however, have limited endurance relative to DRAM. Potential solutions to this endurance challenge exist in the form of fine-grain wear leveling techniques and aggressive error tolerance approaches. While the existing approaches to wear leveling and error tolerance are sound and demonstrate true potential, their studies have been limited in that i) they have not considered the interactions between wear leveling and error tolerance and ii) they have assumed a simple write endurance failure model where all cells fail uniformly. In this paper we perform a thorough study and characterize such interactions and the effects of more realistic non-uniform endurance models under various workloads, both synthetic and derived from benchmarks. This study shows that, for instance, variability in the endurance of cells significantly affects wear leveling and error tolerance mechanisms and the values of their tuning parameters. It also shows that these mechanisms interact in subtle ways, sometimes cancelling and sometimes boosting each other's impact on overall endurance of the device.
{"title":"Characterizing the impact of process variation on write endurance enhancing techniques for non-volatile memory systems","authors":"M. Cintra, Niklas Linkewitsch","doi":"10.1145/2465529.2465755","DOIUrl":"https://doi.org/10.1145/2465529.2465755","url":null,"abstract":"Much attention has been given recently to a set of promising non-volatile memory technologies, such as PCM, STT-MRAM, and ReRAM. These, however, have limited endurance relative to DRAM. Potential solutions to this endurance challenge exist in the form of fine-grain wear leveling techniques and aggressive error tolerance approaches. While the existing approaches to wear leveling and error tolerance are sound and demonstrate true potential, their studies have been limited in that i) they have not considered the interactions between wear leveling and error tolerance and ii) they have assumed a simple write endurance failure model where all cells fail uniformly. In this paper we perform a thorough study and characterize such interactions and the effects of more realistic non-uniform endurance models under various workloads, both synthetic and derived from benchmarks. This study shows that, for instance, variability in the endurance of cells significantly affects wear leveling and error tolerance mechanisms and the values of their tuning parameters. It also shows that these mechanisms interact in subtle ways, sometimes cancelling and sometimes boosting each other's impact on overall endurance of the device.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"73 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133951088","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ignasi Paredes-Oliva, P. Barlet-Ros, X. Dimitropoulos
Extracting knowledge from big network traffic data is a matter of foremost importance for multiple purposes ranging from trend analysis or network troubleshooting to capacity planning or traffic classification. An extremely useful approach to profile traffic is to extract and display to a network administrator the multi-dimensional hierarchical heavy hitters (HHHs) of a dataset. However, existing schemes for computing HHHs have several limitations: 1) they require significant computational overhead; 2) they do not scale to high dimensional data; and 3) they are not easily extensible. In this paper, we introduce a fundamentally new approach for extracting HHHs based on generalized frequent item-set mining (FIM), which allows to process traffic data much more efficiently and scales to much higher dimensional data than present schemes. Based on generalized FIM, we build and evaluate a traffic profiling system we call FaRNet. Our comparison with AutoFocus, which is the most related tool of similar nature, shows that FaRNet is up to three orders of magnitude faster.
{"title":"FaRNet: fast recognition of high multi-dimensional network traffic patterns","authors":"Ignasi Paredes-Oliva, P. Barlet-Ros, X. Dimitropoulos","doi":"10.1145/2465529.2465743","DOIUrl":"https://doi.org/10.1145/2465529.2465743","url":null,"abstract":"Extracting knowledge from big network traffic data is a matter of foremost importance for multiple purposes ranging from trend analysis or network troubleshooting to capacity planning or traffic classification. An extremely useful approach to profile traffic is to extract and display to a network administrator the multi-dimensional hierarchical heavy hitters (HHHs) of a dataset. However, existing schemes for computing HHHs have several limitations: 1) they require significant computational overhead; 2) they do not scale to high dimensional data; and 3) they are not easily extensible. In this paper, we introduce a fundamentally new approach for extracting HHHs based on generalized frequent item-set mining (FIM), which allows to process traffic data much more efficiently and scales to much higher dimensional data than present schemes. Based on generalized FIM, we build and evaluate a traffic profiling system we call FaRNet. Our comparison with AutoFocus, which is the most related tool of similar nature, shows that FaRNet is up to three orders of magnitude faster.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"65 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130082037","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Symmetric-ISA (instruction set architecture) asymmetric-performance multicore processors (AMPs) were shown to deliver higher performance per watt and area than symmetric CMPs for applications with diverse architectural requirements. So, it is likely that future multicore processors will combine big power-hungry fast cores and small low-power slow ones. In this paper, we propose a novel thread scheduling algorithm that aims to improve the throughput-fairness trade-off on AMP systems. Our experimental evaluation on real hardware and using scheduler implementations on a general-purpose operating system, reveals that our proposal delivers a better throughput-fairness trade-off than previous schedulers for a wide variety of multi-application workloads including single-threaded and multithreaded applications.
{"title":"Delivering fairness and priority enforcement on asymmetric multicore systems via OS scheduling","authors":"J. C. Saez, Fernando Castro, D. Chaver, M. Prieto","doi":"10.1145/2465529.2465532","DOIUrl":"https://doi.org/10.1145/2465529.2465532","url":null,"abstract":"Symmetric-ISA (instruction set architecture) asymmetric-performance multicore processors (AMPs) were shown to deliver higher performance per watt and area than symmetric CMPs for applications with diverse architectural requirements. So, it is likely that future multicore processors will combine big power-hungry fast cores and small low-power slow ones. In this paper, we propose a novel thread scheduling algorithm that aims to improve the throughput-fairness trade-off on AMP systems. Our experimental evaluation on real hardware and using scheduler implementations on a general-purpose operating system, reveals that our proposal delivers a better throughput-fairness trade-off than previous schedulers for a wide variety of multi-application workloads including single-threaded and multithreaded applications.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134284617","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Quantifying quality of experience for network applications is challenging as it is a subjective metric with multiple dimensions such as user expectation, satisfaction, and overall experience. Today, despite various techniques to support differentiated Quality of Service (QoS), the operators still lack of automated methods to translate QoS to QoE, especially for general web applications. In this work, we take the approach of identifying unsatisfactory performance by searching for user initiated early terminations of web transactions from passive monitoring. However, user early abortions can be caused by other factors such as loss of interests. Therefore, naively using them to represent user dissatisfaction will result in large false positives. In this paper, we propose a systematic method for inferring user dissatisfaction from the set of early abortion behaviors observed from identifying the traffic traces. We conduct a comprehensive analysis on the user acceptance of throughput and response time, and compare them with the traditional MOS metric. Then we present the characteristics of early cancelation from dimensions like the types of URLs and objects. We evaluate our approach on four data sets collected in both wireline network and a wireless cellular network.
{"title":"Detecting user dissatisfaction and understanding the underlying reasons","authors":"Å. Arvidsson, Y. Zhang","doi":"10.1145/2465529.2465538","DOIUrl":"https://doi.org/10.1145/2465529.2465538","url":null,"abstract":"Quantifying quality of experience for network applications is challenging as it is a subjective metric with multiple dimensions such as user expectation, satisfaction, and overall experience. Today, despite various techniques to support differentiated Quality of Service (QoS), the operators still lack of automated methods to translate QoS to QoE, especially for general web applications.\u0000 In this work, we take the approach of identifying unsatisfactory performance by searching for user initiated early terminations of web transactions from passive monitoring. However, user early abortions can be caused by other factors such as loss of interests. Therefore, naively using them to represent user dissatisfaction will result in large false positives. In this paper, we propose a systematic method for inferring user dissatisfaction from the set of early abortion behaviors observed from identifying the traffic traces. We conduct a comprehensive analysis on the user acceptance of throughput and response time, and compare them with the traditional MOS metric. Then we present the characteristics of early cancelation from dimensions like the types of URLs and objects. We evaluate our approach on four data sets collected in both wireline network and a wireless cellular network.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129203591","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Crowdsourcing systems like Amazon's Mechanical Turk have emerged as an effective large-scale human-powered platform for performing tasks in domains such as image classification, data entry, recommendation, and proofreading. Since workers are low-paid (a few cents per task) and tasks performed are monotonous, the answers obtained are noisy and hence unreliable. To obtain reliable estimates, it is essential to utilize appropriate inference algorithms (e.g. Majority voting) coupled with structured redundancy through task assignment. Our goal is to obtain the best possible trade-off between reliability and redundancy. In this paper, we consider a general probabilistic model for noisy observations for crowd-sourcing systems and pose the problem of minimizing the total price (i.e. redundancy) that must be paid to achieve a target overall reliability. Concretely, we show that it is possible to obtain an answer to each task correctly with probability 1-ε as long as the redundancy per task is O((K/q) log (K/ε)), where each task can have any of the $K$ distinct answers equally likely, q is the crowd-quality parameter that is defined through a probabilistic model. Further, effectively this is the best possible redundancy-accuracy trade-off any system design can achieve. Such a single-parameter crisp characterization of the (order-)optimal trade-off between redundancy and reliability has various useful operational consequences. Further, we analyze the robustness of our approach in the presence of adversarial workers and provide a bound on their influence on the redundancy-accuracy trade-off. Unlike recent prior work [GKM11, KOS11, KOS11], our result applies to non-binary (i.e. K>2) tasks. In effect, we utilize algorithms for binary tasks (with inhomogeneous error model unlike that in [GKM11, KOS11, KOS11]) as key subroutine to obtain answers for K-ary tasks. Technically, the algorithm is based on low-rank approximation of weighted adjacency matrix for a random regular bipartite graph, weighted according to the answers provided by the workers.
{"title":"Efficient crowdsourcing for multi-class labeling","authors":"David R Karger, Sewoong Oh, D. Shah","doi":"10.1145/2465529.2465761","DOIUrl":"https://doi.org/10.1145/2465529.2465761","url":null,"abstract":"Crowdsourcing systems like Amazon's Mechanical Turk have emerged as an effective large-scale human-powered platform for performing tasks in domains such as image classification, data entry, recommendation, and proofreading. Since workers are low-paid (a few cents per task) and tasks performed are monotonous, the answers obtained are noisy and hence unreliable. To obtain reliable estimates, it is essential to utilize appropriate inference algorithms (e.g. Majority voting) coupled with structured redundancy through task assignment. Our goal is to obtain the best possible trade-off between reliability and redundancy. In this paper, we consider a general probabilistic model for noisy observations for crowd-sourcing systems and pose the problem of minimizing the total price (i.e. redundancy) that must be paid to achieve a target overall reliability. Concretely, we show that it is possible to obtain an answer to each task correctly with probability 1-ε as long as the redundancy per task is O((K/q) log (K/ε)), where each task can have any of the $K$ distinct answers equally likely, q is the crowd-quality parameter that is defined through a probabilistic model. Further, effectively this is the best possible redundancy-accuracy trade-off any system design can achieve. Such a single-parameter crisp characterization of the (order-)optimal trade-off between redundancy and reliability has various useful operational consequences. Further, we analyze the robustness of our approach in the presence of adversarial workers and provide a bound on their influence on the redundancy-accuracy trade-off.\u0000 Unlike recent prior work [GKM11, KOS11, KOS11], our result applies to non-binary (i.e. K>2) tasks. In effect, we utilize algorithms for binary tasks (with inhomogeneous error model unlike that in [GKM11, KOS11, KOS11]) as key subroutine to obtain answers for K-ary tasks. Technically, the algorithm is based on low-rank approximation of weighted adjacency matrix for a random regular bipartite graph, weighted according to the answers provided by the workers.","PeriodicalId":306456,"journal":{"name":"Measurement and Modeling of Computer Systems","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129918768","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：