Improving Throughput of BigData Applications

Abstract

The paper describes various performance problems and solutions to improve throughput of BigData Application like Redis, Kafka, memcache, Cassandra, ElasticSearch,..etc. Most of the solution to the problems are achieved by some of the techniques like bypassing linux kernel, minimizing system calls, efficiently using the multi core machine using asynchronous programming, one thread per core, DPDK, .. etc. Modern machines are very different from those of just 10 years ago. They have many cores and deep memory hierarchies (from L1 caches to NUMA) which reward certain programming practices and penalizes others, Unscalable programming practices (such as taking locks) can devastate performance on many cores. Shared memory and lock-free synchronization primitives are used in solving some of the problems. The paper was concluded with the test prototype of Redis with efficient network path that resulted 37X perf improvement over the baseline.