通过分离键和值来优化LevelDB

Lei Wang, G. Ding, Yulong Zhao, Dingzeyu Wu, Chengrui He
{"title":"通过分离键和值来优化LevelDB","authors":"Lei Wang, G. Ding, Yulong Zhao, Dingzeyu Wu, Chengrui He","doi":"10.1109/PDCAT.2017.00074","DOIUrl":null,"url":null,"abstract":"LevelDB uses the merger mechanism for data integration during the writing process. During this process, the value will move together with the key, causing a lot of unnecessary data rewriting. This paper presents a structure that the key stored separately from the value. And the value is stored in a separate file (we call it Value File), with value offset in the file and length information stored in LevelDB. Test results show that the optimized LevelDBs sequential write performance is reduced by about 40%. But random write and overwrite performances improve more than 200%. And with the increase of the number of tests records, the improvement becomes more and more obvious. The amount of rewriting data and the number of merging files, depending on the length of every different record, reduce about 80% averagely, which significantly improves the performance of the original program.","PeriodicalId":119197,"journal":{"name":"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)","volume":"10 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2017-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"6","resultStr":"{\"title\":\"Optimization of LevelDB by Separating Key and Value\",\"authors\":\"Lei Wang, G. Ding, Yulong Zhao, Dingzeyu Wu, Chengrui He\",\"doi\":\"10.1109/PDCAT.2017.00074\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"LevelDB uses the merger mechanism for data integration during the writing process. During this process, the value will move together with the key, causing a lot of unnecessary data rewriting. This paper presents a structure that the key stored separately from the value. And the value is stored in a separate file (we call it Value File), with value offset in the file and length information stored in LevelDB. Test results show that the optimized LevelDBs sequential write performance is reduced by about 40%. But random write and overwrite performances improve more than 200%. And with the increase of the number of tests records, the improvement becomes more and more obvious. The amount of rewriting data and the number of merging files, depending on the length of every different record, reduce about 80% averagely, which significantly improves the performance of the original program.\",\"PeriodicalId\":119197,\"journal\":{\"name\":\"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)\",\"volume\":\"10 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2017-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"6\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/PDCAT.2017.00074\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/PDCAT.2017.00074","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 6

摘要

LevelDB在写入过程中使用合并机制进行数据集成。在这个过程中,值会随着键一起移动,导致大量不必要的数据重写。本文提出了一种键与值分开存储的结构。值存储在一个单独的文件中(我们称之为值文件),文件中的值偏移量和长度信息存储在LevelDB中。测试结果表明,优化后的leveldb顺序写性能降低了约40%。但是随机写入和覆盖性能提高了200%以上。并且随着检测记录数量的增加,这种改善也越来越明显。根据每条不同记录的长度,重写数据的数量和合并文件的数量平均减少了80%左右,这大大提高了原程序的性能。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
Optimization of LevelDB by Separating Key and Value
LevelDB uses the merger mechanism for data integration during the writing process. During this process, the value will move together with the key, causing a lot of unnecessary data rewriting. This paper presents a structure that the key stored separately from the value. And the value is stored in a separate file (we call it Value File), with value offset in the file and length information stored in LevelDB. Test results show that the optimized LevelDBs sequential write performance is reduced by about 40%. But random write and overwrite performances improve more than 200%. And with the increase of the number of tests records, the improvement becomes more and more obvious. The amount of rewriting data and the number of merging files, depending on the length of every different record, reduce about 80% averagely, which significantly improves the performance of the original program.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
Implementing Algorithmic Skeletons with Bulk Synchronous Parallel ML Managing Bytecode and ISA Compatibility with an Enhanced Toolchain Improved Online Algorithms for One-Dimensional BinPacking with Advice A Case Study in Higher Education Domain Based on a Prototype for Business Process Models Improvement: BPMoQualAssess NMFDIV: A Nonnegative Matrix Factorization Approach for Search Result Diversification on Attributed Networks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1