Yu Chen, Jun Zhang, Zhicheng Wang, Gejian Liao, Shu Liu, Hai Tan, Guowei Yang, Ying Fang, Shuai Wang, Zhaoqun Sun
{"title":"A Faster Read and Less Storage Algorithm for Small Files on Hadoop","authors":"Yu Chen, Jun Zhang, Zhicheng Wang, Gejian Liao, Shu Liu, Hai Tan, Guowei Yang, Ying Fang, Shuai Wang, Zhaoqun Sun","doi":"10.1109/ICCEAI52939.2021.00040","DOIUrl":null,"url":null,"abstract":"Massive small files access is the main challenge for the Hadoop Distributed File System. To solve these problems, we present a new Algorithm of archive file, A Faster Read and Less Storage Algorithm for Small Files on Hadoop. A new logical file name is used to identify the file which generated by the pair in the name node. Our experiments show that the algorithm is around 76.6% faster than original HDFS in the time of file storing, and around 31.9.6% faster than original HDFS in the time of file reading, around 73.9% less than original HDFS in the memory consumption of namenode.","PeriodicalId":331409,"journal":{"name":"2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI)","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICCEAI52939.2021.00040","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1
Abstract
Massive small files access is the main challenge for the Hadoop Distributed File System. To solve these problems, we present a new Algorithm of archive file, A Faster Read and Less Storage Algorithm for Small Files on Hadoop. A new logical file name is used to identify the file which generated by the pair in the name node. Our experiments show that the algorithm is around 76.6% faster than original HDFS in the time of file storing, and around 31.9.6% faster than original HDFS in the time of file reading, around 73.9% less than original HDFS in the memory consumption of namenode.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
