A fine-grained compression scheme for block transmission acceleration over IPFS network

Abstract

Interplanetary File System (IPFS) network, a prominent peer-to-peer decentralized file-sharing network, is extensively utilized by decentralized applications such as Blockchain and Metaverse for data sharing. With a substantial upsurge in data, compression schemes have been incorporated into the native IPFS protocol to enhance the efficiency of block transmission over an IPFS network. However, the protocols extended by these schemes drastically impair data shareability with the native protocol, contravening the core principles of IPFS-based distributed storage.

To address the issue, this paper presents a fine-grained compression scheme for the native IPFS protocol. Its primary concept is that blocks rather than files are considered as the smallest compressible units for block transmission, thereby ensuring the preservation of the data shareability with the protocol. To achieve the scheme, a three-layer architecture is firstly proposed and incorporated into the protocol as an independent compression component to support a variety of compression algorithms. Consequently, Exchange layer and Storage layer in the protocol are extended by leveraging the component to achieve block-level compression and decompression in the workflows of upload and download for the acceleration of block transmission over an IPFS network. Furthermore, a block pre-request approach is proposed and incorporated into Exchange layer to improve the block request mechanism in the layer which frequently causes block request awaiting, reducing block provision speed for compression algorithms, thus downgrading the acceleration. A comprehensive evaluation indicates that this extended IPFS protocol by our scheme has the same level of data shareability as the native IPFS protocol, and contributes to block transmission performance enhancement in download by up to 69% and in upload by as much as 201% relative to the current leading coarse-grained compression scheme, referred to as IPFSz.