Title: Towards higher efficiency in a distributed memory storage system using data compression

Authors: Xiaoyang Yu; Songfeng Lu; Tongyang Wang; Xinfang Zhang; Shaohua Wan

Addresses: Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China; School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China ' Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China ' Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China ' School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China ' Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen 518110, China

Abstract: As the amount of data grows, achieving an appropriate trade-off among computation, storage and network transportation will be beneficial for a distributed memory storage system, leading to higher overall efficiency. To this end, we explore a method to achieve this trade-off by introducing data compression technology in a transparent manner. Instead of focusing on specific compressed data structures, we target block level compression for a general-purpose storage system to incorporate a wide range of existing data analysis frameworks and usage scenarios, especially with big data. A prototype is implemented and evaluated based on the memory-centric distributed storage system Alluxio to provide transparent compression and decompression during write/read operations. The extensive experiments for data with different types of compression ratio are conducted and the experimental results prove that our approach can achieve huge write/read throughput.

Keywords: data storage systems; system performance; data compression; distributed memory storage.

DOI: 10.1504/IJBIC.2022.128090

International Journal of Bio-Inspired Computation, 2022 Vol.20 No.4, pp.232 - 240

Accepted: 27 Oct 2020
Published online: 05 Jan 2023 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article