Title: A new replica placement strategy based on multi-objective optimisation for HDFS

Authors: Yangyang Li; Mengzhuo Tian; Yang Wang; Qingfu Zhang; Dhish Kumar Saxena; Licheng Jiao

Addresses: Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an, China ' Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an, China ' Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an, China ' Department of Computer Science, City University of Hong Kong, Hong Kong ' Department of Mechanical and Industrial Engineering, Indian Institute of Technology Roorkee, Roorkee, India ' Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an, China

Abstract: Distributed storage systems like the Hadoop distributed file system (HDFS) constitute the core infrastructure of cloud platforms which are well poised to deal with big-data. An optimised HDFS is critical for effective data management in terms of reduced file service time and access latency, improved file availability and system load balancing. Recognising that the file-replication strategy is key to an optimised HDFS, this paper focuses on the file-replica placement strategy while simultaneously considering storage and network load. Firstly, the conflicting relationship between storage and network load is analysed and a bi-objective optimisation model is built, following which a multi-objective optimisation memetic algorithm based on decomposition (MOMAD) and its improved version are used. Compared to the default strategy in HDFS, the file-replica placement strategies based on multi-objective optimisation provide more diverse solutions. And competitive performance could be obtained by the proposed algorithm.

Keywords: Hadoop; Hadoop distributed file system; HDFS; replica placement; multi-objective optimisation; memetic algorithm.

DOI: 10.1504/IJBIC.2020.108994

International Journal of Bio-Inspired Computation, 2020 Vol.16 No.1, pp.13 - 22

Accepted: 08 Dec 2018
Published online: 14 Aug 2020 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article