Title: A reinforcement learning approach to map reduce auto-configuration under networked environment

Authors: Cheng Peng; Canqing Zhang; Cheng Peng; Junfeng Man

Addresses: School of Computer and Communication, Hunan University of Technology, Zhuzhou 412007, China; School of Information Science and Engineering, Central South University, Changsha 410083, China ' School of Computer and Communication, Hunan University of Technology, Zhuzhou 412007, China ' School of Energy, Power and Mechanical Engineering, North China Electric Power University, Beijing 102206, China ' School of Computer and Communication, Hunan University of Technology, Zhuzhou 412007, China; School of Information Science and Engineering, Central South University, Changsha 410083, China

Abstract: Hadoop-an open-source implementation of MapReduce is widely used for distributed processing model of large-scale data-intensive applications, configuration is crucial to the performance of MapReduce. As they expect that end users determine appropriate MapReduce parameters for running a job, which require in-depth knowledge of system and may lead to performance degradation. We propose a reinforcement learning approach to enable automated tuning configuration of MapReduce parameters, the RL approach has an initialisation policy with offline learning to reduce online learn time in different circumstance. Experimental results demonstrate that the approach can auto-configure the system, have better computers performance and shorter running time.

Keywords: Hadoop; parameter configuration; Q learning; performance optimisation.

DOI: 10.1504/IJSN.2017.084342

International Journal of Security and Networks, 2017 Vol.12 No.3, pp.135 - 140

Received: 20 Mar 2015
Accepted: 24 Mar 2015
Published online: 06 Jun 2017 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article