Title: Resource management for deadline constrained MapReduce jobs for minimising energy consumption

Authors: Adam Gregory; Shikharesh Majumdar

Addresses: Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada ' Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada

Abstract: Cloud computing has emerged as one of the leading platforms for processing large-scale data intensive applications. Such applications are executed in large clusters and data centres which require a substantial amount of energy. Energy consumption within data centres accounts for a considerable fraction of costs and is a significant contributor to global greenhouse gas emissions. Therefore, minimising energy consumption in data centres is a critical concern for data centre operators, cluster owners, and cloud service providers. In this paper, we devise a novel energy aware MapReduce resource manager for an open system, called EAMR-RM, that can effectively perform matchmaking and scheduling of MapReduce jobs each of which is characterised by a service level agreement (SLA) for performance that includes a client specified earliest start time, execution time, and a deadline with the objective of minimising data centre energy consumption. Performance analysis demonstrates that for a range of system and workload parameters experimented with the proposed technique can effectively satisfy SLA requirements while achieving up to a 45% reduction in energy consumption compared to approaches which do not consider energy in resource management decisions.

Keywords: resource management on clouds; MapReduce with deadlines; constraint programming; energy management; big data analytics; job turnaround time; big data; service level agreement.

DOI: 10.1504/IJBDI.2018.094996

International Journal of Big Data Intelligence, 2018 Vol.5 No.4, pp.270 - 287

Received: 14 Feb 2017
Accepted: 05 Sep 2017
Published online: 28 Sep 2018 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article