Title: Prospects, challenges and latest developments in designing a scalable big data stream computing system

Authors: Dawei Sun; Chunxiao Liu; Dongfeng Ren

Addresses: Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China; School of Information Engineering, China University of Geosciences, Beijing 100083, China ' College of Information Science and Technology, Bohai University, Jinzhou, Liaoning 121013, China ' Network Security Department, Neusoft Corporation, Shenyang 110179, China

Abstract: In the big data era, big data stream computing, as a computing paradigm, is gaining traction for real-time and online data computing applications, and is specially designed to solve the dilemma of real-time data stream computing by processing data online within real-time constraints. It is used to compute large amounts of data in the form of continuous data streams. Each computation is represented by a data stream graph, usually a directed graph. In this paper, the computing paradigm of big data stream computation in data stream graph is given, and some application scenarios and big data stream characteristics are presented. A series of challenges in designing a scalable big data stream computing system in big data environments are summarised by referring to some research results of big data stream computing system, which include stateless system architecture, elastically adaptive scheduling strategy and fine-grained fault tolerance strategy. All these challenges will greatly help us to understand big data steam computing, and provide a high-level guidance in designing an excellent and useful big data stream computing system.

Keywords: big data stream computing; task scheduling; system design; stateless system architecture; adaptive scheduling; fine-grained fault tolerance.

DOI: 10.1504/IJWMC.2015.072567

International Journal of Wireless and Mobile Computing, 2015 Vol.9 No.2, pp.155 - 160

Received: 14 Apr 2015
Accepted: 06 May 2015
Published online: 19 Oct 2015 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article