Title: Design and application of real-time network abnormal traffic detection system based on Spark Streaming

Authors: FuCheng Pan; DeZhi Han; Yuping Hu

Addresses: College of Information Engineering, Shanghai Maritime University, Shanghai, 201306,China ' College of Information Engineering, Shanghai Maritime University, Shanghai, 201306,China ' School of Information, Guangdong University of Finance and Economics, Guangdong, 510320, China

Abstract: In order to realise the rapid analysis and identification of abnormal traffic in real-time networks, a distributed real-time network abnormal traffic detection system (DRNATDS) was designed, which could effectively analyse abnormal network traffic. DRNATDS provided effective real-time big data analysis platform and guaranteed network security. The paper proposes K-means algorithm based on relative density and distance, integrated with Spark Streaming and Kafka. It could effectively detect various network attacks under real-time data stream. The experimental results show that DRNATDS has good high availability and stability. Compared to other algorithms, K-means algorithm based on relative density and distance could more effectively identify abnormal network traffic and improve the recognition rate.

Keywords: Spark Streaming; Kafka; network abnormal traffic identification; K-means.

DOI: 10.1504/IJES.2019.102428

International Journal of Embedded Systems, 2019 Vol.11 No.5, pp.562 - 572

Received: 28 Jul 2018
Accepted: 27 Sep 2018
Published online: 24 Sep 2019 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article