Title: Big data block impact within big data environment

Authors: Ron Ziv; Oded Koren; Nir Perel

Addresses: School of Industrial Engineering and Management, Shenkar – Engineering. Design. Art., 12 Anne Frank St., Ramat-Gan, Israel ' School of Industrial Engineering and Management, Shenkar – Engineering. Design. Art., 12 Anne Frank St., Ramat-Gan, Israel ' School of Industrial Engineering and Management, Afeka Tel-Aviv Academic College of Engineering, 38 Mivtza Kadesh St., Tel-Aviv, Israel

Abstract: Handling data is becoming more and more complex. A higher velocity of data is created as more people have access to data generating devices such as computers, mobile phones, medical devices, home appliances, etc. Data files, such as user activity logs, system logs and so on, are stored in HDFS big data platform in various sizes, which takes into consideration the business requirements, infrastructure parameters, administration decisions, and other factors. Dividing the data files (in various volumes) without taking into consideration the HDFS™ predefined block size, may create performance issues that can affect the system's activity. This paper presents how HDFS™ block design affects the performance of Apache™ Hadoop® big data environment by testing different architectures for reading, writing, and querying identical datasets. We designed three scenarios to illustrate different file divisions on the big data platform. The findings present a significant impact on the performance of a system in accordance with the architecture deployed.

Keywords: information management; architecture; HDFS; performance; big data; block partition.

DOI: 10.1504/IJITM.2023.130064

International Journal of Information Technology and Management, 2023 Vol.22 No.1/2, pp.140 - 159

Accepted: 14 May 2019
Published online: 05 Apr 2023 *