Title: Resource monitoring framework for big raw data processing

Authors: Mayank Patel; Minal Bhise

Addresses: Distributed Databases Research Group, Dhirubhai Ambani Institute of Information and Communication Technology (DA-IICT), Gandhinagar, Gujarat, India ' Distributed Databases Research Group, Dhirubhai Ambani Institute of Information and Communication Technology (DA-IICT), Gandhinagar, Gujarat, India

Abstract: Scientific experiments, simulations, and modern applications generate large amounts of data. Analysing resources required to process such big datasets is essential to identify application running costs for cloud or in-house deployments. Researchers have proposed keeping data in raw formats to avoid upfront utilisation of resources. However, it poses reparsing issues for frequently accessed data. The paper discusses detailed comparative analysis of resources required by in-situ engines and traditional database management systems to process a real-world scientific dataset. A resource monitoring framework has been developed and incorporated into the raw data query processing framework to achieve this goal. The work identified different query types best suited to a given data processing tool in terms of data to result time and resource requirements. The analysis of resource utilisation patterns has led to the development of query complexity aware (QCA) and resource utilisation aware (RUA) data partitioning techniques to process big raw data efficiently. Resource utilisation data have been analysed to estimate the data processing capacity of a given machine.

Keywords: big raw data; database management systems; DBMSs; in-situ engines; query processing; resource monitoring.

DOI: 10.1504/IJBDI.2024.138935

International Journal of Big Data Intelligence, 2024 Vol.8 No.2, pp.110 - 124

Received: 02 Apr 2022
Accepted: 18 Oct 2022
Published online: 04 Jun 2024 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article