Resource monitoring framework for big raw data processing
by Mayank Patel; Minal Bhise
International Journal of Big Data Intelligence (IJBDI), Vol. 8, No. 2, 2024

Abstract: Scientific experiments, simulations, and modern applications generate large amounts of data. Analysing resources required to process such big datasets is essential to identify application running costs for cloud or in-house deployments. Researchers have proposed keeping data in raw formats to avoid upfront utilisation of resources. However, it poses reparsing issues for frequently accessed data. The paper discusses detailed comparative analysis of resources required by in-situ engines and traditional database management systems to process a real-world scientific dataset. A resource monitoring framework has been developed and incorporated into the raw data query processing framework to achieve this goal. The work identified different query types best suited to a given data processing tool in terms of data to result time and resource requirements. The analysis of resource utilisation patterns has led to the development of query complexity aware (QCA) and resource utilisation aware (RUA) data partitioning techniques to process big raw data efficiently. Resource utilisation data have been analysed to estimate the data processing capacity of a given machine.

Online publication date: Tue, 04-Jun-2024

The full text of this article is only available to individual subscribers or to users at subscribing institutions.

 
Existing subscribers:
Go to Inderscience Online Journals to access the Full Text of this article.

Pay per view:
If you are not a subscriber and you just want to read the full contents of this article, buy online access here.

Complimentary Subscribers, Editors or Members of the Editorial Board of the International Journal of Big Data Intelligence (IJBDI):
Login with your Inderscience username and password:

Username:        Password:          Forgotten your password?

Want to subscribe?
A subscription gives you complete access to all articles in the current issue, as well as to all articles in the previous three years (where applicable). See our Orders page to subscribe.

If you still need assistance, please email subs@inderscience.com