Article: A framework for collective I/O style optimisations at staging I/O nodes Journal: International Journal of Big Data Intelligence (IJBDI) 2016 Vol.3 No.2 pp.79 - 91 Abstract: Data-intensive applications are largely influenced by the I/O performance on high performance computing systems, and the scalability of such applications primarily depends on the scalability of the I/O subsystem itself. To mitigate the effects of I/O operations, recent high end systems make use of staging nodes to delegate I/O requests and thus decouple I/O and compute operations. The data staging layers however, lack the benefit that could be obtained from the collective I/O style optimisations that client side parallel I/O operations provide. In this paper, we present the compactor framework developed as a part of the exascale fastforward I/O stack. The compactor framework introduces optimisations at the data staging nodes, including collective buffering across requests from multiple processes, write stealing to service read requests at the staging node, and write morphing to optimise multiple write requests from the same process. The compactor framework is evaluated on a PVFS2 file system using micro-benchmarks, the flash I/O benchmark and a parallel image processing application. Our results indicate significant performance benefits of up to 70% due to the optimisations of the compactor. Inderscience Publishers - linking academia, business and industry through research

Title: A framework for collective I/O style optimisations at staging I/O nodes

Authors: Vishwanath Venkatesan; Mohamad Chaarawi; Quincey Koziol; Neil Fortner; Edgar Gabriel

Addresses: Department of Computer Science, University of Houston, 4800 Calhoun Blvd, Houston, TX 77204, USA ' The HDF Group, 1800 South Oak Street, Suite 203, Champaign, IL, 61820, USA ' The HDF Group, 1800 South Oak Street, Suite 203, Champaign, IL, 61820, USA ' The HDF Group, 1800 South Oak Street, Suite 203, Champaign, IL, 61820, USA ' Department of Computer Science, University of Houston, 4800 Calhoun Blvd, Houston, TX 77204, USA

Abstract: Data-intensive applications are largely influenced by the I/O performance on high performance computing systems, and the scalability of such applications primarily depends on the scalability of the I/O subsystem itself. To mitigate the effects of I/O operations, recent high end systems make use of staging nodes to delegate I/O requests and thus decouple I/O and compute operations. The data staging layers however, lack the benefit that could be obtained from the collective I/O style optimisations that client side parallel I/O operations provide. In this paper, we present the compactor framework developed as a part of the exascale fastforward I/O stack. The compactor framework introduces optimisations at the data staging nodes, including collective buffering across requests from multiple processes, write stealing to service read requests at the staging node, and write morphing to optimise multiple write requests from the same process. The compactor framework is evaluated on a PVFS2 file system using micro-benchmarks, the flash I/O benchmark and a parallel image processing application. Our results indicate significant performance benefits of up to 70% due to the optimisations of the compactor.

Keywords: parallel I/O; input-output; data staging nodes; collective I/O; HDF5; exascale fastforward I/O; high performance computing; optimisation; parallel processing; image processing.

DOI: 10.1504/IJBDI.2016.077359

International Journal of Big Data Intelligence, 2016 Vol.3 No.2, pp.79 - 91

Received: 22 Aug 2014
Accepted: 11 Feb 2015
Published online: 29 Jun 2016 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article

Title: A framework for collective I/O style optimisations at staging I/O nodes

Keep up-to-date