Title: A data cleaning method for water quality based on improved hierarchical clustering algorithm

Authors: Qingxuan Meng; Jianzhuo Yan

Addresses: College of Electronic Information and Control Engineering, Beijing University of Technology, Beijing, China ' College of Electronic Information and Control Engineering, Beijing University of Technology, Beijing, China

Abstract: Identifying and rectifying incomplete water quality data is of vital importance. A data cleaning method based on improved balanced iterative reducing and clustering using hierarchies (BIRCH) clustering algorithm is proposed. The clustering feature tree of water quality data is constructed and the cluster vector of the clustering feature tree is obtained by the agglomerative method. The optimal cluster number is determined according to the Bayesian Information Criterion and the nearest clustering ratio. The Pauta criterion is used to detect the global outlier and artificial neural network (ANN) is used to fill in outliers and missing values. Finally, the improved data cleaning method is applied to water quality monitoring data of Beijing wastewater treatment plant. The experimental results show that the data cleaning method can not only detect abnormal values and missing values accurately, but also normalise and complete missing data.

Keywords: outliers; water quality monitoring; multivariate data; clustering; artificial neural network; ANN.

DOI: 10.1504/IJSPM.2019.104120

International Journal of Simulation and Process Modelling, 2019 Vol.14 No.5, pp.442 - 451

Received: 04 Sep 2018
Accepted: 04 Mar 2019
Published online: 14 Dec 2019 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article