Article: De-duplication of aggregation authority files Journal: International Journal of Metadata, Semantics and Ontologies (IJMSO) 2012 Vol.7 No.2 pp.114 - 130 Abstract: This paper presents PACE (Programmable Authority Control Engine), an authority control tool conceived to maintain 'aggregation authority files'. These are obtained as continuous aggregations of records originating from a variable set of information systems with heterogeneous and duplicated content. To facilitate record deduplication in the presence of such heterogeneity and dynamicity, PACE user interfaces enable an iterative curation process, where data curators can: (i) configure algorithms for the identification of record duplicates; (ii) open work sessions where algorithm configurations can be run and evaluated; (iii) merge the identified record duplicates to disambiguate the authority file and (iv) repeat this cycle several times. PACE supports a tunable probabilistic similarity measure and performs record matching with a customisable variation of the sorted neighbourhood heuristic. Finally, it addresses the underlying performance and scalability issues by exploiting multi-core parallel processing and Cassandra's storage systems, to support I/O performances that scale up linearly with the number of records. Inderscience Publishers - linking academia, business and industry through research

Title: De-duplication of aggregation authority files

Authors: P. Manghi; M. Mikulicic; C. Atzori

Addresses: Istituto di Scienza e Tecnologie dell'Informazione 'Alessandro Faedo', Consiglio Nazionale delle Ricerche, Via G. Moruzzi 1, 56124, Pisa, Italy ' Istituto di Scienza e Tecnologie dell'Informazione 'Alessandro Faedo', Consiglio Nazionale delle Ricerche, Via G. Moruzzi 1, 56124, Pisa, Italy ' Istituto di Scienza e Tecnologie dell'Informazione 'Alessandro Faedo', Consiglio Nazionale delle Ricerche, Via G. Moruzzi 1, 56124, Pisa, Italy

Abstract: This paper presents PACE (Programmable Authority Control Engine), an authority control tool conceived to maintain 'aggregation authority files'. These are obtained as continuous aggregations of records originating from a variable set of information systems with heterogeneous and duplicated content. To facilitate record deduplication in the presence of such heterogeneity and dynamicity, PACE user interfaces enable an iterative curation process, where data curators can: (i) configure algorithms for the identification of record duplicates; (ii) open work sessions where algorithm configurations can be run and evaluated; (iii) merge the identified record duplicates to disambiguate the authority file and (iv) repeat this cycle several times. PACE supports a tunable probabilistic similarity measure and performs record matching with a customisable variation of the sorted neighbourhood heuristic. Finally, it addresses the underlying performance and scalability issues by exploiting multi-core parallel processing and Cassandra's storage systems, to support I/O performances that scale up linearly with the number of records.

Keywords: authority control; record deduplication; record merge; record aggregation; sorted neighbourhood; candidate identification; performance; scalability; Cassandra; metadata; ontologies; semantics; aggregation authority files; information systems.

DOI: 10.1504/IJMSO.2012.050014

International Journal of Metadata, Semantics and Ontologies, 2012 Vol.7 No.2, pp.114 - 130

Received: 27 Dec 2011
Accepted: 19 Feb 2012
Published online: 31 Dec 2014 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article

Title: De-duplication of aggregation authority files

Keep up-to-date