Article: A feature-level attention-based deep neural network model for sentence embedding Journal: International Journal of Intelligent Systems Technologies and Applications (IJISTA) 2022 Vol.20 No.5 pp.414 - 435 Abstract: Building a model to represent the semantic of a sentence is crucial for going beyond a sequence of words to a more abstract yet relevant representation. The relevance of such models is ubiquitous in various natural language processing tasks. Following this context, we capitalise on deep learning for proposing a new model-based attention mechanism. Our contribution aims at embedding the sentence meaning in an unsupervised iterative way. The word and the sentence embeddings therefore influence each other. Our model is inspired from the recursive auto-encoders. We coupled our model with a novel attention mechanism computed at the feature-level. This mechanism aims to increase representation power by focusing on important features of words within a sentence to refine the constructed meaning representation of this sentence. To highlight our newly proposed contribution, we carried out an exhaustive experimental study for evaluating the quality of the learned representations on semantic similarity task. The obtained results demonstrate the faithfulness of our learned semantic representations. Inderscience Publishers - linking academia, business and industry through research

Title: A feature-level attention-based deep neural network model for sentence embedding

Authors: Amal Bouraoui; Salma Jamoussi; Abdelmajid Ben Hamadou

Addresses: MIRACL Laboratory, Higher Institute of Computer Science and Multimedia, Sfax University, Technopole of Sfax: Av. Tunis Km 10, B.P. 242, Sfax-3021, Tunisia ' MIRACL Laboratory, Higher Institute of Computer Science and Multimedia, Sfax University, Technopole of Sfax: Av. Tunis Km 10, B.P. 242, Sfax-3021, Tunisia ' MIRACL Laboratory, Higher Institute of Computer Science and Multimedia, Sfax University, Technopole of Sfax: Av. Tunis Km 10, B.P. 242, Sfax-3021, Tunisia

Abstract: Building a model to represent the semantic of a sentence is crucial for going beyond a sequence of words to a more abstract yet relevant representation. The relevance of such models is ubiquitous in various natural language processing tasks. Following this context, we capitalise on deep learning for proposing a new model-based attention mechanism. Our contribution aims at embedding the sentence meaning in an unsupervised iterative way. The word and the sentence embeddings therefore influence each other. Our model is inspired from the recursive auto-encoders. We coupled our model with a novel attention mechanism computed at the feature-level. This mechanism aims to increase representation power by focusing on important features of words within a sentence to refine the constructed meaning representation of this sentence. To highlight our newly proposed contribution, we carried out an exhaustive experimental study for evaluating the quality of the learned representations on semantic similarity task. The obtained results demonstrate the faithfulness of our learned semantic representations.

Keywords: sentence embedding; semantics; recursive auto-encoders; attention mechanism; semantic similarity; feature-level attention.

DOI: 10.1504/IJISTA.2022.125610

International Journal of Intelligent Systems Technologies and Applications, 2022 Vol.20 No.5, pp.414 - 435

Accepted: 08 Feb 2022
Published online: 16 Sep 2022 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article

Title: A feature-level attention-based deep neural network model for sentence embedding

Keep up-to-date