Title: Multi-modal rumour detection using bilinear pooling and domain adversarial neural networks

Authors: Chao Wang; Hongwei Zhang; Jinrui Zhang; Lichuan Gu

Addresses: School of Information and Computer, Anhui Agricultural University, Key Laboratory of Agricultural Electronic Commerce of the Ministry of Agriculture, HeFei, 230036, China ' School of Information and Computer, Anhui Agricultural University, Key Laboratory of Agricultural Electronic Commerce of the Ministry of Agriculture, HeFei, 230036, China ' School of Information and Computer, Anhui Agricultural University, Key Laboratory of Agricultural Electronic Commerce of the Ministry of Agriculture, HeFei, 230036, China ' School of Information and Computer, Anhui Agricultural University, Key Laboratory of Agricultural Electronic Commerce of the Ministry of Agriculture, HeFei, 230036, China

Abstract: Rapid development in the internet era has made posting and obtaining information easier, leading to a sharp increase in rumour numbers. The images are more deceptive than traditional text rumours, making sources and authenticity hard to verify. Therefore, online rumours combining texts and images are more harmful. Detection of multi-modal rumours has become a new challenge. However, most existing methods are difficult to solve this problem and only adopt the standard concatenation for achieving feature fusion among different modes. Accordingly, the fused rumour features can barely effectively capture complementarity and difference among multi-modal data. This study aimed to propose an end-to-end model, named multi-modal rumour detection using bilinear pooling (BL) and domain adversarial neural networks (BPDANN), which adopts BL for multi-modal feature fusion to complement with the other. Further, the event classification module was designed to remove event-specific features and maintain shared features between events based on domain adversarial neural networks. Two text feature extraction methods and two BL methods were combined in pairs for multi-modal feature fusion to verify the effectiveness of BPDANN. Finally, the evaluation was conducted on two public multi-modal rumour datasets, Weibo and Twitter. The results exhibited that BPDANN outperformed current state-of-the-art methods.

Keywords: bilinear pooling? BL; deep learning? multi-modal fusion? rumour detection? social media.

DOI: 10.1504/IJSN.2023.134116

International Journal of Security and Networks, 2023 Vol.18 No.3, pp.175 - 188

Received: 09 Mar 2023
Accepted: 11 Mar 2023
Published online: 11 Oct 2023 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article