Title: Classification of diversified web crawler accesses inspired by biological adaptation

Authors: Naomi Kuze; Shu Ishikura; Takeshi Yagi; Daiki Chiba; Masayuki Murata

Addresses: Graduate School of Engineering Science, Osaka University, Osaka, Japan ' Graduate School of Information Science and Technology, Osaka University, Osaka, Japan ' NTT Secure Platform Laboratories, Tokyo, Japan ' NTT Secure Platform Laboratories, Tokyo, Japan ' Graduate School of Information Science and Technology, Osaka University, Osaka, Japan

Abstract: To discover and prevent attacks, it is necessary to collect data about the attacks using honeypots and to identify malicious accesses from collected data. In this study, we focus on detecting a massive number of crawler accesses, which complicates the detection of malicious accesses. We adapt AntTree, a bio-inspired clustering scheme that is highly scalable and adaptable, for crawler detection. We also designed a feature vector for crawler detection and propose a cluster interpretation method of AntTree. Our results show that the proposed bio-inspired mechanism can detect crawlers with a low false-negative rate, which is an advantage over conventional schemes for detecting various types of crawler.

Keywords: network security; web vulnerability scanning detection; web honeypots; ant-based clustering.

DOI: 10.1504/IJBIC.2021.114877

International Journal of Bio-Inspired Computation, 2021 Vol.17 No.3, pp.165 - 173

Accepted: 30 Aug 2020
Published online: 10 May 2021 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article