Title: Refining malware detection with enhanced machine learning algorithms using hyperparameter tuning

Authors: Walid El Mouhtadi; Mohamed El Bakkali; Yassine Maleh; Soufyane Mounir; Karim Ouazzane

Addresses: LaSTI Laboratory, National School of Applied Sciences, Sultan Moulay Slimane University, Khouribga, Morocco ' Mohammadia School of Engineering EMI, Mohammed V University, Rabat, Morocco ' LaSTI Laboratory, National School of Applied Sciences, Sultan Moulay Slimane University, Khouribga, Morocco ' LaSTI Laboratory, National School of Applied Sciences, Sultan Moulay Slimane University, Khouribga, Morocco ' Cyber Security Research Centre, London Metropolitan University, London N7 8DB, UK

Abstract: The aim of this research is to investigate and demonstrate the advantages and limitations of various machine learning techniques for malware classification, specifically focusing on portable executable (PE) files. The study addresses common challenges in machine learning, such as overfitting and underfitting, by employing ensemble methods and pre-processing techniques, including feature selection and hyperparameter tuning. The primary objective is to enhance classifier performance in distinguishing between malicious and benign PE files. Through a comparative analysis of machine learning methodologies such as random forests, decision trees, and gradient boosting, the study highlights the superiority of the random forests algorithm, achieving an impressive accuracy rate of 99%. By thoroughly evaluating the strengths and limitations of each algorithm, the research provides valuable insights into effectively handling diverse malware categories. This paper underscores the significance of ensemble methods, feature engineering, and pre-processing in improving classifier performance for malware classification, specifically in the context of portable executable files.

Keywords: malware detection; machine learning; ML; optimisation; hyperparameter tunning; data balancing; feature selection.

DOI: 10.1504/IJCCBS.2024.139100

International Journal of Critical Computer-Based Systems, 2024 Vol.11 No.1/2, pp.48 - 67

Received: 06 Sep 2023
Accepted: 06 Nov 2023
Published online: 13 Jun 2024 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article