Title: Toward signature extraction of Metasploit encoding algorithms using static analysis

Authors: Mohammadreza Ramezani-Chemazi; Maede Ashouri-Talouki

Addresses: Department of IT Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran ' Department of IT Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran

Abstract: Shellcode is a code injected by the attackers to vulnerable software to gain access to the command prompt. The byte patterns of shellcodes help the intrusion detection systems to detect this type of shellcodes. To avoid detection, encoding algorithms is used by the attacker to encode the byte patterns. The detection of these encoded shellcodes is a challenging problem. To detect these encoded shellcodes, we perform a static analysis of encoding algorithms of Metasploit engine to extract the byte patterns (signature) of these algorithms. Then, we introduce a regular expression-based language called GtS to express these signatures. The experimental results show the effectiveness of our signatures in terms of accuracy and false positive rate.

Keywords: shellcode; Metasploit; encoding algorithms; static analysis; signatures.

DOI: 10.1504/IJSN.2018.092472

International Journal of Security and Networks, 2018 Vol.13 No.2, pp.71 - 83

Received: 14 Sep 2016
Accepted: 05 Jan 2018
Published online: 21 Jun 2018 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article