Malware detection and classification using transformer-based learning

Abstract

Malware detection and classification assumes significance owing to the rapid prop agation and proliferation of new variants. Grouping these variants into families with similar traits allows us to develop mitigation techniques that work for an entire family. Machine learning algorithms are used extensively for malware detection and classifica tion tasks with selected features taken from executable files. Although these methods have shown good performance, the choice of features chosen constrain their ability to detect novel malware samples. To alleviate this limitation, recently, deep learning methods are used, which propose to automate the feature engineering task by extract ing hidden semantic relationships between elements (raw bytes, opcodes, API calls, etc.) of the file. In this thesis, we describe Transformer-based malware detection and family iden tification methods. Our first proposed method uses static analysis to extract opcode sequences from executable files, which are used to train a Transformer-based model for Windows malware detection and classification. We show that our proposed method can perform malware detection using only a short sequence of opcodes taken from the portable executable files. A more sophisticated malware uses code obfuscation techniques or memory-based attacks to avoid detection. Such memory-based malware reside in the RAM to carry out their attacks. To detect the obfuscated malware, we use memory dumps obtained using dynamic analysis. We represent these memory dumps as images to be directly used as an input to a Transformer-based model. We also compare the detection and classification performance of Transformer-based model with a few conventional ma chine learning models. Our extensive experiments with different datasets demonstrate that our proposed techniques achieve better classification performance compared to recent methods. Malware is a threat not restricted to a single operating system. Android-based malware attacks have gained tremendous pace owing to the widespread use of mobile devices. Hence, we extend the previously mentioned technique that uses short sequences of opcodes to detect benign and malicious Android applications.