Malware Detection Using Hybrid Vision Transformer with CNN backbone

Abstract

Traditional signature-based malware detection techniques struggle to keep up with the evolving nature of malicious software. These traditional methods often fail to detect novel, obfuscated, or polymorphic malware, posing significant challenges for cybersecurity professionals. As a result, there has been a growing shift toward intelligent, learning-based techniques that can generalize across malware variants. In this paper, we propose a novel hybrid deep learning model that leverages the power of Convolutional Neural Networks (CNNs) and Vision Transformers (ViT) to enhance malware detection. The approach begins by converting executable (.exe) files into grayscale images, thereby transforming raw binary data into a visual domain compatible with modern computer vision models. CNNs are utilized for local feature extraction, capturing spatial relationships such as byte-level patterns and textures within the image, while ViT models global dependencies and contextual relationships across the image using self-attention mechanisms. This synergy enables the model to effectively detect both simple and complex malware families with high accuracy. The proposed architecture is evaluated on a benchmark malware image dataset and demonstrates superior performance, achieving an accuracy of 98.22% and precision 98.35%, along with excellent F1-Score and recall capability. The results indicate the model's robustness, scalability, and potential for integration into real-time malware detection systems to enhance cybersecurity infrastructure. © 2025 IEEE.