Leveraging Memory Forensic Features for Explainable Obfuscated Malware Detection with Isolated Family Distinction Paradigm

Abstract

In the IoT edge computing era, inevitable and ubiquitous presence of the internet is opening the door for numerous cyberattacks. Obfuscated malware adds layers of difficulty to detect complex modern cyber attacks by evading AI-enabled Next-Generation Anti-Virus (NGAV) scanners and breaching digital privacy. To tackle this problem, in this paper, we propose “Augmented Sparse Projection Oblique Random Forest (AugSPORF)”, an Explainable sparse projections based Oblique Random Forest (ORF) with Isolated Family Distinction (IFD) Paradigm to detect multiple obfuscated malware belonging to Spyware, Ransomware, and Trojan families effectively. Irrespective of obfuscation, malware variants possess common behavior and family traits aligned with their families and leave traces in the memory on execution. To begin with this motivation, we handle the huge dimension of memory forensic features with sparse random projections. Next, we perform feature importance aware training with ORF to learn inherent behavioral features of malware families by isolating the target family, and distinguishing with other families. Further, the model's scalability is assessed by increasing the number of malware families. To offer an insightful conclusion on the predictions, an Interpretable Machine Learning (IML) layer is interleaved to generate a report of explanations, thereby enhancing the interpretability of the model. The proposed approach yields an average accuracy of 96.76%, 96.45%, and 97.33% in detecting sub-families of Spyware, Ransomware, and Trojan respectively. Improved accuracy is also demonstrated by benchmarking the performance of AugSPORF on UCI repository datasets. © 2025 Elsevier Ltd