Detecting packed executable file: Supervised or anomaly detection method?

Abstract

Executable packing is an evasion technique used to propagate malware in the wild. Packing uses compression and/or encryption to thwart static analysis. There are universal unpackers available which can extract original binary from any type of packer, however they are computationally expensive as they are based on dynamic analysis which requires malware execution. A possible approach is to use machine learning techniques for classifying whether an executable is packed or not packed. Although supervised machine learning methods are good at learning packer specific features, these require collecting data from each packer and extracting features specific to it which may not be feasible practically. In this paper we propose a semi-supervised technique and an anomaly based detection method to identify packed executable files. We measure the distance between representative generated from a packed and non-packed binary training data and estimate the class based on its nearest distance in semi-supervised method. In anomaly detection we generate a representative cluster from known non-packed samples and find the radius of cluster and compare the distance of a test executable with that of radius to decide either it as normal or packed one. We experiment with few distance measures and report detection performance of these methods on two datasets. © 2016 IEEE.