Detecting Anomalous Behavior in VoIP Systems: A Discrete Event System Modeling

Abstract

Session initiation protocol (SIP) is an application layer protocol used for signaling purposes to manage voice over IP connections. SIP being a text-based protocol is vulnerable to a range of denial of service (DoS) attacks. These DoS attacks can render the SIP servers/SIP proxy servers unusable by depleting memory and CPU time. In this paper, we consider two types of DoS attacks, namely, flooding attacks and coordinated attacks for detection. Flooding attacks affect both stateless and stateful SIP servers while coordinated attacks affect stateful SIP servers. We model the SIP operation as discrete event system (DES) and design a new state transition machine, which we name as probabilistic counting deterministic timed automata (PCDTA) to describe the behavior of SIP operations. We also identify different types of anomalies that can occur in a DES model, which appear in the form of illegal transitions, violating timing constraints, and appear in number which is otherwise not seen. Subsequently, we map various DoS attacks in SIP to a type of anomaly in DES. PCDTA can learn probabilities of various transitions and timings delay from a set of nonmalicious training sequences. A trained PCDTA can detect anomalies, and hence various DoS attacks in SIP. We perform a thorough experiment with computer simulated SIP traffic and report the detection performance of PCDTA on various attacks generated through custom scripts. © 2005-2012 IEEE.