Novel support vector machine algorithms for classification problems

Abstract

In machine learning, support vector machine (SVM) [1], twin SVM (TWSVM) [16] and variants of TWSVM [8{14] have achieved a considerable popularity these days. However, TWSVM formulation still suffers from the following two shortcomings: (i) TWSVM deals with the inverse matrix calculation in the Wolfe-dual problems which is intractable for large scale datasets with numerous features and samples. (ii) TWSVM minimizes the empirical risk instead of the structural risk in its formulation. With the advent of huge amounts of data today, these disadvantages render TWSVM as an ineffective choice for pattern classifi cation tasks. We propose an efficient model entitled \Large scale least squares twin support vector machines (LS-LSTSVM)" for large scale analysis which recti es all the aforementioned shortcomings. The proposed model is a least squares version of twin bounded SVM (TBSVM) [20]. By introducing a different Lagrange function to the primal formulations, LS-LSTSVM only required to solve an unconstrained optimization dual problem without computing inverse matrix. Moreover, LS-LSTSVM also does not employ kernel generated surfaces for the nonlinear case, using the kernel trick directly; this ensures that the proposed LS-LSTSVM model is superior to the original TWSVM [16] and LSTSVM [18]. Further, We solve the dual formulation using sequential minimal optimization (SMO) technique. Compared to TBSVM, the proposed LS-LSTSVM leads to an exceedingly simple, fast and most appropriate algorithm for large scale analysis. Furthermore, we prove the convergence of SMO algorithm in this work. Numerical experiments on several real-world benchmark and NDC based large scale datasets demonstrate that the proposed LS-LSTSVM is feasible for large datasets, and, in most cases, outperforms the results obtained by the baselines models. The standard support vector machine (SVM) [1] with hinge loss function suffers from feature noise sensitivity and instability. Employing pinball loss function in SVM provides noise insensitivity to the model as it maximizes the quantile distance. However, pinball loss function simultaneously causes the model to lose sparsity by penalizing correctly classi fied samples. In order to overcome the aforementioned shortcoming, we propose a novel \Sparse support vector machine with pinball loss (Pin-SSVM)" for solving classifi cation problems. Pin-SSVM employs L1-norm in Pin-SVM which leads to a robust, sparse and noise insensitive model. Further, we can derive its solution simply by solving a linear programming problem (LPP) instead of solving quadratic programming problem (QPP) in SVM. The time complexity of the proposed Pin-SSVM is similar pin-SVM. Hence, solving an LPP with two linear inequality constraints does not effect the computational complexity. Exhaustive experiments on several benchmark UCI datasets, with and without noise, demonstrate the noise insensitivity and sparsity of the proposed Pin-SSVM.