Construction and Practice of the Optimal Smooth Semi-Supervised Support Vector Machine

1 Introduction

Support vector machines (SVMs) were introduced by Vapnik^[1] in the early 1990s. As an effective method of data mining, SVMs are widely applied in many fields ranging from text categorization^[2], image retrieval^[3], face recognition^[4], intelligent goal-driven^[5], information security^[6] to credit risk evaluation^[7], bankruptcy prediction^[8], time series forecasting^[9], etc. The standard SVM is based on supervised learning. In this result, a large number of labeled samples are required to ensure the preferable classification accuracy. But manual labelling is often a slow, expensive, and error prone process. By contrast, the semi-supervised support vector machines (S³VMs)^[10–12] can take advantage of labeled samples and unlabeled samples simultaneously, and then reduce the labelling cost significantly. Hence, S³VM has widely attracted attention of some researchers^[13–15]. Since the unconstrained optimization problem of S³VM is non-convex and non-smooth, most of the fast algorithms cannot be used to solve the S³VMs problem. With this regard, many famous researchers were involved in the development of smooth S³VM (SS³VM) in recent years^[15–18]. The SS³VM models, smoothed by Gaussian function^[12], polynomial functions^[19], and the cubic spline function^[20], have been proposed successively. However, the influences of all parameters of SS³VM on the classification results have not been deeply studied. In fact, the effects of parameters on the performance of SS³VM cannot be ignored. Therefore, selecting the appropriate parameters of SS³VM is crucial.

In this paper, we firstly explore the smoothing method and propose a new SS³VM model based on a biquadratic spline function. Secondly, we focus on the optimization of parameters of the SS³VM models. A hybrid approach combined with the Genetic Algorithm (GA)^[21] and SS³VM is introduced. Finally, we evaluate the new model through numerical experiments. Four dataset are used to test the efficiency of the new model. Moreover, the effect of the number of labeled samples on the classification accuracy of the new model is analyzed.

2 The SS³VM Model

We consider the two-class classification problem. The dataset consist of m labeled samples xxi,yii=1m and l unlabeled samples x¯x¯ii=1l, where xxi,x¯x¯i∈RRn,yi∈1,−1.xxi(i=1,2,⋯,m)andx¯x¯i(i=1,2,⋯,l) are represented by an m × n matrix A_m×n and B_l×n, respectively. The result of classification for labeled samples is determined by an m × m diagonal matrix D with 1 or –1 along its diagonal. Precisely, a standard unconstrained optimization model of S³VM is given as follows^[22]:

Here, ω is the normal vector to the bounding plane, b is a bias value, e_i (i = 1, 2) is a column vector of 1’s, with e₁∈ R^m, e₂ ∈ R^l. c and c* are the penalty parameters. Ʌ(t) = max(0, 1 –t) is called the hinge loss function and Ʌ(|t|) = max(0, 1 – |t|) is called the symmetric hinge loss function^[6]. If u = (u₁, u₂, · · · , u_n) ∈ Rⁿ, then Ʌ(u) = (Ʌ(u₁), Ʌ(u₂), · · · , Ʌ(u_n))^T. Nevertheless, the Ʌ(t) and Ʌ (|t|) are not differentiable. As a result, the objective function in the Model (1) is non-convex and non-smooth to which precludes the applications of many fast optimization methods such as BFGS algorithm[²³, ²⁵]. So we modify the Model (1) as below:

Here, f(x, r) is an arbitrary smooth function at origin, which is used to approximate the symmetric hinge loss function and r is the relevant parameter. The 1-norm being substituted by 2-norm avoid the non-smoothness of the hinge loss function Ʌ(x)at +1 and the symmetric hinge loss function at +1 and –1. In next section, a biquadratic spline function is constructed to replace the f(x, r)in Model (2).

3 The SS³VM Model Based on the Biquadratic Spline Function

3.1 Construction of Biquadratic Spline Function

Definition 1

Fork>1,y1>0,m,n∈Z+,letx0=−1k,x1=0,x2=1kbe a set of nodes. The function s(x, k) is defined as follows:

s(x,k)=s0(x,k),−1k≤x<0s1(x,k),0≤x≤1kΛx,x>1k(3)

Here s₀(x,k) and s₁(x,k) are the n-times polynomials. Ifs(x, k) satisfies the following conditions:

1. s(d)(x0,k)=0,d=2,3,⋯,m;k>0,s′(x0,k)=1,s(x0,k)=1−1k;

2. s(d)(x2,k)=0,d=2,3,⋯,m;k>0,s′(x2,k)=−1,s(x2,k)=1−1k;

3. s0(d)x1−0,k=s1d(x1+0,k),d=1,2,⋯,m,s0(x1+0,k)=s1(x1−0,k)=y1.

Then it is called the n-times spline function with the m-orders smooth condition at origin which approximating Λ(|x|).

Theorem 1

Letk>1andx0=−1k,x1=0,x2=1kbe a set of nodes. Then there exists a unique biquadratic spline s(x, k) with the second derivative at origin approximating Λ(|x|). The function must have the following expression:

s(x,k)=18k3x4−34kx2−38k+1,−1k≤x≤1kΛx,x>1k(4)

Proof

Let s(x, k) be a quartic spline function with the second derivative at origin, which satisfies the conditions in Definition 1. We derive the equation for s(x, k) on −1k,1k. Set s⁽³⁾(x_i, k) = M_i(i = 0, 1, 2). ∀x∈−1k,0, it has s(x, k) = s₀(x, k). Since s₀(x, k) is a polynomial of degree four on [−1k,0],s₀⁽³⁾(x, y) is a linear function. The expression of s₀⁽³⁾(x, y) can be given by

s0(3)(x,k)=M0(x1−x)h0+M1(x−x0)h0=−M0kx+M1k(x−x0)(5)

The above equation is integrated successively 3 times, we obtain:

s0(x,k)+a12x2+a2x+a3=−M0kx44!+M1k4!(x−x0)4

where a₁, a₂, and a₃ are constants of the integration. According to condition (a) in Definition 1, we can determine: a1=−M02k,a2=−M03k2−1,a3=−M08k3−1.

Similarly, on 0,1k, we have

s1(x,k)+b12x2+b2x+b3=M2kx44!−M1k4!(x2−x)4(6)

where b₁, b₂, and b₃ are constants of the integration. Based on condition (b) in Definition 1, it follows that b1=M22k,b2=−M23k2+1,b3=M28k3−1.

In this way, we show that s(x, k) is piecewise polynomial of degree 4 with parameters M₀, M₁, and M₂ on −1k,1k. Furthermore, applying condition (c) in Definition 1, we obtain the matrix equation:

32313k20−13k2121⋅M0M1M2=0−20(7)

Because the coefficient matrix of Equation (7) is nonsingular, we can get the unique solution M₀ = –3k², M₁ = 0, and M₂ = 3k². Finally, the biquadratic spline function s(x, k) with the second derivative at origin approximating Λ(|x|) is obtained as Equation (4).

Theorem 2

Let Λ(|x|) be the symmetric hinge loss function and s(x, k) be the general biquadratic spline function given in Model (4). Then, for x ∈ Rand k > 1, we have:

1. 0 ≤ s(x, k) ≤ Λ(|x|);

2. 0≤Λ2x−s2(x,k)≤38k(2−38k).

Proof

If x∈1k,+∞∪−∞,−1k, both the inequalities (a) and (b) are true, since s(x, k) = Λ(|x|). It remains to show that the inequalities are true on x∈−1k,1k.

(a). If x∈−1k,0, then s(x,k)=s0(x,k)=18k3x4−34kx2−38k+1. Thus s0(2)(x,k)=32k3x2−32k≤0⇒minxs′0(x,k)=s′0(0,k)=0. Hence, s(x, k) is an increasing function. It follows that

1−38k=s0(x,k)≥s0(−1k,k)=1−1k≥0(8)

In addition, let λ(x,k)=Λx−s0(x,k)=x−18k3x4+34kx2+38k. Then

λ(2)(x,k)=−32k3x2+32k≥0⇒λ′(k,x)≥0(9)

Therefore, λ(x, k) is also an increasing function on −1k,0.minx⁡λ(x,k)=λ(−1k,k)=0 is obtained. So 0 ≤ s₀(x, k) < Λ(|x|), when x∈−1k,0.

If x∈0,1k,s(x, k) = s₁(x, k). Similar to the first part, the inequality (a) can also be obtained.

(b). If x∈−1k,0, since λ(x, k) is an increasing, the maximum 38k can be obtained at the point x = 0. Hence,

0≤Λx−s0(x,k)≤38k.(10)

In addition, Λ(|x|) + s₀(x, k) = λ(x, k) + 2s₀(x, k). Inequalities (8) and (10) imply that: Λx+s0(x,k)≤2−38k.

Thus, 0≤Λ2(|x|)−s02(x,k)≤38k(2−38k).

Similarly, if xx∈0,1k,wehave0≤Λ2x−s2(x,k)≤38k(2−38k) as well.

In order to show the approximating accuracy of different smooth functions, the comparison diagram of smooth performance is shown in Figure 1.

Figure 1

Different smooth functions approximate Λ(|x|)

As shown in Figure 1, the biquadratic spline function is closer to the symmetric hinge loss function than both the Gaussian function and the polynomial function. Though the approximate accuracy of the biquadratic spline function is slightly lower than that of the cubic spline function, the biquadratic spline function has more simple expression.

4 The Optimal SS³VM Model Based on GA

One of the big problems in SS³VM is the selection of the values of parameters that will allow good performance. But, it is not known beforehand what values are the best for the model. In order to select the suitable parameters of the SS³VM, the GA is applied^{[21, 26–28]}. GA is an artificial intelligence procedure based on the theory of natural selection and evolution. Unlike the conventional optimization methods, it has the advantages consisting of parallel search, solving complex problems, and large search space.

The SS³VM training algorithm and GA are combined to optimize the parameters of SS³VM. For ease of notation, we named this hybrid method as GA/SS³VM method. Figure 2 shows the overall procedure of the GA/SS³VM method.

Figure 2

Overall procedure of GA/SS³VM method

The detailed steps of GA/SS³VM method are given as below:

1. Define the string (or chromosome).

   According to the Model (2), the parameters c, c*, and r need to be optimized. So, the individual is defined as x = (c, c*, r). The chromosome of x is encoded as a l-bit string which consists of l₁ bits standing for c, l₂ bits standing for c*, and l₁ bits standing for r. Here, l = (l₁+l₂ + l₃).

2. Determine the fitness function.

   The fitness of an individual of population is based on the performance of SVM. So, the prediction accuracy of SS³VM is taken as the fitness function F(x).

3. Initialization.

   1. Define the size of population N, probability of crossover P_c, and probability of mutation P_m.

   2. Generate an initial population consisted of N l-bit strings described in Step 1 randomly: P(L)={xxj(L)=(cj,cj∗,rj)L,j=1,2,⋯,N}, where L is the number of the current generation population. In initial generation, L = 0.

4. Decode the jth string to obtain the corresponding individual xxjL=(cj,cj∗,rj)L.

5. Apply x_j(L) to the SS³VM model to compute the Fitness F (x_j (L)).

6. Evolution;

   1. Find the worst fitness FminL, the best fitness FmaxL, and the corresponding xxminL in the Lth generation population.

   2. Replace xxminLwithxxminL.

7. Calculate the fitness of the Lth generation population: TL=∑j=1NFxxjL.

8. Reproduction.

   1. Compute cumulative probabilities qj=∑i=1jpi(j=1,2,⋯,N),wherepi=FxxiLTL

   2. Generate N random numbers r_1i, i = 1, 2, · · ·, N in [0, 1]. For each r_1i, If q_j–1 < r_1i ≤ q_j, then select the jth string, otherwise, select the first string such that r_1i ≤ q₁.

9. Generate offspring population P(L + 1) by performing crossover and mutation.

   1. Crossover: For two parent individuals, a random number r₂in [0, 1] is generated. If r₂ < P_c, choose a random crossover point and exchange the genetic code of the two parent individuals on this point to obtain two new child individuals.

   2. Mutation: Generate a random number r₃ in [0, 1] and select a bit randomly. If r₃ < P_m, then operate mutation for the bit.

10. If the terminal condition is satisfied, output the best individual xxmax(L+1). Otherwise, do Steps 4~10. Terminal condition: The maximum number of generations L_max is reached or the optimum individual does not improve during successive generations.

5 Data Preparation

6 Numerical Experiments

6.2 Experimental Results and Comparisons

In this section, the results that corresponding to the three parts of the experiments are obtained and analysed. Figure 3 shows the evolution process of the optimal parameters selection on Heart dataset. As it is shown in Figure 3, for four SS³VM models, the stop criterion of GA process is satisfied at about 10th or 15th generation (L = 10 or L = 15), i.e., the optimal parameters of four SS³VM models are obtained at about 10th or 15th generation. The procedures of the parameters selection for QSAR, Wine, and Wilt datasets are similar to that of the Heart dataset. It is emphasized that the selection procedures are distinct for different training sets. Moreover, the process is likely not constant for same training sets because of the stochastic of GA. Table 3 summarizes the optimal parameters for the different SS³VM models on the above four datasets. The training accuracy, testing accuracy, and training time are also shown in Table 3. As it can be observed, comparing with other models, the 4SS³VM model achieves the preferable training accuracy and testing accuracy on four datasets with less CPUtime.

Figure 3

Evolution of the parameters selection. (a), (b), (c), and (d) show the parameters selection procedures of GSS³VM, PSS³VM, 3SS³VM and 4SS³VM respectively (NB: Heart Dataset, labeled proportion 20%)

Table 3

GA/SVM performance on four datasets

Data sets	Models	Optimal parameters	Training accuracy/%	Testing Testing/%	CPUtime /s
Heart	GSS3VM	c = c* = 28	77.33	74.44	0.1814
	FSS3VM	c = c* = 7n = 4	77.33	73.33	2.0603
	3SS3VM	c = c* = 1 k = 82	77.77	73.33	0.1193
	4SS3VM	c = c* = 7 k = 2113	78.88	74.44	0.1148
QSAR	GSS3VM	c = c* = 31	71.55	72.15	0.4419
	FSS3VM	c = c* = 31 n = 8	72.54	74.14	2.9183
	3SS3VM	c = c* = 30 k = 1564	71.69	67.61	0.5777
	4SS3VM	c = c* = 28 k = 2093	71.97	74.44	0.3584
Wine	GSS3VM	c = c* = 25	65.97	67.60	2.4921
	FSS3VM	c = c* = 5n = 3	66.83	67.60	5.3202
	3SS3VM	c = c* = 15 k = 1109	65.97	67.60	5.3202
	4SS3VM	c = c* = 14 k = 1294	65.97	67.60	2.1299
Wilt	GSS3VM	c = c* = 18	98.54	97.78	2.6598
	FSS3VM	c = c* = 18 n = 9	98.54	97.78	18.9791
	3SS3VM	c = c* = 1 k = 100	98.54	97.78	3.6831
	4SS3VM	c = c* = 28 k = 640	98.54	97.78	3.5875

In order to compare the classification accuracy between the optimal SS³VM models and the non-optimized SS³VM models. For each model, 20 groups of parameters are selected randomly and Heart dataset is used to train and test these models. Let J be the index of parameters group. Figure 4 shows the comparison result. It can be easily found that the performance of the optimal SS³VM models outperforms that of the models with non-optimized parameters on training set. Though the testing accuracy of the optimal models is not the best, it is good enough. That means the GA/SS³VM method contributes to the improvement of the classification accuracy, which greatly supports our claims. The performance of the optimal SS³VM models on other datasets has the similar results.

Figure 4

Accuracy comparison between the optimal models and others with arbitrary parameters. (a), (b), (c), and (d) show the compare results of GSS³VM, PSS³VM, 3SS³VM and 4SS³VM respectively (NB: Heart Dataset, labeled proportion 20%)

Table 4 displays the classification accuracy and CPU time of the optimal 4SS³VM under the different labeled proportion. It can be seen that the training and testing accuracy have a little volatility about less than 5%. That means the classification accuracy cannot be remarkably improved when the number of labeled samples increases. It implies that the optimal 4SS³VM model has higher computational efficiency, i.e., a small amount labeled samples can be utilized to guarantee the classification accuracy. Thus, the cost of manual labeling can be cut down greatly.

Table 4

Effect of percentage of labeled samples to the 4SS³VM

Data sets	Labeled Proportion	Training accuracy	Testing accuracy	CPUtime /s
Heart	20	78.88	74.44	0.1148
	30	78.88	74.44	0.1862
	40	81.66	72.22	0.2157
	50	82.77	74.44	0.3143
QSAR	20	71.69	72.44	0.3584
	30	72.26	73.29	0.5167
	40	74.39	71.59	0.7695
	50	70.98	73.01	0.7692
Wine	20	65.97	67.60	2.4299
	30	66.27	66.99	3.7138
	40	66.00	67.00	5.3567
	50	66.03	67.66	14.7719
Wilt	20	98.54	97.78	3.5875
	30	98.54	97.78	7.2430
	40	98.54	97.78	7.5340
	50	98.30	98.27	15.4175

7 Conclusions

In this paper, a biquadratic spline function for smoothing the S³VM is proposed. According to the analysis about approximation accuracy, the biquadratic spline function has a preferable performance. Further, a new approach named GA/SS³VM method which integrating SS³VM and GA is presented. Here, GA is used to optimize parameters of SS³VM models. By the hybrid GA/SS³VM method, the optimal SS³VM model is obtained. The optimal SS³VM model is experimentally evaluated on four real datasets. The results show that the SS³VM models optimized by GA/SS³VM approach achieve higher classification accuracy than other non-optimized SS³VM models. In particular, the optimal SS³VM model based on biquadratic spline function has the desirable classification accuracy and the best computational efficiency. Meanwhile, the classification accuracy of new model is insensitive to the labeled proportion which means that good classification accuracy can be achieved with a small amount labeled samples.

For future work, we intend to apply the kernel function for SS³VM and optimize the kernel function, parameters simultaneously.