A Method of Biomedical Information Classification based on Particle Swarm Optimization with Inertia Weight and Mutation

2.1 Support Vector Machine

SVM is a machine learning method proposed in the 1990s based on statistical learning theory [52]. It is based on the principle of structural risk minimization. The kernel function method of SVM is used to transform the inseparable problem in low dimensional space into linearly separable problem in high dimensional space.

For a given sample set of SVM (xi, yi), i=1, 2,…, n, yi∈{+1,-1}, where xi denotes the input data, yi denotes the output labels, n denotes the number of samples, and the hyperplane can be expressed as:

where w denotes the weight vector, and b is the threshold. The optimization problem of SVM is described as:

where C is the penalty parameter, which is used to control the degree of punishment for the error sample.

The above problem can be transformed into Lagrange dual form, and the final decision function is:

f (x) is the decision function. When different sample data is fed into the function, the decision function will predict the corresponding label.

2.2 Particle Swarm Optimization Algorithm

PSO is an intelligent evolutionary algorithm proposed by Kennedy and Eberhart based on the foraging process of birds [21]. In the PSO algorithm, the process of optimizing the problem is regarded as the process of bird feeding. Each particle in the PSO algorithm is abstracted as a solution to the optimization problem, which is described by two parameters: the position and the velocity of the particle. In each iteration, the velocity and position of the particle are updated by the formulas:

where, i denotes the ith particle; d denotes the dimension; t is the number of iterations; xid denotes the position of the ith particle in the d dimension, and xid is limited in the interval [popmin, popmax]; vid(t) denotes the velocity of the particle; w is the inertial weight, which controls the influence of contemporary velocity on the next generation’s; pbestid(t) denotes the individual best position, and gbestd(t) denotes the global best position, c₁ and c₂ are acceleration factors; r₁ and r₂ are the random numbers in interval [0, 1].

Set z=c₁r₁[pbestid(t)-xid(t)]+c₂r₂[ gbestd(t)-xid(t)]. Figure 1 shows the above two formulas are represented with vectors in two dimensions. Figure 1(a) denotes the velocity formula, and the velocity v(t+1) is generated by wv(t) and z. Figure 1(b) denotes the position formula; the position of the new particle x(t+1) is added by the position x(t) and the velocity v(t+1).

Figure 1

Updating of a PSO algorithm with vector representation in two dimension. (a) velocity update schematic. (b) position update schematic

3.1 Mutation Strategy of PSO

The mutation strategy of PSO is mainly to change the position of the particle. Figure 2 shows the effect of mutation strategy on PSO algorithm. The solid line denotes the original position and the dotted line denotes the new position after the mutation. Since the mutation strategy does not directly affect the velocity of the particle, the vector v(t+1) in Figure 2(a) does not change. The position of the particle is changed from x(t) to x(t)’ after mutation in Figure 2(b), and the position of the next generation of particle is changed from x(t+1) to x(t+1)’ by adding with v(t+1). If we adjust the particle’s moving distance reasonably, we can control particle movement in a certain range, which is beneficial for the particle to jump out of the local optimal solution.

Figure 2

The effect of mutation strategy on the PSO algorithm. (a) speed update schematic; (b) position updating schematic after introducing the mutation strategy

Alireza proposed an adaptive mutation strategy, and the distance of particle moving can be automatically adjusted according to fitness values [53]. The calculation method of variation is as follows:

where xi’ is the position of the ith particle after mutation. βij represents a random number that obeys a Gaussian distribution with a mean of 0 and a standard deviation of 1. Parameter Mt can control the distance of mutation, which is calculated according to the fitness of global best position. popmax is the maximum value of the particles search space, and tanh is a monotonic increasing function. F (gbestd(t)) represents the best fitness of the population. Because the value of tanh is in the interval [0, 1], the value of Mt can be controlled adaptively within a certain range.

Stacey and Jancic proposed a Gaussian mutation algorithm (GPSO) [54]. The basic idea is changing the position of a particle when it is in the local optimal solution, and the mutation formula of particle xi is:

where xi is the position of the current particle and xi’ is the position after mutation. Gaussian(σ) is a random number that obeys a Gaussian distribution with a mean of 0 and a standard deviation of σ. The value of σ is 0.1 times the length of search space. The ability of the particle to jump out of the local optimal solution is increased by mutation. When using Gaussian PSO, a Gaussian distribution operator will be added to the position of a particle. According to the characteristics of Gaussian distributions, the value of the operator is smaller in most cases, resulting in the particle moving in a small range. Therefore, the Gaussian mutation has a strong local search ability and poor global search ability. In addition, there are many variants of the Gaussian strategy, such as Zhan and Lu’s neighbor heuristic and Gaussian cloud learning particle swarm optimization algorithm [55].

Wang and Li proposed a Cauchy mutation strategy (CPSO) [56]. The mutation formula of the particle is:

where W is the mean vector of all particle velocities. n is the size of the population. N is a random number that obeys the Cauchy distribution. Because the probability density function image of a Cauchy distribution is a relatively smooth strip, the two ends are larger and the middle is smaller, which makes the position of the particles change greatly, so the Cauchy mutation has stronger jumping ability. In addition, there are many variants based on Cauchy mutation, such as adding a scaling factor on the Cauchy mutation to control the distance the particle moves [57].

Brockmann and Sokolov found the Levy flight pattern [58], which indicated that most of the cases are changed in a small range, and occasionally a small part of the situation would move to a distant position. Hakl and Uguz applied Levy mutation in particle swarm optimization (LFPSO) [59], and the mutation formula is:

where α denotes the step size of the particle moves, and Levy(β) denotes the distribution of the Levy with parameter β. β is a variable between [0, 2]. Due to the occasional long distance movement of Levy flight, the particle’s position may move in a wide range occasionally during mutation, causing the particle to jump out of the local optimal solution.

Wang and Wu combined the Gaussian, Cauchy and Levy mutation strategies by an adaptive approach [60]. Nishio and Kushida proposed an adaptive PSO with multidimensional mutation strategy [61].

Li and Liu proposed a stochastic mutation method (RPSO) [62]. The particle mutated by the following formul'a:

where popmax and popmin denote the maximum and the minimum of the particle search range respectively. r is a random number obeying a uniform distribution in the interval (0, 1).

Zhang and Lu proposed a feedback mutation particle swarm optimization algorithm (FBPSO) [63]. The mutation formula is:

where β obeys a Gaussian distribution with the mean of 0 and the standard deviation of σ. fit(i) is the fitness of the ith particle, and fitgbest is the global fitness, fitavg is the mean fitness of all the particles. Since σ changes according to fitness, this mutation strategy can adjust the distribution of mutation positions based on the information of the particle.

3.2 Inertia Weight Strategy of PSO

Kennedy and Eberhart introduced the inertial weight parameter in the original PSO algorithm, and proposed the particle swarm algorithm with inertial weight [21].

Figure 3 shows the effect of inertia weight in a PSO algorithm. In Figure 3(a), the vector changes from wv(t) to wv(t)’ after changing the inertia weight w, and then the velocity changes from v(t+1) to v(t+1)’. In Figure 3(b), the new position of the particle is changed from x(t+1) to x(t+1)’ by adding v(t+1)’ to the position x(t). Therefore, the inertia weight in PSO algorithm essentially changes the position of particles. When using a larger inertia weight, the position of particle will also change greatly, which facilitates the global exploration of the algorithm in a large surrounding space. When the inertia weight is smaller, the position changes less, which facilitates the local search of the algorithm in the additional small space.

Figure 3

The effect of the inertia weight on PSO algorithm. (a) the speed update schematic after changing the inertia weight. (b) the position update schematic after changing the inertia weight

PSO algorithm is nonlinear and highly complex in the search process. In order to improve the ability to monitor the population, Alireza proposed an inertia weight strategy with fitness feedback [53]. The formula is:

where tanh and F (gbestd(t)) are the same as the definitions in formula (7), which are abbreviation for hyperbolic tangent and the fitness of the current best solution. In this strategy, the value of inertia weight is limited to the interval [0.5, 1]. When the fitness does not decrease, w(t) is large and changes slowly, which benefits to global exploration. When the fitness reduces, w(t) is small, which benefits to local exploitation.

Eberhart and Shi proposed a random inertial weight (RANDPSO) [47]. The inertia weight is calculated as:

where rand is the random number between [0, 1]. w is a random number in the interval [0.5, 1]. Since the value of inertia weight is random, the result of the algorithm is contingent.

Yang and Gao proposed an exponential inertial weight strategy (LHNPSO) [64]. The inertial weight formula is

where wmax and wmin are the maximum and minimum of inertia weight respectively. α=1/π². The inertia weight decreases with the number of iterations, which benefits to global exploration and local exploitation of the algorithm.

Nickabadi and Ebadzadeh proposed an adaptive inertia weight, in which the inertia weight is adjusted by the ratio of successful particles (AIWPSO) [65]. The inertia weight formula is:

where, n is the size of the population. fit(pbesti(t)) denotes individual fitness. Ps(t) is the search success rate. If the individual fitness of a particle is smaller than the individual fitness of the last generation, the particle searches successfully, and Si(t)=1; otherwise, the particle does not search successfully, Si(t)=0. The inertia weight is beneficial to supervise the information of particles in the population.

Chauhan and Deep used global best position and individual best position to adjust the inertia weight (DESIWPSO) [66]. The inertia weight of the ith particle is calculated as:

Taherkhani and Safabakhsh proposed a multi–dimensional changing inertia weight strategy (SAIWPSO) [67]. In this method, the inertia weight of the next generation is automatically adjusted by the position information of the contemporary particle. What’s more, the concept of ith particle searches success is introduced:

where fit(xi(t+1)) is the fitness of the ith particle in t+1 generation. fit(pbesti(t)) is individual fitness. If fit(xi(t+1))< fit(pbesti(t)), the particle searches successfully, and δi(t)=1, otherwise, the particle does not search successfully, δi(t)=-1. The inertia weight formula is:

where wij(t) is the inertia weight of the ith particle in the jth dimension. w₀ is the initial inertia weight. σ is the standard deviation of all particles in the jth dimension. ε is a smaller positive number, ε=0.005.

All of the above inertia weight strategies do not make full use of the information of the particle population, and cannot supervise the status of the population very well. Therefore, we proposed a multi-information fusion inertia weight (MDAPSO) [68]. In order to supervise the state of the population, we refer to other adaptive inertia weights, and introduce the velocity and position of a particle in the inertia weight, as shown in formulas Y and Z. In order to control the overall trend of inertia weight, we refer to the decreasing inertia weight with iterations, and introduce λ₁ and λ₂, and in order to avoid the particles falling into local optimal solutions, we introduce a random disturbance. In order to increase the diversity of inertia weight, the inertia weight w_id is changed at each iteration, each particle and each dimension. Finally, the formula of inertia weight is as follows:

In the formula, min and max are used to limit the inertia weight wij(t) within the interval [0.1,1]. α=0.9, β=0.55, γ=0.5. case1, case2 and case3 are three conditions; δi(t-1) is used to determine whether the particle searches successfully. case1 indicates that the particle has searched successfully for two generations, and case2 indicates that the particle has not searched successfully for two generations. case3 denotes the above two cases are not satisfied.

In the proposed inertia weight formula (27), we refer to α, β and γ as weight coefficients and limit them in the interval [0, 1]. Weight coefficients denote the influence ratios of Y, Z and rand on the inertia weight. To obtain the weight coefficients more easily and quickly, we use two coefficients to fix one and subsequently test the other one with the step length of 0.1, and the principle is shown in Figure 4. The method fixes β and γ first. Because the range of each weight coefficient lies in the interval [0, 1], the midpoint of the interval is chosen as the fixed value. We fix β=0.5 and γ=0.5 at first and calculate the fitness of each benchmark function when α is different and finally select the α value when the fitness of each benchmark function is minimum. In the same manner, we fix α and γ to obtain β and fix α and β to obtain γ. Figure 5(a), (b) and (c) show the effect of α on the benchmark functions when β and γ are fixed. The horizontal axis is α, which is in the interval [0,1], and the vertical axis is the error between the fitness and theoretical value. The error is small, and the performance of the algorithm is better. From Fig. 5(a), (b) and (c), we note that when α=0.9, the algorithm has a small error. From Fig. 5(d), (e) and (f), we find that β=0.5. From Fig. 5(g), (h) and (i), we observe γ=0.5. We obtain the optimal coefficient of each solution, but the solutions are optimal under certain conditions, and their combination is not necessarily the optimal solution of the algorithm. Therefore, we need to finely adjust α, β and γ to select the most appropriate solution near the three solutions. Eventually, we find that when α=0.9, β=0.55 and γ=0.5, the algorithm achieves a better result.

Fig. 4

Method for obtaining parameters α, β and γ in this paper

Fig. 5

Effects of different α, β and γ on benchmark functions

The pseudo code for our improved inertia weight PSO algorithm is shown in Algorithm 1.

Algorithm 1

Initialize values: n, D, c1, c2, tmax, popmin, pop max, vmin,

vmax, wid

fori=1 to n

initialize xi=xi1, xi2, xi3,…, xiD, vi=vi1,vi 2,vi 3,…,vi D

end

evaluate the values fitness(xi)

pbesti=xi

set the particle with best fitness to gbest

fort=1 to tmax

fori=1 to n

ford=1 to D

vid(t+1)=widvid(t)+c1r1[pbestid(t)-

xid(t)]+c2r2[gbestd(t)-xid(t)]

end

x(t+1)= x(t)+ v(t+1)

evaluate the values fitness(xi)

if (fitness(xi)< fitness(pbesti))

pbesti=xi

end

if (fitness(xi)< fitness(gbest))

gbest=xi

end

end

fori=1 to n

ford=1 to D

update w_id

end

end

end

The flow chart of the algorithm is shown in the figure 6.

Fig 6

Flow chart of MDAPSO algorithm

4.1 Benchmark Function and Parameter Setting

In order to test the performance of the improved strategies, the different inertial weight strategies and mutation strategies were applied to optimize the benchmark functions that Taherkhani and Safabakhsh [67] used. All the benchmark functions were optimized for the minimum value. Table 3 shows the formula, dimension, search space and global minimum value of each test function. f₁-f₈ are unimodal benchmark functions, mainly to test the optimizing precision f₉-f₁₅ are multimodal benchmark functions, and f₁₆-f₂₀ are rotating multimodal benchmark functions, mainly to test the ability of the algorithm jumping out of local optimal solutions. The parameters of the PSO algorithm have a great influence on the algorithm. In order to ensure the fairness of testing, all the other parameters of the algorithm were the same except the inertia weight strategy and mutation strategy adopted by different literatures: population size n=20, learning factor c₁=1.49445, c₂=1.49445, maximum velocity v_max=0.2(popmax-popmin), minimum velocity v_min=-v_max, dimension D and particle search range [popmin,popmax] are determined by the benchmark function, iteration number t_max=10000D, and each algorithm runs 30 times independently. The final result of each benchmark function is the mean value of 30 results. The experimental platform is an HPZ640 workstation with 32G memory and Windows7 64 bit operating system. The software is MATLAB R2014a and parallel computing is used. We provide an example for parallel computing (https://github.com/zhangming8/APSO) In this example, it is the implementation of the improved PSO proposed by Alireza [53], so some of the parameters are the same as in the original paper. Table 1 and Table 2 are the summaries of the improvement strategies that have been discussed earlier.

4.2 Comparing Mutation Strategies without Inertial Weight

The benchmark functions were optimized by PSO with different mutations when the inertia weight strategies were not used. The result is shown in Table 4. The “BestNumber” of the last line in the table denotes the best number of optimization result of each PSO algorithm. As can be seen from Table 4, when the inertia weight strategy is not used, the maximum value of BestNumber corresponding to m₄ is 5, so the performance of mutation m₄ is the best. From the results, the good results of m₄ mainly concentrated in the multimodal benchmark functions. The main reason is that m₄ introduces the Levy flight strategy, which makes the particle occasionally produce a larger jump range and promotes the algorithm to jump out of the local optimal solution. The performance of m₆ and m₃ are also not bad. The main reason is that m₆ used an adaptive mutation strategy in which the mutation distance can be changed according to the fitness value of the population, so as to better monitor the information of the population and promote the algorithm to jump out of the local optimal solution. m₃ used a uniformly distributed mutation operator, and the overall optimization performance is also good.

4.3 Comparing Inertial Weight Strategies without Mutation

In this section, the benchmark functions were optimized by PSO with different inertia weight when the mutation strategies were not used. The result is shown in Table 5. When the mutation strategy is not used, the maximum value of BestNumber corresponding to w₆ is 8, so the performance of w₆ is the best. It shows good performance not only in unimodal benchmark functions, but also in multimodal benchmark functions. The main reason is that w₆ improves the diversity of the population, enhances the supervising ability of the population, and balances the global exploration and local exploitation ability very well. In addition, the performance of w₅ is also not bad, especially in unimodal benchmark functions. The main reason is that w₅ uses the position of the particle to adjust the inertia weight. Meanwhile, the inertia weight changes in each iteration, each particle and each dimension, which helps to improve the diversity of the population.

4.4 Comparing the best inertia weight strategy and mutation strategy

In the previous two sections, the best mutation strategy and the best inertia weight strategy were obtained. We also compared the best inertia weight and the best mutation

strategy together, and the results are shown in Table 6. It can be seen from Table 6 that the overall performance of inertia weight and mutation strategy are similar. As for the single peak functions f₁, f₂, f₃, the inertia weight w₆ obtained higher precision than the mutation m₄, so a good inertia weight strategy has great influence on the precision of the PSO algorithm for the single peak function. For the multimodal functions, the mutation strategy shows better results, so the mutation improves the ability of the algorithm to jump out of the local optimal solution.

Table 6

Results of the best inertia weight and mutation strategy

	m4	w6
f1	1.43E-02	5.59E-161
f2	1.64E-01	6.04E-164
f3	1.74E-02	1.65E-32
f4	1.42E-05	2.13E-05
f5	8.55E-03	1.27E+00
f6	9.19E-06	1.05E-02
f7	3.69E-05	5.09E-03
f8	3.85E-03	2.80E-03
f9	2.33E+02	4.03E+02
f10	1.89E-05	5.67E-07
f11	3.25E-05	1.25E-04
f12	4.06E-03	2.07E-02
f13	6.73E-02	7.40E-04
f14	2.04E-06	8.13E-01
f15	7.13E-03	8.79E-07
f16	6.83E+01	5.92E+02
f17	1.13E-04	6.95E-05
f18	4.23E-02	4.85E+00
f19	2.41E-03	1.78E-03
f20	6.46E-02	1.70E-03
BestNumber	10	10

4.5 Comparing the Combinations of Inertia Weight Strategies and Mutation Strategies

In the PSO algorithm, the combined inertia weight and mutation strategy can further improve the performance of the algorithm. Figure 7 is the PSO algorithm updating process with the combination of inertia weight and mutation strategy. Figure 7(a) denotes velocity updating, and Figure 7(b) denotes position updating. In Figure 7(a) the vector changes from wv(t) to wv(t)’ after changing the inertia weight w, and then the velocity changes from v(t+1) to v(t+1)’. In Figure 7(b), the position of the particle changes from x(t) to x(t)’ after mutation, therefore, the new position of the particle is changed from x(t+1) to x(t+1)’ by adding v(t+1)’ to the position x(t)’. The position of the particle is changed by the combination of inertia weight and mutation operations.

Fig.7

Updating principle of PSO when inertia weight and mutation are used. (a) velocity updates, (b) position updates

In order to compare the combinatorial performance of inertia weight and mutation strategies, benchmark functions were used to optimize the combinations from w₁ to w₇ and m₁ to m₆. The parameters used were the same as in Section 4.1 In order to clearly reflect the performance of each combination, the “BestNumber” of each combination relative to other combinations was calculated. Figure 8 shows the BestNumber when all the inertia weight strategies and mutation strategies were combined. The transverse axis denotes the different inertia weight strategies and the longitudinal axis is the BestNumber. Each figure denotes 7 combinations. By comparing the 42 combinations, the maximum value of BestNumber is 6, which corresponded to the best combination of w₃+m₅ in Figure 8(e). Although inertia weight w₆ is the best without mutation strategy and mutation m₄ is the best without inertia weight strategy, the combination of them (w₆+m₄) is not the best. The main reason is that the mutation strategy directly changes the position of the particle, whereas the

Figure 8

The BestNumber when all the inertia weight strategies and mutation strategies are combined

inertia weight strategy changes the position indirectly by changing the velocity of particle. The essence of both strategies is to change the position of particle. Therefore, using different inertia weight and mutation strategies will interfere with each other and affect the result of the PSO algorithm. The reason for this phenomenon can also be obtained from Figure 7.

5.1 Dataset and Test Process

In this section, different classification models were evaluated by the datasets in the UCI machine learning library proposed by Bache and Lichman [69]. Informed consent: Informed consent has been obtained from all individuals included in this study. The datasets used in this study included Breast Cancer, Diabetes, Liver-disorders, Parkinsons, Statlog (heart), and Lung-A (lung cancer) [70], and the specific information of each dataset is given in Table 7.

In the test process, SVM based on RBF kernel function was used to classify the above datasets. Meanwhile, the PSO algorithm was used to optimize the penalty factor (C) and kernel parameter (g) in SVM, and the process was as follows:

1. Initialize the particle swarm, set the population size and the number of iterations, and initialize the position and velocity of particles randomly;

2. Train the SVM model, and calculate the classification accuracy based on current C and g;

3. Calculate the inertia weight;

4. Update the velocity and position;

5. Update the individual best position pbest and global best position gbest;

6. Modify the particle position by mutation;

7. Judge the stop condition and return to Step2 if it is not satisfied, otherwise continue;

8. End.

5.2 Results

From Section 4 and Section 5, it is known that m₄ is the best when using mutation strategy, and w₆ is the best when using inertia weight strategy. The combination of w₃ and m₅ is the best when using the combination of inertia weight strategy and mutation strategy. This paper compared the three PSO algorithms based on mutation strategy m₄, inertia weight strategy w₆, and the combination of w₃ and m₅ to optimize penalty factors (C) and RBF kernel parameters (g) in SVM. The other parameters of PSO were set as follows: the population size was 20; the acceleration factors c₁ and c₂ were 1.49445; the maximum number of iteration was 1000; the minimum value of the penalty factor C and the kernel parameter g was 0.01, and the maximum value was 100; random initialization was performed before the algorithm ran. The maximum velocity of the particle was 0.2 times the range of search space, and the minimum velocity was the opposite of the maximum velocity. We used LIBSVM tool proposed by Chang and Lin during the test^[71]. 10-fold cross validation was used during the test and repeated 5 times. The test results were averaged. Figure 9 shows the classification accuracy curves of the three PSO algorithms during iteration. Table 8 shows the final accuracies of all the datasets.

Figure 9

The accuracy curves of the three PSO algorithm during iteration on different datasets

From Figure 9, the accuracy of using w₆ on Parkinsons and Lung-A datasets was the highest, the accuracy of using combination w₃+m₅ on other datasets was the highest, and the classification accuracy of using m₄ was the lowest. During the iterations, the combination w₃+m₅ improved rapidly in the early stage on Breast Cancer and Diabetes datasets, and converged to the highest classification accuracy eventually. For the Liver-disorders dataset, the accuracy of w₃+m₅ and w₆ were almost the same at the beginning, but w₃+m₅ converged to the highest accuracy finally. For the Parkinsons dataset, the final accuracies of m₄ and w₆ were the same, and both of them were higher than w₃+m₅. For the Lung-A dataset, all the algorithms converged to a stable value quickly, and the accuracy of w₆ was obviously higher than m₄ and w₃+m₅. For the Statlog (heart) dataset, w₃+m₅ not only converged faster but also converged to a higher accuracy than m₄ and w₆. In general, both the combination w₃+m₅ and the inertial weight w₆ that we proposed were the best, which means that a good algorithm does not show the best results when solving all problems, and this phenomenon also conformed to the “no free lunch” theorem proposed by Wolpert and Macready [72]. Therefore, in the actual classification, we can use the combination w₃+m₅ first. If the improvement is not obvious, we can use w₆ and choose a better algorithm. If the two strategies have the same effect, we can use the strategy that is easier to implement.