An Optimized K-Harmonic Means Algorithm Combined with Modified Particle Swarm Optimization and Cuckoo Search Algorithm

1 Introduction

Data clustering is a popular data mining technique that is applied for extracting the reasonable organization of objects in a given data set. This technique classifies similar objects into different groups, or more precisely, the partitioning of a data set into subsets, so that each part (subset) has some similarities and common characters. In fact, a set of patterns are gathered into clusters based on the similarity among each cluster. Clustering is an important technique applied in many application domains, including document clustering [18], fraud detection [17], flow shop scheduling [29], machine learning [3], wireless mobile sensor networks [31], biomedical data [12], image processing [49], demand forecast [26], and financial classifications [34]. Many data clustering algorithms have been presented in the previous literatures with different approaches.

Clustering algorithms can be generally divided into two groups: hierarchical algorithms and partitional algorithms. Hierarchical algorithm finds nested clusters either in agglomerative or in divisive [19], and partitional algorithm divides the data sets into some clusters whose members have nothing in common with each other [16, 24, 43]. The most popular and extensively used algorithm among partitioning algorithms is the k-means (KM) algorithm. It easily clusters a large data set with the best runtime. However, the results of the KM algorithm are very sensitive to positions of the initial cluster centers in the problem space [50]. It also has a local optimum problem [20] and does not have any criterion for computing the number of clusters. K-harmonic means (KHM) is an alternative algorithm to solve the sensitivity to initialization problem of KM methods [51]. This algorithm minimizes the harmonic average from all points in N to all centers in K. This approach proposes more robust results than KM with different initial configurations. KHM solves the problem of initialization using a built-in boosting function [48]. However, it easily runs into local optima like the KM algorithm. To overcome the shortcomings of the KM and KHM algorithms, some heuristic algorithms have been combined with these methods. Recently, evolutionary and meta-heuristics like genetic algorithm (GA) [13], ant colony optimization (ACO) [39, 40], artificial bee colony [23, 46], particle swarm optimization (PSO) [7], bacterial foraging optimization [37], cuckoo search (CS) optimization [45], and other optimization algorithms have been hybridized with standard and basic clustering algorithms, including KM, fuzzy KM, and KHM to reach the required quality and performance in clustering processes. These algorithms try to solve the weaknesses of the KM and KHM algorithms. However, they also have several limitations. For example, Tabu search and simulated annealing algorithms suffer from low-quality results and low convergence speed problems [14].

Xin-She and Deb proposed the CS algorithm via Levy flight in 2009 [45]. CS via Levy flights is based on the interesting breeding behavior such as brood parasitism of certain species of cuckoos. The basic ideas applied are the aggressive reproduction strategy of cuckoo and usage of Levy flights. The CS algorithm is being widely used in engineering optimization problems [17] with exceptionally good results. In addition, PSO and ACO have convergence problems. PSO is a versatile population-based stochastic optimization technique. The algorithm maintains a population of particles where each particle represents a potential solution to an optimization problem. In the regular PSO [25], the diversity loss is mainly due to the strong desirability of the global best particle, which results in that all the particles quickly converge on a local or global optimum where the global best particle locates [42].

To solve PSO weaknesses, we propose a modified version of PSO with better convergence that is combined with KHM, called KHM–MPSO, to meet the common factors. Generally, in most clustering algorithms, the main goals are to meet the required quality in clusters, such as processing time, stdev parameters, F-measure, and hError and kError [28]. In this paper, the following performance metrics are used in the comparative analysis: (i) the accuracy of final clustering results and (ii) the speed of convergence. The test suit chosen for this paper consists of eight real data sets and two artificial data sets (see Table 1). On the basis of the experimental results, it is found that the proposed KHM–MPSO performs cluster analysis with better quality and performance in comparison to PSO, KHM, and PSOKHM algorithms.

The rest of this paper is organized as follows. Section 2 briefly presents the current related works in clustering analysis and PSO. The CS via Levy flight, regular PSO, and KHM algorithm is presented in Section 3. In Section 4, the proposed KHM–MPSO clustering algorithm is explained. Section 5 shows the described experiment setting and results. Finally, the conclusion is presented in Section 6.

2 Related Works

The PSO has been used for clustering in many studies. An efficient hybrid clustering based on fuzzy PSO, ACO, and KM algorithms, called FAPSO–ACO–K, is presented in Ref. [36]. The results obtained from this technique are very notable in performance improvement of information clustering. The PSOKHM data clustering algorithm proposed a hybrid algorithm based on KHM and PSO [48]. This algorithm solves the KHM’s local optima problem and PSO’s slow convergence speed. The MOIMPSO clustering algorithm is a hybrid of multiobjective clustering algorithm and PSO that was presented to obtain a single best solution from the Pareto optimal archive [35]. By combining two genetic and PSO algorithms, Kao and Zahara invented a new method in which it has benefitted from jump and junction operator for genetic [21]. This approach could solve different problems of continual functions. In addition, significant changes have been obtained in finding the response to general optimization and convergence ratio. They also combined the KM algorithm, Nelder–Mead simplex search, and PSO, called K–NM–PSO [22]. The K–NM–PSO searches for cluster centers of an arbitrary data set as does the KM algorithm, but it can effectively find the global optima. They used the KM algorithm alone to generate one particle in the initial population. It implemented a Nelder–Mead search only on the best m+ 1 particles in each iteration, where m is the number of attributes, and then the rest of the population is moved toward the best particle of the whole population and toward the best neighbor. Van Der Merwe and Engelbrecht used PSO algorithm to solve the KM clustering problem. The algorithm is extended to use KM clustering to seed the initial swarm [44].

FC–MOPSO is another research that combined the multiobjective particle swarm (MOPSO) approach with the fuzzy clustering (FC) technique [4]. In FC–MOPSO, the migration concept is used to exchange information between different subswarms and to ensure their diversity. A new approach based on PSO and radial basis function neural networks, PSO–OSD, has been developed in Ref. [11]. PSO–OSD used the PSO algorithm, which is not sensitive to the initial values of the cluster centers. Chuang et al. combined chaotic map PSO (CPSO) with an accelerated convergence rate strategy, and introduced this accelerated CPSO (ACPSO) in their research [8]. ACPSO searches through arbitrary data sets for appropriate cluster centers. Yang et al. introduced a hybrid method (called PSOKHM) based on combining PSO and KHM to enhance the global search ability of their algorithm [48]. PSOKHM repeats KHM four times in each generation for which it employs eight generations to improve particles within the population. Furthermore, the PSO algorithm repeats eight times in each generation. A new approach for clustering based on a particle swarm optimizer for dynamic optimization problems, CPSO, is presented in Ref. [6]. CPSO employs a hierarchical clustering method to track multiple peaks based on a nearest neighbor search strategy. Kiranyaz et al. proposed a PSO algorithm and fractional global best formation technique for multidimensional search in dynamic environment [27]. The GAI–PSO method is the combination of PSO, GA, and KM algorithm to find global optimum and fast convergence [1]. The GAI–PSO algorithm searches the solution space to find the optimal initial cluster centroids for the next phase. The next phase is a local refining stage utilizing the KM algorithm that can efficiently converge to the optimal solution. The GSOKHM algorithm is another method that has been presented to improve the efficiency of KHM using the PSO algorithm and GA [10]. Xin-She and Deb [45] has applied the CS algorithm for clustering. They have evaluated CS with GA and PSO using standard benchmark functions. In their study, the CS algorithm is used with Levy flight and is found to be performing better compared to the other two methods. ICAKHM is a novel method on the basis of a hybrid KHM algorithm and a modified version of the imperialist competitive algorithm (ICA) [2]. This version of the ICA method uses the genetic operators of crossover and mutation to prevent the premature convergence, helping KHM to evade the local optima problem similar to many other evolutionary algorithms. The evaluation result of the ICAKHM method [33] reveals that its results are often suitable. However, this algorithm usually is unstable, and its result may or may not be improved. We have compared our proposed algorithm with the ICAKHM method in Section 5. In addition, Ref. [33] presents a survey of the relevant literature in this field.

3 The Regular Cuckoo, PSO, and KHM Clustering Algorithms

Data clustering is aimed at finding out a reasonable organization for the objects of a given data set by identifying and quantifying similarities or dissimilarities among the objects [32]. In fact, clustering includes some qualities based on which a data set can be divided into parts (cluster) so that the components of each part have the most similarity with each other and the least similarity with members of the other parts. The goal of data clustering is to minimize the objective function, in this case a squared error function [41].

The cluster centers are represented by Eq. (2).

where k= 1, 2, …, K is the number of clusters, f(k) is the objective function, x_i, i= 1, 2, …, n_k are the patterns in the kth cluster, and C_k is center of the kth cluster.

Before the explanation of our proposed hybrid method (KHM–MPSO) for clustering, Lévy flight, regular PSO, and KHM algorithms are briefly discussed for immediate reference.

3.1 CS via Lévy Flight

The CS algorithm is a novel meta-heuristic technique [45]. The algorithm mimics the breeding behavior of cuckoos (to lay their eggs in the nests of other birds). CS is based on three idealized rules: (i) each cuckoo lays a single egg into a randomly chosen host nest from among n nests; (ii) the nests with better-quality eggs will join the next generation; (iii) the number of available hosts’ nests is fixed, and the host bird discovers the egg laid with a probability p_a ∈ [0,1].

On the basis of these three rules, the basic steps of CS can be summarized as the pseudo-code shown in Figure 1. When generating new solutions xi(t+1) from the old one (xi(t)), Lévy flight is performed for a cuckoo ith with the parameter 1 < λ < 3 as follows:

Figure 1:

Pseudo-code of Levy Flight Cuckoo Search Algorithm.

(3) xi(t + 1) = xi(t) + α⊕Lévy(λ),

(4) Lévy ~ u = t−λ  1 < λ < 3,

where α > 0 is the step size that should be related to the scales of the problem of interests. In most cases, we can use α= 1. The product ⊕ means entry-wise multiplications. This entry-wise product is similar to those used in PSO, but here the random walk via Lévy flight is more efficient in exploring the search space as its step length is much longer in the long run. The Lévy flight essentially provides a random walk while the random step length is drawn from a Lévy distribution [Eq. (4)].

4 The Proposed Clustering Algorithm

In this section, we describe an improved clustering algorithm based on a modified version of the PSO (MPSO) algorithm and KHM, called KHM–MPSO. We have combined KHM and MPSO to form a hybrid clustering algorithm that maintains the qualities of MPSO and KHM and solves their convergence and sensitivity problems. The MPSO provides a partition of data points without any prior knowledge. Meanwhile, the KHM algorithm can obtain high-quality initializations from the MPSO, and provide better input to MPSO to accelerate its convergence.

In the MPSO algorithm, we use a one-dimensional array to encode cluster centers as particles. Every particle or candidate solution in the population consists of a one-dimensional array with length of d× k cells to show all cluster centers. KHM–MPSO tries to find an optimal partition of k optimal number of compactness and well-separated clusters. The proposed algorithm is built based on two main steps where at each step, only one type of move is done by particles. The first step is to escape from local optimums and migrate away from unsuitable places in the search space. The second stage is to converge to the global optimum. These two steps are repeated alternately until the termination criteria are satisfied (e.g. maximum number of iteration achieved or no change occur in certain number of iterations). KHM–MPSO applies KHM to the particles in the swarm every 10 generations such that the fitness value of each particle is improved. In the proposed algorithm, CS via Levy flight has been used, which is efficient to find new suitable neighbors and better solutions [47]. Sometimes, the best particle of PSO is selected by Levy flight CS instead of the PSO algorithm if the objective function of the generated PSO solution is weaker than that generated by the Levy flight Cuckoo solution. Besides, in comparison to basic or regular PSO, there are two new concepts in the modified version: (i) gworst: the worst point in the current population or “global worst” and (ii) pworst: the worst point in the memory of each particle or “personal worst.” The global worst is the fitness value of that candidate solution that has the worst value for objective function (maximum value in minimization problems). This value is found by all particles in the swarm. The second concept is the worst place that every particle of the population has seen during their move. This concept has been used differently in our previous research work with different impact and objective functions [15]. In the MPSO, the positions of particles depend on their own current worst solution and their group’s previous worst. In our proposed algorithm, the gbest with gworst and pbest with pworst can dynamically be used instead of each other. For each particle, the worst fitness values, pworst, and for the whole swarm, gworst, are computed in each iteration. A particle is shown as Figure 2.

Figure 2:

Representation of a Particle.

The best previous position of the particle in ith iteration is calculated as

pworsti(t) = [pworsti, 1t, pworsti, 2t, …, pworsti, mt] m is the number of solutions.

In regular PSO, we had global best and personal best, which were the best values for objective function found by all particles in the swarm and the best values for objective function found by every particle thus far. At each iteration, after finding these specific positions, the particles move in the presence of two distinct steps. The logic of movements considered is first “escaping from bad points and areas” and then convergence to appropriate places. At the first step, which we call acceleration step, particles find the suitable area of search by moving away from unsuitable areas. In fact, this move causes particles to spread in the search space and search for good solutions in a wide area, and in case of entering local optima they can bypass it. At the next step, which we also call the convergence step, all particles try to move toward the global optimum based on their personal memory and best particle. The fitness function of the KHM–MPSO clustering is the objective function of the KHM algorithm. From the mathematical inference of PSO, a larger inertia weight performs more efficient global search and a smaller one means a more effective local search. Thus, Saatchi and Hung [40] used Eq. (11), which decreases inertia weight with increasing the number of iterations linearly:

This equation has also been used in MPSO. The main steps of the proposed MPSO algorithm are summarized as the pseudo-code in Figure 3.

Figure 3:

Pseudo-code of the Modified PSO Algorithm.

In this algorithm, x_i(t) and v_i(t) respectively show the position and the velocity of particle i at time or iteration t. ωv_i(t + 1) and Sv_i(t + 1) respectively calculate the velocity of particle i based on pworst and pbest solutions. Pbest_i(t) is the best position found by particle i that keeps the fitness value of the best candidate solution encountered by the considered particle thus far. Gbest_i(t) is the best position found by the whole swarm thus far, and ω is an inertia weight scaling the previous time step velocity. Pworst_i(t) is the worst position found by particle i that keeps the fitness value of the worst candidate solution. The c₁ and c₂ coefficients are two constant coefficients [0, 2] that control the influence of the best personal position of the particle (pbest_i(t)) and the best global position (gbest_i(t)), where c1+ c2 ≤ 4. rand₁ and rand₂ are random values in the range [0, 1]. K is a constriction factor for updating the particle’s flying velocity. Through the constriction factor, the algorithm can have better convergence and stability. ω_max and ω_min are the maximum and minimum of the inertia weights, respectively, and t is the iteration counter.

Our proposed hybrid algorithm (KHM–MPSO) maintains the merits of KHM and PSO and cuckoo algorithms. The pseudo-code of the proposed KHM–MPSO algorithm is represented in Figure 4.

Figure 4:

Pseudo-code of the Proposed KHM–MPSO Algorithm.

5 Experimental Design

We test our proposed algorithm on seven data sets and compare it with other well-known algorithms. These data sets are eight real data sets and two artificial data sets that are named as Iris, Wine, Wisconsin breast cancer (denoted as Cancer), contraceptive method choice (denoted as CMC), and Ripley’s glass with different number of clusters, data objects, and features for every data object [5]. These data sets cover low, medium, and high dimensions. A brief description of these data sets is explained below.

6 Experimental Results

To compare the performance of our algorithm with those of other approaches, each algorithm is 100 times for each of the data sets and averaged at the end. The simulation results are demonstrated in Table 3.

The simulation results given in Table 3 show that KHM–MPSO and PSOKHM are very precise, and on average, KHM–MPSO is more precise than PSOKHM. Furthermore, in all other data sets, our algorithm has a small stdev compared to the other algorithms except the cancer data set (for PSOKHM). For instance, the results obtained on the Iris data set show that KHM–MPSO converges to the global optimum of 96.6228 in most of the runs, whereas the best solutions of KHM, PSO, ICAKHM, and PSOKHM are 97.8396, 98.7741, 96.6362, and 96.6301, respectively. Additionally, the obtained best, average, and worst solutions of the KHM, PSO, PSOKHM, and KHM–MPSO algorithms indicate that KHM–MPSO is the best one for all data sets, except the cancer data set. Nevertheless, the obtained results for best and average solutions by the PSOKHM algorithm are good and close to KHM–MPSO’s results, whereas the worst solution of KHM–MPSO is of higher quality than that of other algorithms. In short, KHM–MPSO has minimum values of the KHM function in Iris, Wine, CMC, and Glass data sets.

On the other hand, the simulation results of Table 3 shows that the F-measure of the proposed algorithm absolutely is better than those of obtained by others in all data sets. It reveals that the clusters are spatially well separated by the KHM–MPSO algorithm.

The stdev of the proposed algorithm is less than that of the other algorithms. It means that KHM–MPSO can find optimal solutions in most of the cases, while other algorithms may be trapped in local optima. Moreover, it often can find high-quality solutions compared to the other algorithms. The best stdev in the Iris data set (with low dimension) belongs to ICAKHM and our proposed KHM–MPSO algorithm. The stdev of the fitness function for these algorithms is 0.01055 in the Iris data set, which is significantly less than that of the other methods. However, ICAKHM does not have a better stdev in all other data sets. For the Cancer data set (with high dimension), the PSOKHM and KHM–MPSO algorithms have better stdev than the other algorithms. Furthermore, the KHM–MPSO algorithm has better stdev than the other algorithms in Wine, CMC, and Glass data sets.

In general, the simulation results shown in Table 3 indicate that the proposed KHM–MPSO algorithm converges to the global optimum with an improved stdev and less function evaluations. This leads logically to the end that KHM–MPSO is a feasible and a robust clustering algorithm.

Owing to the close similarity between PSOKHM and the proposed algorithm, we compare them considering more details that are presented in Tables 4 and 5. Tables 4 and 5 are the results of the objective function KHM(X,C), F-measure, and runtime criteria, which are in accordance with different p values, p= 2.5, p= 3, and p= 3. The tables show the means and stdev (in brackets) for 100 independent runs. Boldface indicates the best result out of the two algorithms.

Owing to the reduction of the value of the KHM(X,C) function and the increase of the F-measure, the KHM–MPSO algorithm generates better clustering quality than the PSOKHM algorithm. In other words, KHM–MPSO improves the F-measure, runtime, and KHM(X,C) measures in most of the runs with different p values. The proposed KHM–MPSO has best runtimes in most of evaluations with different p values. The evaluations shown in Tables 4 and 5 clearly show that the KHM–MPSO algorithm has a small runtime in comparison with the PSOKHM algorithm in all data sets, except the Cancer data set. Consequently, the accuracy, correctness, and convergence of our proposed algorithm are more satisfactory and robust than the PSOKHM and other compared algorithms.

Finally, an execution of KHM–MPSO and other mentioned algorithms on Artset2 data set is shown in Figure 5.

Figure 5:

An Execution of KHM, PSO, PSOKHM, and KHM–MPSO Clustering Algorithms on Artset2 Data Set.

The analysis of this figure also proves the improved quality of clustering by the proposed algorithm. It can be seen that KHM–MPSO can cluster objects more clearly with the best F-measure and KHM(X,C) values. The main drawback for KHM–MPSO, ICAKHM, and PSOKHM is their running time in comparison to the KHM algorithm. The KHM algorithm has the best running time. However, KHM–MPSO has a better running time than ICAKHM and PSOKHM.

In the end, to indicate a significant difference between the results of the proposed KHM–MPSO algorithm with other algorithms, statistical analysis was carried out. We applied the Friedman test to realize whether there are substantial differences in the results of the clustering algorithms. In this test, the α was set to 0.05 (α= 0.05) as the level of confidence in all cases. Table 6 reveals the obtained results of mean ranking of these algorithm by Friedman’s test based on best and average KHM() function as well as F-measure. Table 7 shows the statistical test in the Friedman test. As shown in the table, the proposed KHM–MPSO algorithm is ranked first, followed by PSOKHM, ICAKHM, PSO, and KM, successively. Furthermore, the Friedman test indicates that the proposed KHM–MPSO algorithm has a significant difference in the results of algorithms.

7 Conclusion

This paper proposed the KHM–MPSO algorithm, a hybrid clustering algorithm, by combining KHM and MPSO using CS via Levy flight algorithm. The MPSO used cuckoo optimization and two new concepts, pworts and gworts, in regular PSO algorithm. Therefore, the combination of the MPSO algorithm with KHM utilized the advantages of KHM and solved the KHM’s shortcomings. It overcame the initialization sensitivity of KHM and achieved the global optima effectively. The new proposed algorithm was tested on several real and artificial data sets. The experiments confirmed that the proposed algorithm was accurate and robust compared to the PSO, KHM, PSOKHM, and ICAKHM algorithms. The proposed KHM–MPSO algorithm not only improved the F-measure and stdev parameters, but it also helped KHM escape from local optima. In the KHM–MPSO algorithm, because of obtaining high-quality initializations from the MPSO, the KHM algorithm provided better output and performance. Our proposed algorithm clustered large data sets faster and more accurately than other algorithms. Yet, it should be mentioned that one drawback of KHM–MPSO is its runtime compared to KHM. KHM has a better running time than other algorithms. As a future work, we investigate on combining PSO and artificial bee colony into KHM to reach a faster convergence, accuracy, and runtime.