Circular convolution-based feature extraction algorithm for classification of high-dimensional datasets

1 Introduction

2 Literature survey

Heart disease is one of the major killers among other diseases, which has caused 7.4 million deaths worldwide in 2015 [25]. Other diseases like cancer, kidney disease, hepatitis are other significant killers worldwide. In the medical field, disease detection is challenging, as the features under consideration may be irrelevant and redundant, causing a reduction in classification accuracy. This irrelevant and redundant feature elimination is essential to reduce classification efforts, reduce the risk of overfitting, and improve classification accuracy. Reduction in the feature set is inversely proportional to classification accuracy and directly proportional to classification time. In the literature, preprocessing steps before the actual data classification include feature reduction, feature selection, feature extraction, and feature optimization. These are popular techniques used for dimensionality reduction [26].

Vipin Kumar et al. discussed different feature selection approaches, such as filters, wrappers, and embedded methods, and their application in the real world. The features could be highly dependent on one another, or there could be too many features. Hence, different feature reduction techniques are used frequently by researchers in different areas. The author described different application areas of feature selection in the real world, such as remote sensing, text categorization, intrusion detection, and image retrieval. Some of the challenges in feature selection, such as large dimensional data, scalability, and stability, are mentioned in this study [27]. A database of 50 subjects is created for human gait analysis [49] by Vijay et al. The author represented data using 24 attributes per instance, which is preprocessed by using the Kalman filter. Different combinations of deep learning and hybrid deep learning classifiers are used to experiment, which shows a significant increase in accuracy (99.34%). The author also worked on a biped robot. The data regarding different walking styles are collected through inertial measurement unit sensors and walking patterns are analyzed [50].

Some of the nature-inspired algorithms for feature selection, such as the binary bat algorithm, particle swarm optimization, and modified cuckoo search algorithms, are discussed by Shrivastava et al. [28,29]. An integrated filter and wrapper method for a sequential search procedure improves classifier performance to tackle the overfitting problem and chances of getting a local optimal solution [30]. Xie and Wu proposed a feature selection algorithm based on association rules, which discovers the class attribute features as per the association analysis theory. Yet, this algorithm’s time complexity is relatively high due to the Apriori algorithm [31]. Alessio Ferone proposed a novel approach of feature selection based on rough set theory [32]. Jinghua Liua et al. proposed a feature selection method based on the distinguishing ability of features. He used the maximum nearest neighbor concept to discriminate against the nearest neighbor of samples and evaluated the quality of features [33]. Tajanpure and Jena [38] put forth a multistage classifier system approach, where the feature decimation according to their level of processing need is done. The first-level features are processed first, and as per the first classifier’s output, further level processing decision is taken. Dua et al. worked on human activity recognition with a proposed deep neural network-based CNN and gated recurrent unit. The proposed method performs features extraction and classification too [51].

Saul Solorio-Fernandez et al. proposed a new unsupervised spectral feature selection method. In many practical problems, the dataset under study is described through numerical and nonnumerical features, that is, mixed datasets. The spectral feature selection-based proposed method uses a kernel and a new spectrum-based feature evaluation measure for deciding the relevance of features. K-nearest neighbor (KNN), Naïve Bayes, and support vector machine (SVM) classifiers are used to find the proposed algorithm’s performance, and their accuracy is compared [39]. Much research is done on the disease diagnosis system based on different approaches, such as learning vector quantization and artificial neural networks [40] and classification algorithms [41,42]. Many dimensionality reduction algorithms are developed by researchers for different applications to work with some advantages and some limitations like overfitting, higher time complexity, and so on. The common base to evaluate these techniques is classifier accuracy and other parameters like specificity, sensitivity, F-measure, and so on [43].

Wei Song proposed an effective content-based feature selection approach to improve clustering performance for genetic algorithms. The conventional genetic algorithms face the problem of slow learning and local minimum due to a high-dimensional exploration space. The proposed approach is a parametric and a nonparametric algorithm to adjust the genetic algorithm operators properly [44].

In the brute-force feature selection method, all possible combinations of the input features are evaluated to find the best subset. Here the computational cost is high, with the considerable danger of overfitting. An important aspect of feature selection techniques is the evaluation of a candidate feature subset and searching through the feature space. If there exist at least two instances with the same feature values but with different class labels, the feature subset is classified as inconsistent [45].

3 Proposed feature reduction system design

Convolution is a way of converting two sequences into another sequence. Each value in the input sequence is viewed as scaling and shifting of unit impulse or delta function in digital signal processing (δ(n)). The output of convolution is simply expressed as shifted and scaled version of the input impulse. From this, it is very clear that the dimensions that are dominant in the input will result in output with the same dominance so the accuracy of the output is improved despite feature reduction [52]. This research proposes a feature extraction method based on convolution. Convolution is one of the basic operations of analog signals. There are two forms of convolution: linear convolution (LC) and circular convolution (CC). LC gives a linear overlapping result of the two sequences, which is nothing but the system’s output when triggered with input x ₁(n) having transformation function x ₂(n). LC relates an output sequence with a given input sequence and impulse response, as shown in equation (1). LC is computed over all relevant values of n, that is, from −∞ to +∞. In mathematical form, if k is any positive integer, convolution is expressed [35] as

whereas circular convolution gives the output when two sequences are circularly overlapped. Table 1 shows the comparison between LC and CC.

One can get the output of CC same as LC [34] if we follow

Here we focus more on the CC since we can reduce more features with it, as shown in the second row of Table 1. From the table, one can reduce only one feature per LC application, whereas one can find the nearest 2 ⁿ value concerning the maximum size of the first or second sequence for CC [34]. For getting the same number of values in both the input sequences, that is, N, both sequences are zero-padded as per need.

CC is one of the important properties of discrete Fourier transform (DFT). Equation (3) represents the mathematical form. If L is number of values in Y(L), that is, output sequence, n is number of values in x ₁(n) and x ₂(n), and N is the nearest 2 ⁿ of [max (length(x ₁(n)), length(x ₂(n)))], then CC is mathematically expressed [35] as

The above convolution equation involves index ((L−n)) _N , known as CC. It is an important property of DFT. The multiplication of a DFT of two sequences corresponds to the CC of two sequences in the time domain [17].

As the proposed algorithm uses CC, we have different methods to find CC.

Consider x ₁(n) and x ₂(n) are the two sequences used to convolve linearly with the number of samples “m” and “n,” respectively, to get the output sequence y(l), containing “l” number of samples.

3.2 Method 2: CC by DFT/inverse discrete fourier transform (IDFT) method

To find the CC of two sequences of length “m” and “n,” condition m = n should meet, and the length of two sequences should be close to the upper 2 ⁿ value. To meet this condition, one has to apply a zero-padding technique to the input sequence and then find the CC of the input sequences.

The architecture of the proposed system based on CC is as follows:

As shown in Figure 1, the input dataset undergoes data-preprocessing techniques. Then normalized data is divided into two groups, with elements nearly equal to the closest 2 ⁿ in each group. These two sets of features act as input to CC to extract a reduced set of features at the output. Afterwards, this reduced set of features is tested with a classifier to judge the proposed feature reduction algorithm’s performance.

Figure 1

Proposed feature reduction system using FrbyCC.

According to the above-mentioned method 1 and method 2 for finding CC, there are two variants of the FrbyCC method.

3.2.1 LCE method (FrbyLCE)

In this method, zero padding is applied to x ₁(n) and x ₂(n) so that each sequence will contain L (L = m + n − 1) number of elements. With this method, at a time, one feature is reduced. Here to reduce the number of attributes, we have to apply FrbyLCE repeatedly on the input.

Algorithm 1

Feature reduction using the LC equivalence method.

Input: F(n) – set of input features.

Output: Y(L) – reduced set of features.

Begin

  1. Divide the number of input features F(n) in two parts x ₁(n ₁) and x ₂(n ₂) … n ₁ ≈ n ₂

  2. Find the number of elements in output, that is,

    L = n ₁ + n ₂ − 1.

  3. Perform zero padding

    x ₁(L) = [x ₁(n ₁), zeropadding (1, L − n ₁)]

    x ₂(L) = [x ₂(n ₂), zeropadding (1, L − n ₂)]

  4. Find CC of x ₁(L) and x ₂(L),

      X ₁(K) = DFT(x ₁(L)),

      X ₂(K) = DFT(x ₂(L)),

      Y(K) = X ₁(K) ∗ X ₂(K), and

      Y(L) = IDFT (Y(K))

End

3.2.2 DFT IDFT circular convolution method (FrbyCC)

Here, depending upon the number of points/samples, that is, N expected in DFT, zero padding is applied. The number of points in DFT could be 2, 4, 8, 16, 32, 64,…

Algorithm 2

Feature reduction using the DFT and IDFT CC method.

Input: F(n) – set of input features.

Output: Y(L) – reduced set of features.

Begin

  1. Divide the number of input features F(n) in two parts x ₁(n ₁) and x ₂(n ₂) … n ₁ ≈ n ₂

  2. Select number of elements in output, that is, N

    N = nearest higher DFT point in the range of 2 ⁿ , which is 2, 4, 8, 16, 32, 64,…

  3. Perform zero padding

    x ₁(n) = [x ₁(n ₁), zeropadding (1, N − n ₁)] and

    x ₂(n) = [x ₂(n ₂), zeropadding (1, N − n ₂)].

  4. Find CC of x ₁(n) and x ₂(n),

      X ₁(K) = DFT(x ₁(n)),

      X ₂(K) = DFT(x ₂(n)),

      Y(K) = X ₁(K) ∗ X ₂(K), and

      Y(L) = IDFT (Y(K))

End

4 Experimental setup

The experiment is carried out on Intel core i5-7200U CPU @ 2.50 GHz with 8 GB RAM and 64-bit Windows 10 Operating System laptop. The feature reduction and classification algorithms are implemented in Matlab 2015b.

5 Experimental evaluation

The proposed algorithm is applied to the dataset for feature reduction. The percent feature reduction is calculated to get the effectiveness of the proposed algorithm. Equation (4) finds percent feature reduction, which is the ratio of dimensions reduced after application of proposed algorithm to the original dimensions present in the dataset.

The reduced dataset is classified using SVM, K-nearest neighbor, and decision tree (DT) classifiers. A summary of an effect of the proposed feature reduction algorithm on different datasets is given in Table 3.

Table 3 shows that the FrbyCC algorithm has >30% reduction for large attribute datasets. Feature reduction also reduces the storage space and execution time of an algorithm. It is always useful to process a dataset containing a large volume of attributes. Metrics, such as accuracy, specificity, sensitivity, F-measure, and so on, are used to assess imbalanced datasets [37]. The evaluation of the proposed system is done by using the most frequently used accuracy metric on different datasets.

FrbyLCE method reduces one attribute per execution of the routine. One has to apply the FrbyLCE five times with the output of the previous stage applied as input to the next stage for reduction by five features. Figure 2 shows a two-stage FrbyLCE in a cascade fashion.

Figure 2

Feature reduction in a two-stage FrbyLCE method.

After the desired reduction in attributes, a classifier evaluates the results. Here two classifiers such as DT and SVM are used to evaluate the proposed method. Table 4 shows the results of the LCE–FrbyLCE method with DT classifier.

Comparing Tables 4 and 5, it is clear that the FrbyLCE method’s use shows improvement in the accuracy of the DT classifier. The LCE method combines the two sets of input attributes to produce a new set of attributes, making DT classifier to classify more correctly on it. Hence, as we reduce features one by one, the DT classifier’s accuracy increases, and from stage III, accuracy remains constant for further reduction in features as observed from Table 4. Also, it is clear in Table 5 that the SVM classifier cannot select decision boundaries as features generated after FrbyLCE are a combination of two sets of input attributes. Here the accuracy decreases as we reduce the number of attributes.

Now, Table 6 shows FrbyCC accuracy evaluated with DT, KNN, and SVM classifiers. The results are taken on nine benchmark datasets from the UCI repository. Accuracy is taken as the performance measure for evaluation. The results are compared with the existing PCA feature reduction algorithm. If the accuracy of a system is represented by A, the accuracy improvement with the proposed FrbyCC method is represented by Δ [53] and calculated as shown in equation (5),

The assessment of Table 6 leads to the fact that classification performance is improved by an average 8% with DT classifier and 5% for SVM classifier. The 18% average accuracy improvement is shown with KNN classifier. The accuracy improvement achieved is purely due to feature extraction done by FrbyCC algorithm. FrbyCC multiplies and adds the results of multiplication; that is, it maintains the importance of features in extracted features too. The features extracted are convolved output of the input features with important features influencing the extracted features. Despite of feature reduction, the impact of important features remains in the extracted output. Also, the difference in the improvement of FrbyCC with different classifiers is observed due to the impact of individual classifier behavior on the respective dataset. This proves that the FrbyCC method was very effective in accuracy terms. FrbyCC reduces more features in a single application on the dataset under consideration.

Table 6 shows the results with tenfold cross-validation. Observations show no drop in the accuracy of the classifier, despite feature reduction. In comparison with PCA, the FrbyCC method gives improved accuracy.

When we compare the FrbyLCE with FrbyCC, FrbyCC seems to be the most effective method. It reduces a significant number of features in one application, whereas FrbyLCE is to be applied in cascade fashion n times to reduce the “n” number of features. To verify accuracy in different datasets, FrbyCC is applied to different datasets, such as Parkinson’s dataset, Arrhythmia, Internet ads, QSAR, and SCADI dataset, and compared with the PCA algorithm. As seen in Table 3, the first five datasets are high dimensional. We get a reduction in features up to 60% with the FrbyCC algorithm. The rest of the datasets have fewer dimensions as compared to the first five. Our focus is on high-dimensional datasets, in which the reduction in features leads to a decrease in execution time and storage space. The best reduction is observed in features if half of the total number of features is near any 2 ⁿ .

The following graphs show each dataset’s accuracy with DT, KNN, and SVM, which implies that all the classifiers work differently on each dataset with improved accuracy.

It is observed from Figure 3 that DT and KNN work well, showing increased accuracy with the FrbyCC method compared to PCA. SVM gives accuracy in line with the PCA algorithm. Since PCA is also a feature extraction technique based on a linear combination of input attributes like FrbyCC, both algorithms work with similar accuracy for SVM.

Figure 3

Comparison of accuracy of FrbyCC and PCA with DT KNN SVM classifier.

In Matlab, time is measured using function tic and toc, which gives execution time in seconds. It is clear from Table 7 that the execution time of the FrbyCC algorithm is less than that of the PCA algorithm as FrbyCC is implemented using a DFT algorithm. Figures 4–6 show the graphical execution time comparison of FrbyCC and PCA feature reduction algorithm for different classifiers.

Figure 4

Comparison of the execution time of FrbyCC and PCA with DT classifier.

Figure 5

Comparison of the execution time of FrbyCC and PCA with KNN classifier.

Figure 6

Comparison of the execution time of FrbyCC and PCA with SVM classifier.

FrbyCC is also tested with Naïve Bayes classifier, which is a probabilistic classifier. Naïve Bayes purely works on the independence assumption between the input features. Hence Naïve Bayes algorithm shows poor accuracy on FrbyCC results. Here FrbyCC algorithm extracts the reduced features by overlapping of original features. Hence Naïve Bayes does not classify with good accuracy.

6 Conclusion

Feature reduction and accuracy are crucial concerns in data classification. High data dimensionality is the main issue in data classification. The problem of high-dimensional data handling is addressed in this research. We proposed a feature extraction method based on convolution to reduce features without loss of classification accuracy. In the first technique described in this article, feature reduction by LCE reduces one feature per application. It works well with the DT algorithm. The second method, FrbyCC, proves very effective in dimensionality reduction. Experiements show that it works well with DT and KNN. For SVM, accuracy is seen inline with the PCA for most of the datasets. It is observed that the feature reduction in each of the dataset depends on its proximity with 2 ⁿ value. The average increase in accuray (Δ) achieved with DT, SVM, and KNN is 8.06, 5.80, and 18.80, respectively, on different benchmark datasets. The proposed algorithm reduces execution time with the use of the DFT/IDFT [52]. Overall in FrbyCC algorithm, with feature reduction, we relish the accuracy with less storage space and execution time.