Enhanced Image-Based Malware Multiclass Classification Method with the Ensemble Model and SVM

1 Introduction

Malware has become one of the most significant threats to network security because it can damage computer systems without the users’ permission (Sharif, Jiwani, Gupta, Mohammed, & Ansari, 2023). From stealing personal information to violating the nation’s and countries’ legitimate rights and interests, criminals commit a range of criminal crimes (Alzahrani, 2023; Zhao, Zhao, Yang, & Xu, 2023). Globally, the number of malware files discovered daily has increased by 5.2% from the average of 360,000 files detected daily in 2020 (Awan et al., 2021a,b).

Based on a 2018 Microsoft Security and Intelligence report, Ethiopia has the top average monthly ransomware encounter rates, followed by Mongolia, Cameroon, Myanmar, and Venezuela. Each of these countries had an average monthly ransomware encounter rate of at least 31% (Diana et al., 2018). Ethiopia’s Information Network Security Agency reported on May 3, 2022, that the country had prevented more than 5,856 cyber-attacks by the time (Ethiopia Situation Report, 2022) and 6,768 attempts were made to carry out cyber-attacks in 2023.

In Africa, the brand anniversary scam is a well-known phishing scheme. Threat actors pose as well-known companies, like Ethiopian Airlines and Ethio Telecom, to persuade unaware individuals into filling out a quick survey or questionnaire in exchange for a free gift. After entering their information, a person must forward the message to 20 friends or 5 WhatsApp groups to be eligible for a reward. Regretfully, those who unintentionally agree with these scammers’ requests will actually offer them with an opportunity to obtain personal information and even device information (INTERPOL, 2023).

Because malware attacks are becoming more frequent, a highly efficient method of malware detection is necessary. Numerous methods have been used for malware detection over the years. These vary from thorough manual labelling to complex hybrid systems (Mohammed, Ibrahim, & Salman, 2021). When it comes to analyzing malware, anti-malware software uses association rule mining, data mining, and information retrieval and extraction techniques. However, using these approaches has led to a sharp increase in the advent of new malware.

Static and dynamic analysis are the most popular techniques and often used methods for classifying malware. The majority of the metadata is extracted from malware binaries via static analysis, which does so without running or launching the files. Without actually executing harmful code, it enables you to recognize malware signatures. However, malware that uses polymorphism or obfuscation packaging can be extremely challenging to analyses using this technique (Djenna, Bouridane, Rubab, & Marou, 2023).

To gather information on malware behavior and its effects on the system in real time, dynamic analysis is performed by watching how the malware behaves while it is being executed. Although dynamic analysis has significant drawbacks, such as the fact that installing it would require a lengthy time to analyze the executable, it is believed to be more dependable and efficient in the long run. Dynamic analysis is always better than static analysis because, although malware may vary in structure, its behavior and other characteristics will never alter. This makes malware detection easier with dynamic analysis (Djenna et al., 2023), but these methods are feature-heavy or demand a high level of domain expertise to construct the features. This presents a challenge because malware develops at a rapid pace and this approach is unable to keep up. Still, most well-known anti-malware programs also implement a signature-based approach, which involves building a local database and storing the malware’s signature patterns (Jung, Bae, Choi, & Im, 2020). Additionally, the adoption of quantum cryptography has the potential to foster the development of highly secure communication channels. This advancement could prove invaluable in combating malicious software and facilitating the transmission of sensitive information (Canto, Sarker, Kaur, Kermani, & Azarderakhsh, 2022; Canto, Kermani, & Azarderakhsh, 2021; Kaur, Canto, Kermani, & Azarderakhsh, 2023; Kermani, Bayat-Sarmadi, Ackie, & Azarderakhsh, 2019).

In the past few years, deep learning (DL) and machine learning (ML) have shown promise in the fields of cybersecurity analytics, malware detection, cyberattack mitigation, botnet identification, malware classification, intrusion detection, prevention, incident response, network traffic analysis, cybercriminal identification, and deep packet inspection. This study aims to implement an ensemble of different neural networks for the detection of malware. The following are the paper’s significant contributions:

1. A novel hybrid classification model for malware detection based on ensemble learning of three models, which enables us to improve malware detection accuracy.

2. Using the support vector machine (SVM) classifier in conjunction with the Least Absolute Shrinkage and Selection Operator (LASSO) algorithm as a feature selection strategy, which chooses the fewest features necessary to enhance the classification performance of an image-based malware detection system.

3. A new dataset was produced for the detailed experimental investigation of DL approaches for multivariate malware detection through the merging of two open-access datasets (the Microsoft malware dataset and the Malimg dataset).

The following is the format of the remaining sections of this study: a sufficient comparison of current techniques and approaches is included in Section 2, along with a discussion of related studies. The methodology employed in this article is explained in Section 3. Results are discussed in Section 4. The research conclusions are presented in Section 5.

2 Related Work

In this study, we examine current malware detection techniques, such as static, dynamic, and visualization analyses, based on DL and ML methods.

Statistical analysis (Krumbach & White, 1964) uses data mining techniques including information gain, principal component analysis based on SVM, J48, and Naïve Bayes classifiers. They extract DLLs, application program interface methods contained in each DLL of a Windows PE file, and PE header information from Windows executables. Principal component analysis is used to reduce the dimensionality of the selected features after information gain and calling frequencies of the raw features are determined to identify valuable subset features. Their system achieves a 99.6% detection rate. Shabtai, Moskovitch, Feher, Dolev, and Elovici (2012) investigated Opcodes using various classifiers and n-grame sizes. According to their claims, the 2-gram Opcode approach performed better than the byte-n-gram methods. Using neural network technology, Saxe and Berlin (2015) created a method to differentiate between malware and benignware. The entropy histogram from binary data, the executed files metadata, and DLL import were computed using four different kinds of features that they collected. These characteristics were converted to vectors of 256 dimensions. They discovered that the FPR was 0.1% and the TRP was 95.2%; nevertheless, their methodology necessitated feature engineering tools. The created method’s accuracy is not sufficient, and neither is it clear how accurate the features of the malware and benign samples are.

Using dynamic analysis, Lim and Moon (2015) created a malware identification technique by analyzing network traffic behavior. They compared character sequences to identify similarities using sequence alignment and flow feature clustering. Lim and Moon (2015) used a hidden Markov model for malware classification, relying on system calls as observed symbols.

To overcome malware family classification challenges, researchers have employed visualization techniques that minimize feature engineering costs and domain-specific knowledge. For instance, Sharif et al. (2023) used convolutional neural network (CNN), Caps-Net, VGG16, ResNet, and InceptionV3 DL algorithms with respect to an image-based malware dataset called Malimg to detect malware. With early stopping conditions, the models were trained up to epoch 50 and were 92% accurate in diagnosing malware. Twenty-five different malware types are categorized using their model. Saridou, Moulas, Shiaeles, and Papadopoulos (2023) used binary visualization on both benign and malicious files to detect malware to build a ResNet50 model. A 93.60% accuracy, 94.48% precision, 92.60% recall, and a 93.53% f-score are attained by their model. AlexNet CNN serves as the foundation for a unique malware detection technique proposed by Zhao et al. (2023). The experimental findings demonstrate that the GCJ dataset has an accuracy of 99.38% and the Microsoft malware dataset has an accuracy of 99.99%. To identify malicious executables, Alzahrani (2023) suggests a stacking-based ensemble strategy that combines CNN, LSTM, and GRU models. The experiment’s findings show that a 99.02% accuracy rate was attained. Awan et al. (2021a,b) present a DL framework based on spatial attention and CNNs for image-based malware classification of 25 popular malware families with and without class balancing. Accuracy, recall, specificity, precision, and the F1 score were used to assess performance on the Malimg benchmark dataset. They found that their proposed class-balanced model achieved 97.68, 97.95, 97.33, 97.11, and 97.32%. Salota and Singh (2023) used modified VGG based on an image-based malware dataset for malware identification. The proposed model has a 99% accuracy rate for correctly identifying malware. In Altaiy, Yildiz, and Bahadır (2023), The Long-Short-Term Memory Network, CNN, and Multitasking Deep Neural Network are the three DL techniques applied in this study. The most successful outcomes of this investigation showed that DNN has an accuracy of 0.982, a precision of 0.988, and a recall of 0.990. Shaukat, Luo, and Varadharajan (2023) suggested an approach that eliminates the necessity for complex feature engineering activities and domain expertise by combining DL and support vector ML. On the Malimg dataset, the suggested method performed better, with an accuracy of 99.06%. The work by Chen, Xing, and Ren (2023) suggests a method for classifying and identifying malware that utilizes bicubic interpolation to enhance the security of plant protection information terminal systems. According to experimental findings, the method can greatly increase the effectiveness of malware categorization. The Microsoft Malware Classification Challenge Dataset (BIG2015) can produce RGB and grey images with an accuracy of 99.76 and 99.62%, respectively. Abhinav, Akshay, Anshad, Mohan, and Usha (2023) demonstrated how effective ensemble learning is at identifying malware, particularly when combining the VGG-16 and Efficient Net models. The study found that the combination of these models achieved an overall accuracy of 96.8%. To successfully and precisely classify malware, Yadav and Tokekar (2023) make use of visualization and suggest a CNN-based DL model. The proposed model achieves 98.179, 97.39, and 97.70% for accuracy, precision, and F1-score, respectively, and with a classification speed of 3 s needed to test 934 unknown malware. Nguyen, Di Troia, Ishigaki, and Stamp (2023) use a variety of techniques to extract features from malware executable files and represent them as images. Next, they concentrate on multiclass classification using generative adversarial networks (GAN) and assess how well their GAN findings compare to those of other well-liked ML methods such as SVM, XGBoost, and RBM. Aurangzeb and Aleem (2023) offer a fast, scalable, and precise DL method for detecting obfuscated Android malware on both real and emulator-based systems. Experiments demonstrate that the suggested approach reliably and successfully detects malware, as well as features that are typically hidden by malicious software. Panda, CU, Marappan, Ma, and Veesani Nandi (2023) proposed a novel ensemble model called Stacked Ensemble Auto Encoder, GRU, and MLP, or SE-AGM, that is trained on the 25 key extracted features that are encoded from the benchmark Malimg dataset to facilitate classification. The SE-AGM achieved an average accuracy of 99.43%.

3 Methodology

3.2 Overview of the Proposed Ensemble Model

Using concatenated deep features collected from malware images of three different models (the CNN architecture of VGGNet, AlexNet, and InceptionNet), our model aims to accurately diagnose malware families. Figure 1 shows the proposed concatenated model architecture. First, the input images are diminished to 32 × 32 to minimize resource and computational time. Second, we used the technique called augmentation to only the classes containing images less than 1,000 to make it possible for the trained model to extract a substantial amount of useful features. We apply different augmentation methods such as rotation of 45°, random vertical and horizontal flips, and random vertical and horizontal shifts. A total of 20,326 images made up the whole malware image dataset prior to augmentation, while 39,742 images exist after augmentation. The dataset was subdivided into three groups: The training set accounted for 70% of the dataset, the validation set for 10%, and the test set for 20%, both before and after augmentation. The model is trained through the execution of the training and validation phases prior to the testing phase. To evaluate how well the trained model performs with the new image dataset, the testing phase is carried out following the training and validation. Third, the most important features from the images are extracted using the concatenated model of the CNN models’ architecture. The features derived from the previous models are then integrated into a single classification descriptor.

Figure 1

Architecture of the proposed model.

To expose the extracted features of VGGNet, AlexNet, and InceptionNet to the LASSO feature selection algorithm, the features are lastly concatenated. The most significant features chosen by LASSO are then fed into the classifiers (Softmax and SVM) to determine which ones are the best. Both model features (VGGNet, AlexNet, and InceptionNet) are concatenated and fed to the previously trained Softmax and SVM models within the test set.

The models were trained for 70 epochs using early halting criteria. All trained models’ outcomes were compared with the support of the assessment measures. Accuracy and loss function are used to evaluate the concatenated model. We employed a categorical cross-entropy loss function, which is frequently used for multi-class problems to quantify the difference between the output predictions and the target output. To advance the model’s accuracy and decrease the loss, the learning rate and other parameters have been adjusted using an adaptive momentum estimation (Adam) optimization function, with a learning rate of 0.001. Finally, the model classification performance is computed using performance metrics.

We used the CNN models listed below, each of which has different capabilities.

3.2.1 VGG 16

It achieves 92.7% top-5 test accuracy on the 14 million images in the ImageNet dataset comprising more than 1,000 classes (Simonyan & Zisserman, 2014). It contains 16 layers requiring to be trained, 5 convolutional blocks, and 3 fully connected layers, and its architecture is shown in Figure 2. To provide the same spatial dimension for every activation map as the previous layer, the convolutional layers employed 3 × 3 kernels with 1 padding and 1 stride as their filter size. Rectified linear unit (ReLU) nonlinearity was introduced to every convolutional layer to speed up training and reduce the chance of vanishing gradient issues. A max-pooling step was also employed at each block end to reduce spatial dimensions. In max-pooling layers, a 2 × 2 kernel filter with no padding and two strides was employed to ensure that the spatial dimensions of the activation map were cut in half as compared to the previous layer. The final 1,000 fully linked Softmax layer was applied after the first two fully connected layers with 4096 ReLU-activated units were used. In this study, we used six convolution layers, two batch normalizations, three MaxPoolings, three dropout layers, and a flattening layer at the end, as presented in Figure 1.

Figure 2

VGG-16 architecture (Simonyan & Zisserman, 2014).

3.2.2 Inception v3

According to He, Zhang, Ren, and Sun (2015), Inception has a significantly reduced computational cost compared to VGGNet or its more successful successors. Its architecture is presented in Figure 3. The primary goal of Inception v3 is to reduce processing power consumption by making changes to the earlier Inception designs (Szegedy, Vanhoucke, Ioffe, Shlens, & Wojna, 2016). As illustrated in Figure 1, we employed an architecture consisting of five convolution layers, one batch normalization, two MaxPoolings, one dropout layer, and a flattening layer at the end.

Figure 3

Inception V3 architecture (Szegedy et al., 2016).

3.2.3 AlexNet

Eight layers make up the architecture: three fully connected layers and five convolutional layers. Its architecture is shown in Figure 4. At 15.3% for top-5 error rate, AlexNet became the winner in the 2012 ImageNet competition. Because the AlexNet model extracts Deep features from its layers within a short period of time, it was selected (Krizhevsky, Sutskever, & Hinton, 2012). Figure 1 illustrates an architecture we employed, which included two AveragePoolings, three convolution layers, and a flattening layer at the end.

Figure 4

The Architecture of AlexNet (Krizhevsky et al., 2012).

This model was chosen due to their excellent performance in several malware picture classification settings. We used two different classifier types and a feature selection approach in our ensemble model: Feature selection: in ML, feature selection is an essential step that selects the optimal attributes to decrease system training time, increase accuracy, and lower problems with overfitting (Cai, Luo, Wang, & Yang, 2018). This study selected important features from each model’s Deep features using the LASSO algorithms (Tibshirani, 1996) to accurately detect each malware among other malware families. The LASSO adjusts the linear regression’s cost function by incorporating a regularization parameter (λ) that penalizes the whole sum of all coefficients to reduce the mean square error, as expressed by the following equation (Tibshirani, 1996, 2011):

(1) MSE ( β ) = ∑ i = 1 , j = 1 m ( a i − p i ) 2 + λ ∑ ∣ β j ∣ ,

where λ is a nonnegative regularization parameter, and βj is a coefficient vector of length m. To minimize the value of the cost function, the absolute values of the weights would need to drop (shrink) as the regularization strength increased. Consequently, the features of the less important weights are eliminated from the subset and their weights become zero as a result of the LASSO. Consequently, LASSO regression offers the noteworthy benefit of carrying out an automated feature selection process. The shrinkage regulating factor, lambda (λ), is largely relied upon by the LASSO. When λ equals 0, all features are taken into account. When λ grows, fewer features are chosen; when λ gets closer to infinity, all features are eliminated from the subset. Let us say that a group of features has a high correlation with one another. Then, since their existence will increase the value of the cost function, LASSO chooses one of them and reduces the coefficient of the remaining ones to zero to choose the optimal feature subset (Tibshirani, 1996, 2011; Wang, Li, & Tsai, 2007). To train and evaluate the two key classifiers for malware family detection, the most significant features derived from the LASSO technique were utilized. The first is the supervised learning technique called Softmax classifier, which is typically applied when there are several classes involved (Ren, Zhao, Sheng, Yao, & Xu, 2017). The second is Multiclass SVMs (Williams, 2003), a well-known and efficient multi-label classification algorithm (Tan, Tan, Jiang, & Zhou, 2020).

4 Experimental Results and Discussion

4.1 Results

The proposed ensemble model achieved 99.99 training accuracy, 99.98% validation accuracy, and 99.78% testing accuracy in malware category classification under the described experimental conditions. The classification accuracy of other baseline CNNs and the proposed ensemble model are presented in Table 3.

Table 3

Classification accuracy of each model and the proposed ensemble model

Ex. no.	Model	Feature selection and classifier		Test accuracy (%)
1.	VGGNet	Feature selection: no	Classifier: Softmax	85
			Classifier: SVM	86
		Feature selection: LASSO	Classifier: Softmax	86.5
			Classifier: SVM	87.5
2.	Inception V3	Feature selection: no	Classifier: Softmax	84.5
			Classifier: SVM	85.5
		Feature selection: LASSO	Classifier: Softmax	82
			Classifier: SVM	83
3.	Alex Net	Feature selection: no	Classifier: Softmax	83.5
			Classifier: SVM	84
		Feature selection: LASSO	Classifier: Softmax	85
			Classifier: SVM	86.5
4.	VGGNet + InceptionV3 + SVM	Feature selection: no		96
		Feature selection: LASSO		98
5.		Feature selection: no		92
	VGGNet + AlexNet + SVM	Feature selection: LASSO		93
6.	InceptionV3 + AlexNet + SVM	Feature selection: no		90
		Feature selection: LASSO		91
7.	VGGNet + InceptionV3 + AlexNet + SVM	Feature selection: no		97
		Feature selection: LASSO		99.78

Table 3 shows that the developed model outperformed all other models in terms of classification accuracy improvement. The findings of the experiment revealed that the proposed ensemble model was superior in accuracy (99.78%) with LASSO feature selection technique than the models that were used without feature selection algorithms (VGGNet + Softmax [85%], VGGNet + SVM [86%], Inception V3 + Softmax [84.5%], Inception V3 + SVM [85.5%] + AlexNet + Softmax [83.5%], AlexNet + SVM [84%], VGGNet + InceptionV3 + SVM [96%], InceptionV3 + AlexNet + SVM [90%], VGGNet + InceptionV3 + AlexNet + SVM [97%] and VGGNet + Softmax [86.5%], VGGNet + SVM [87.5%], InceptionV3 + Softmax [82%], InceptionV3 + SVM [83%] + Alex-Net + Softmax [85%], AlexNet + SVM [86.5%], VGGNet + InceptionV3 + SVM [98%], VGGNet + AlexNet + SVM [93%], and InceptionV3 + AlexNet + SVM [91%]) with feature selection algorithm.

As demonstrated by Figure 5, a growing trend in both training and validation accuracies shows that the model is both learning from the training data and has good generalization capabilities to new data. The gap between training and validation accuracies does not differ much; it may be a hint that overfitting is not occurring and the model functions well on both sets of data. A consistent and gradual reduction in both training and validation losses signifies the model’s enhancement over time, suggesting that it learns the underlying patterns in the data without overfitting, as observed in Figure 5. In conclusion, Figures 5 and 6 offer essential insights into the model’s learning process from the data and its capability to provide precise predictions on unseen data.

Figure 5

Training and validation accuracy.

Figure 6

Training and validation loss.

Table 4 provides a comprehensive comparison between various malware classification approaches, encompassing both ML and DL techniques, alongside our proposed model. Notably, our suggested model shows superior performance in terms of classifying more malware samples and its accuracy compared to existing approaches. This suggests that our model exhibits enhanced efficacy in identifying and categorizing malware instances. The experimental result shows that the proposed model reveals marvelous result both in performance and efficiency in 34 types of malware identification, this much types of class are not considered by other previous works. Prior to employing feature selection techniques, the model required an average of 2.5 s to categorize a malware sample. However, after implementing the LASSO feature selection technique, this time reduced significantly to 1.15 s. This reduction in processing time signifies the effectiveness of the LASSO feature selection technique in optimizing the model’s performance and streamlining the malware identification process. Overall, the comparison presented in Table 4 emphasizes the superiority of our proposed model in terms of malware classification, supported by empirical evidence of its efficiency and effectiveness in identifying malware samples within minimal processing time.

Table 4

A comparative summary

No.	Method	Feature selection	No of class	Accuracy (%)
1.	ResNet50 (Saridou et al., 2023)	—	2	93.60
2.	Alex Net (Zhao et al., 2023)	—	9	99.99
3.	Spatial Attention CNN with VGG19 (Awan et al., 2021a,b)	—	25	97.68
4.	VGG16 + ResNet-50 + SVM (Vasan, Alazab, Wassan, Safaei, & Zheng, 2020)	PCA	25	99.5
5.	DL + SVM (Shaukat et al., 2023)	—	25	99.06
6.	CNN + Cycle-GAN (Chen et al., 2023)	—	9	99.76
7.	VGG-16 and Efficient Net	—	25	96.8
8.	CNN (Yadav & Tokekar, 2023)	—	25	98.179
9.	SE-AGM (Panda et al., 2023)	—	25	99.43
10.	Proposed (VGGNet + InceptionV3 + AlexNet + SVM)	LASSO	34	99.78

5 Conclusion

This study presents an effective ensemble DL approach based on the LASSO feature selection algorithm to classify malware families. It attained high accuracy while increasing detected malware varieties. Generally, the accuracy of the proposed ensemble model was 99.78%. This shows the efficiency of the model in extracting Deep features from the 34 malware images and achieving accurate detection of malwares. The experimental outcomes exposed that the LASSO feature selection algorithm improved the approach performance with a few selected Deep features instead of using the total Deep features in the classification process. Moreover, the LASSO algorithm improves the model performance, in reducing number of selected features, and execution time. As well as, the SVM classifier provided high system performance compared with the Softmax classifier. The proposed ensemble DL model resulted in the best results of 99.78% testing accuracy, 99.99% training accuracy, 99.98% validation accuracy, 99.93% precision, 99.67% sensitivity, 99.95% specificity, and 99.87% F1-score. This performance was achieved by extracting Deep features from the combined dataset Malimg dataset and Microsoft malware dataset using VGGNet, Inception V3, and Alex Net model, applying the LASSO feature selection algorithm.

For the future researchers, we recommend the use of different segmentation and noise-filtering techniques that may increase the performance of the model. Additionally, it is recommended the inclusion of an integrated approach that combines malware analysis, post-quantum cryptography, and ML concepts to achieve better accuracy as adopting post-quantum cryptography measures is crucial in ensuring robust security in the face of advancing quantum computing technology to establish resilient encryption methods that can withstand potential quantum-based attacks (Canto, Kermani, & Azarderakhsh, 2022; Cintas-Canto, Kermani, & Azarderakhsh, 2022; Cintas-Canto, Kaur, Mozaffari-Kermani, & Azarderakhsh, 2023; Kermani, 2007; Kermani, Azarderakhsh, & Mirakhorli, 2016; Koziel, Jalali, Azarderakhsh, Jao, & Mozaffari-Kermani, 2016; Mozaffari-Kermani, Azarderakhsh, Ren, & Beuchat, 2016; Niasar, Azarderakhsh, & Kermani, 2020).