Hyperparameters optimization of evolving spiking neural network using artificial bee colony for unsupervised anomaly detection

3.1 Anomaly detection

Anomaly or outlier detection identifies unexpected patterns, observations, or events in datasets that differ from the norm [33]. Anomaly detection stands out as a frequently encountered challenge within the realm of machine learning, exhibiting a diverse array of applications across various domains. Such applications encompass fraud detection, defect detection, and intrusion detection [34]. Anomalies can arise from unforeseen processes producing the data. For instance, in chemistry, an anomaly could be triggered by the improper execution of an experiment. Similarly, within medicine, a particular disease might give rise to unusual symptoms, constituting an anomaly. Moreover, in predictive maintenance, an anomaly can signal the early onset of system malfunction [35]. Popular techniques for anomaly detection include clustering-based, probabilistic-based, density-based, distribution-based, and distance-based approaches [36].

The anomalies generally fall into three categories significantly influencing the anomaly detection algorithm: point, contextual, and collective anomalies [1]. Point anomalies could be considered anomalous, among other instances in the dataset, such as a building temperature value having a typical shape and a noticeable rise/fall in the value, which may be considered an abnormal value. Contextual anomalies are observations considered normal but abnormal in other instances. For example, while heavy traffic on the highway is usual during working hours, it is improper traffic behavior after midnight. A collective anomaly is a set of related points that are normal individually but anomalous if taken together; an example of a collective anomaly is cardiac behavior, which cannot be detected by a single time interval observation but may be recognized by collective signals. Graphical representations of these anomaly categories are shown in Figure 1.

Figure 1

Anomaly categories [1].

3.2 eSNNs

eSNN is one of the most promising SNN architectures. It was first presented by Wysoski et al. [37] and used for visual pattern recognition. eSNN increases the adaptability of SNN by combining the produced neurons incrementally to identify the pattern in the given problem [38]. In this kind of network, the spiking neurons are generated (evolved) and incrementally combined to collect clusters and patterns from new data [12]. The eSNN iteratively created repositories for each class to make new data available without re-training the learned dataset [39]. The numerical data is converted into an encoding or spike trail for each input sample. All input samples are distributed among neurons using overlapping Gaussian receptive field neurons [40]. Each input feature is represented in a spatio-temporal spike pattern after encoding. Each GRF pre-synaptic neuron center μ j and width are calculated [27]

Here, I max n and I min n represent the interval values of the nth feature in a given window size, N is the number of receptive fields (GRF) for each feature, and parameter β (also known as overlap factor) controls the degree to which Gaussian random fields overlap. The output of neuron j is represented as

where x is the input sample, and the time of firing for every pre-synaptic neuron j is defined as

where T is the synchronization time or spike interval.

To create initial neurons, the LIF model is used. The neuron will only fire whenever the post-synaptic potential (PSP) value exceeds its threshold value and fires at most once.

PSP of the ith neuron is identified as

where w j i denotes the weight between pre-synaptic neuron j and output neuron i, order(j) represents the rank of the pre-synaptic neuron spike, and mod represents the modulation factor with value in [0, 1]. A new output neuron is created and linked to every pre-synaptic neuron in the previous layer, and the weights are determined using rank order for each sample that falls within the same class as follows:

The threshold γ _i of a newly generated output neuron is defined as

where C is a user-fixed value in [0, 1].

The weight vector for a recently produced output neuron is compared with existing output neurons in the repository. When the Euclidean distance between the weight vector of the already created output neuron and either of the prior output neurons is less than or equal to the SIM, then they are merged, and the following formulas are employed to calculate their thresholds and weight vectors:

The variable M defines the count of earlier mergers involving similar neurons in the learning history of the eSNN. Once a merger occurs, the vector of weight for the newly formed output neuron is no longer considered, and the model is presented with the updated pattern. If no one of the existing neurons in the repository matches the newly generated output neuron (based on the SIM parameter), it is added to the repository. The entire concept of eSNN may be found in [41].

Recently, the OeSNN-UAD approach was introduced by Maciąg et al. [2] as a detector for streaming data, which adapts the OeSNN proposed by Lobo et al. [27]. In contrast to OeSNN, OeSNN-UAD did not split the output neurons into distinct classes; instead, it consisted of three additional modules: value correction, anomaly classification, and generation of output values of candidate output neurons, as shown in Figure 2. Under two separate conditions, the OeSNN-UAD classifies the input value as anomalous. First, if no output neuron in the repository is firing. Second, if the difference between the actual input value and its OeSNN-UAD prediction exceeds the average of prediction error plus a user-given multiplicity of the standard deviation of the most recent prediction errors, more details are in [2]. In this work, the OeSNN-UAD was employed as a baseline architecture.

Figure 2

OeSNN-UAD architecture [2].

3.3 Overview of ABC algorithm

Metaheuristic algorithms are stochastic optimization approaches that search for the most optimal solution to an optimization problem from among all potential solutions [42]. Some of these algorithms focus on the alteration and enhancement of a singular candidate solution, exemplified by simulated annealing, tabu search, and iterated local search. Other algorithms, such as PSO, CS, ACO, and FPA, are population-based, which means that several candidate solutions are maintained and improved during the search until an optimal solution is obtained when the solutions of the whole population are similar [43,44]. Most population-based metaheuristic algorithms were influenced by the collective intelligence of swarms of animals and insects, including fish, birds, bacteria, ants, and fireflies. Honeybees are one of the social creatures that exhibit interesting behaviors and traits. When they act as swarms, honeybees are fascinating creatures that display a variety of unexpectedly clever behaviors [45].

The ABC is one of the bee-based algorithms developed by Karaboga [46]. ABC involves three kinds of bees: employed, onlooker, and scout bees. The employed bees move randomly to explore positions of food source, and when they reach the food source, they generate a new solution and use a greedy selection technique to compare it with the previous one. Then, by waggle dancing, they interact with the onlooker bees about the path of food source, profitability, and nectar amount. The onlooker bees select the new food source based on a probability proportional to the value of the food source. When the food source does not improve after specified trials, a different food source chosen randomly will be abandoned and replaced. The employee bee assigned to that food position leaves it and goes to work as a scout bee [47]. The ABC procedure requires a cycle of four phases: initialization, employed, onlooker, and scout phases:

5.1 Datasets description

The experimental datasets used in this research were obtained from the literature that is publicly available as anomaly detection benchmarks, namely the NAB [49] and Yahoo Webscope [50]. These two main datasets are separated into various categories, and each one includes numerous time series with labeled anomalies, which vary in length and anomalies contained. Both datasets are commonly used to evaluate the performance and accuracy of unsupervised anomaly detection methods. [14]. Table 1 shows the properties of these two benchmark datasets, which illustrate the total number of data files, the total number of observations, and the number of anomalies in each subgroup of the main datasets. Additionally, it identifies whether the anomalies in the dataset are single “point” or “collective” anomalies.

The NAB dataset is a univariate time series divided into seven labeled sub-datasets, both artificial and real, with six including anomalies and one containing only normal data, which was not considered in the experiments. The number of input values in data files varies between 1,000 and 22,000 instances. Overall, there are 58 time series, and each one includes time stamps and data values files, and the anomaly labels for each data file are defined within a separate set of files. Each time series exhibits a distinct set of characteristics, including temporal noise, short- and long-term periodicities, and concept drift. The data comes from a diverse of sources, including recordings of online advertisement clicks, metrics obtained from AWS servers, real data files such as hourly registered taxi schedules in New York City or CPU utilization, as well as freeway traffic recordings detailing speed or travel time and Twitter volume statistics. Additionally, NAB includes artificially generated data files to assess anomalous behaviors not yet observed in real data. Notably, all-time series within the NAB are imbalanced, with the proportion of anomalies in a time series being less than 10% on average. Figure 5 shows two sample files included in the NAB dataset.

Figure 5

Examples plots for sample files from NAB: Normal time series shown in blue lines, anomalies are labeled with red circles [8]. (a) AWS CloudWatch CPU utilization data. (b) Machine temperature sensor data.

The Yahoo Webscope datasets consist of four sub-benchmarks, A1, A2, A3, and A4 including 367 real and synthetic time series. Each subgroup contains 1,420–1,680 occurrences. The timestamps of the dataset are observed hourly, and anomalous values are indicated by humans according to the Yahoo S5 dataset’s guidelines. In synthetic time series, the anomalous values have appeared in random locations. The A1 benchmark includes 67 data files with real input values reflecting login activities on the Yahoo network. Each file is structured with three main time series: timestamps, input values, and labels indicating whether each input value is anomalous or not. The A2 benchmark comprises 100 synthetic data files, each containing anomalous values in the form of a single instance. Each file in this benchmark includes three time series: timestamp, input values, and labels. Besides, the A3 benchmark consists of 100 synthetic data files with anomalies in the form of single anomalous values. Beyond the three standard time series (timestamps, input values, and anomalies labels), these data files also include additional time series (trend, noise, seasonality, and changepoint). On the other hand, the A4 benchmark comprises 100 synthetic data files with anomalies. Most anomalies in this benchmark correspond to sudden shifts from one considerably distinct input data trend to another [2,14]. Although both A3 and A4 benchmarks share the same time series types, we utilize only the input values in all the experiments, discarding additional time series. Here, within the Yahoo Webscope dataset, all-time series are imbalanced, with the average proportion of anomalous input values typically falling below 1%. Figure 6 depicts the time series samples from the four sub-benchmarks.

Figure 6

Examples plots for the Yahoo dataset. Normal time series are shown in blue lines, and anomalies are labeled with red circles [51]. (a) A1 benchmark. (b) A2 benchmark. (c) A3 benchmark. (d) A4 benchmark.

5.2 Parameters setting

The ABC parameters were defined before the experimental setup, as listed in Table 2, with colony size value determined according to El-Abd [52]. In Table 3, the values of the hyperparameters of OeSNN are provided based on the literature [2]. The window size (Wsize) and anomaly classification factor (ε) have the most significant effect on anomaly detection [8,14], so their values are initialized following guidelines from [2,53,54].

5.3 Results on NAB dataset

Table 4 presents the obtained result for Precision, Recall, average F1-score, BA, and MCC produced by OeSNN-ABC for all types of time series in the NAB dataset.

Next, we continue to evaluate the proposed OeSNN-ABC method by conducting a comparison against some of the state-of-the-art methods presented by Maciąg et al. [2]. It should be noted from Table 5 that OeSNN-ABC betters the results obtained by the other detectors with a significant margin regarding the average F1-score for all NAB data files. The performance on the ArtificialWithAnomaly dataset is especially impressive, with a score of 0.800 that significantly outperforms the next top performer, OeSNN-UAD, at 0.427. The superiority trend is consistent across real-world datasets, with OeSNN-ABC maintaining a significant lead. For example, in the RealTraffic dataset, OeSNN-ABC scores 0.545, surpassing DeepAnT of 0.223 and OeSNN-UAD of 0.340. This finding confirms that good parameter selection enables the network to ensure the best output by integrating OeSNN with the metaheuristic optimization algorithm.

In Table 6, we provide the obtained result for recall and precision metrics compared with benchmarking anomaly detectors for selected data files from the NAB dataset. A high precision indicates that few normal samples are misclassified as anomalous, whereas a low recall means that many anomalies are missed. In addition, a high recall reflects that the model can identify a significant proportion of positive instances, which is important for detecting as many positives as possible, even if it means producing more false positives [55]. It can be observed that almost all of the detectors achieved high precision and low recall. The primary cause of this poor recall is the labeling method used in the NAB dataset [14]. The recall value below 0.01 reveals how limited the detector’s ability is to detect anomalies. Conversely, the OeSNN-ABC, in most cases, is characterized by much larger values of the recall. Moreover, OeSNN-ABC exhibits notably greater precision and recall for certain data files; thus, it is more efficient in detecting anomalies than the compared approaches.

Table 7 describes evaluation measures for the proposed OeSNN-ABC in contrast to the method introduced by Zhang et al. [56] and the OeSNN-UAD [2] methods for the RealKnownCause and RealTweets time series from the NAB dataset. Based on the results, it can be observed that the method of Zhang et al. has high precision values but low recall values, which implies that this method has a minimal ability to detect anomalies. That is because high precision values mean that the detector considers a few cases anomalous even if not labeled as anomalies in the data file. On the other hand, low recall values indicate that the OeSNN-UAD detector missed many anomalies. For the data files machine-temperature system-failure, TV-CRM, and TV-GOOG, the result of OeSNN-ABC is second best. Together, the present findings confirm that OeSNN-ABC performs noticeably better than other approaches.

Figure 7 illustrates the performance of the OeSNN-ABC compared to several deep-learning approaches for anomaly detection across multiple time series. It is worth noting that the proposed method consistently outperforms other techniques such as MadGan, MS Azure (CNN), Dense AE, and RE-ADTS (CV). The OeSNN-ABC performs notably well on the ArtificialWwithAnomaly scoring 0.80, which is much higher than the next best performer. This reveals that the approach may effectively detect artificially created anomalies. However, the performance on RealTweets is relatively modest compared to its success in other datasets, where its efficiency drops to 0.39, demonstrating that the approach may struggle to handle less organized or more dynamic data. In comparison, models like MadGan perform better in the RealTweets time series, achieving 0.44. For instance, in the RealAWSCloudWatch, the OeSNN-ABC lags behind the Dense AE model, which outperforms the proposed method by a margin of 0.10.

Figure 7

F1-score measure of OeSNN-ABC against deep learning approaches MadGan, MS Azure, Dense AE are presented in [57], RE-ADTS [58], and AWRM [59], Source: Created by the authors.

5.3.2 Comparative analysis with other optimization algorithms

Table 9 presents the comparison between OeSNN-ABC and the optimization algorithms: PSO, GWO, FPA, WOA, and GS in terms of F1-score. It should be noted that all these algorithms are tested under identical settings to assure fairness. The maximum number of iterations is 100, and the population size is 40. Observation showed that the ABC algorithm showed a notable and promising result when used as an optimizer for the hyperparameters of OeSNN.

Table 9

Result of average F1 score evaluation of OeSNN integrated with ABC, PSO, GWO, FPA, WOA, and GS for the NAB dataset

Time series	PSO	GWO	FPA	WOA	GS	ABC
ArtificialWithAnomaly	0.787	0.781	0.773	0.714	0.427	0.800
RealAdExchange	0.379	0.349	0.343	0.203	0.234	0.408
RealAWSCloudwatch	0.512	0.536	0.500	0.470	0.342	0.540
RealKnownCause	0.420	0.438	0.414	0.409	0.324	0.455
RealTraffic	0.529	0.510	0.505	0.477	0.340	0.545
RealTweets	0.381	0.367	0.339	0.325	0.310	0.395

The bolded numbers indicate the best results.

Furthermore, the experimental results presented in Figure 8 offer valuable insights into the performance of the proposed method across a range of diverse datasets, when compared to other optimization algorithms. OeSNN-ABC demonstrates consistent superiority over other algorithms, exhibiting exceptional efficacy in key metrics including precision, recall, F1-score, BA, and MCC. It is noteworthy that while algorithms like PSO and GWO exhibit competitive performance on RealAdExchange and RealKnownCause, respectively, they lack the consistent performance of OeSNN-ABC across a wide range of datasets.

Figure 8

Evaluation measures of OeSNN-ABC against PSO, GWO, FPA, WOA, and GS optimization algorithms: (a) Precision. (b) Recall. (c) F1-score. (d) BA. (e) MCC. Source: Created by the authors.

In Table 10, we present the optimal values of Wsize and anomaly classification factor ε obtained for the OeSNN-ABC as compared to other optimization algorithms across different time series that were used in the experiments of the results, which are presented in Table 6. It can be observed from Table 10 that for every time series, the combination of hyperparameter values is different. Thus, to achieve good performance by OeSNN, a correct combination of hyperparameters is required. Once the algorithm finds a correct combination of values for the hyperparameters, the OeSNN will classify at the best rate of accuracy.

Table 10

Optimal window sizes and anomaly classification factors used by ABC, PSO, GWO, FPA, WOA, and GS for data files from Table 6

Time series	Data files	ABC		PSO		FPA		GWO		WOA		GS*
		Wsize	ε	Wsize	ε	Wsize	ε	Wsize	ε	Wsize	ε	Wsize	ε
RealAWSCloudWatch	ec2-cpu-utilization-5f5533	220	3	310	2	150	3	150	2	210	6	100	2
	rds-cpu-utilization-cc0c53	600	9	600	11	600	11	600	13	600	11	100	4
RealAdExchange	exchange-2-cpc-results	70	7	100	2	100	2	110	2	100	2	300	2
	exchange-3-cpc-results	100	4	100	2	100	4	100	4	110	4	600	7
RealKnownCause	ambient-temperature- system-failure	340	8	530	2	320	4	290	4	230	3	500	6
	cpu-utilization-asg-misconfiguration	600	3	510	2	590	3	600	3	600	3	600	3
	ec2-request-latency-	100	4	390	6	600	4	100	4	100	4	400	5
	machine-temperature- system-failure	120	15	450	9	100	14	120	16	430	10	300	4
	nyc-taxi	120	3	590	2	270	3	120	3	100	3	100	3
	rouge-agent-key-hold	70	17	260	2	190	17	190	19	190	4	100	5
	rouge-agent-key-updown	380	17	380	2	370	16	450	5	380	15	300	6
RealTraffic	occupancy-6005	50	6	170	2	600	5	160	2	310	6	300	2
	occupancy-t4013	600	2	590	2	600	3	500	3	100	4	600	2
	speed-6005	260	3	460	2	290	3	260	3	190	4	600	2
	speed-7578	30	6	600	2	150	6	110	3	110	4	100	4
	speed-t4013	250	3	470	2	260	3	250	3	470	3	400	3
	TravelTime-387	490	6	100	2	340	7	100	4	370	7	100	2
	TravelTime-451	120	4	130	4	140	6	100	8	130	6	100	5
RealTweets	Twitter-volume-GOOG	330	3	340	2	240	5	320	3	600	2	400	3
	Twitter-volume-IBM	80	9	280	2	170	4	130	6	130	7	150	5

*Result presented by Maciąg et al. [2].

5.4 Results on the Yahoo dataset

The average values for the precision, recall, average F1-score, BA, and MCC for all types of time series in the Yahoo Webscope dataset obtained by the proposed method are presented in Table 11.

In Table 12, we show the comparison of the obtained average F1-score values for each of the Yahoo Webscope categories for the OeSNN-ABC against state-of-the-art approaches: DeepAnT, EGADS, and Twitter presented in [14]; OeSNN-D presented in [28]; and OeSNN-UAD presented in [2]. The suggested OeSNN-ABC method posed beneficial outcomes in some cases, particularly in the A1 and A2 benchmarks, with scores of 0.832 and 0.881, respectively, greatly exceeding most approaches and exhibiting competitive results to the outcome provided by DeepAnT. However, the findings emphasize the context-dependent nature of anomaly detection techniques. For example, DeepAnT performs well throughout the synthesis time series A2, A3, and A4 standards, but is downgraded in the real time series in A1 with scores of 0.460.

Figure 9 depicts the results of a comparative performance analysis of various deep learning anomaly detection models over four-time series (A1–A4). It is worth noting that the OeSNN-ABC received the highest scores in both A1 benchmark and A2 benchmark, with scores of 0.83 and 0.88, respectively, indicating remarkable performance. It outperformed competitor models such as MadGan and MS Azure (CNN). These results indicate that the approach works well for processing certain forms of time series data. However, the OeSNN-ABC performs less well in the A3 and A4 benchmarks, scoring 0.57 and 0.46, respectively. In these cases, the proposed method either matches or falls behind MadGan, which has a higher A3 score (0.58).

Figure 9

F1-score measure of OeSNN-ABC against deep learning approaches: MadGan [57], MS Azure (CNN) [58], Dense AE [58], RE-ADTS (CV) [59], and AWRM [60]. Source: Created by the authors.

5.4.2 Comparative analysis with other optimization algorithms

In this subsection, OeSNN-ABC is compared with optimization algorithms: PSO, GWO, FPA, WOA, and GS. It can be viewed from the reported result in Table 14 that OeSNN-ABC outperforms other methods in terms of average F1-score for A1 benchmark and A2 benchmark categories. For A3 benchmarks and A4 benchmarks, the FPA algorithm exhibits a better average F1 score than other methods. This is due to FPA’s higher recall value compared to OeSNN-ABC, which positively impacts its F1-score performance.

Table 14

Result of average F1 score evaluation of OeSNN integrated with ABC, PSO, GWO, FPA, WOA, and GS for the YAHOO dataset

Time series	PSO	GWO	FPA	WOA	GS	ABC
A1 benchmark	0.565	0.781	0.766	0.771	0.697	0.832
A2 benchmark	0.418	0.867	0.820	0.835	0.69	0.881
A3 benchmark	0.325	0.534	0.753	0.502	0.409	0.571
A4 benchmark	0.244	0.424	0.765	0.396	0.342	0.469

The bolded numbers indicate the best results.

Figure 10 depicts the evaluation metrics for the proposed method OeSNN-ABC against other optimization algorithms utilized for comparison, including Precision, Recall, average F1-score, BA, and MCC. It is worth noting that OeSNN-ABC produced the best results in terms of the average F1-score for the A1 and A2 benchmarks time series, demonstrating its proficiency in balancing precision and recall across diverse anomaly characteristics. It is notable that FPA outperforms other methods for the A3 and A4 benchmarks time series. This superiority can be due to FPA’s greater recall values, which greatly impact the accuracy.

Figure 10

Evaluation measures of OeSNN-ABC against PSO, GWO, FPA, WOA, and GS optimization algorithms: (a) Precision. (b) Recall. (c) F1-score. (d) BA. (e) MCC. Source: Created by the authors.

5.5 Significance analysis

To further validate the outcomes provided by the proposed OeSNN-ABC method, the t-test is used in this study to verify if a statistically significant difference exists between OeSNN-ABC and other state-of-the-art. The null hypothesis shows that the proposed model and other models’ accuracy do not significantly differ. The null hypothesis is not rejected when the state level is more than 0.05, and it is rejected when the state level is less than 0.05. The p values in Table 15 indicate that the proposed OeSNN-ABC outperforms alternative techniques significantly. As a result, the null hypothesis is rejected at the default 5% significance level.

For additional verification of the obtained results, the Wilcoxon signed-rank test was carried out to assess if the proposed OeSNN-ABC significantly outperformed the PSO, GWO, FPA, WOA, and GS optimization algorithms. Table 16 presents the outcome of the test, which is conducted at a significant level of 5%. From this table, we can conclude that the gaps in the performance are statistically significant for OeSNN-ABC against other optimization algorithms; so, the null hypothesis is rejected at the 5% significance level.

The study emphasizes that the combination of optimizers, in particular the ABC, significantly improves the results of the OeSNN model for unsupervised anomaly detection. By optimizing the OeSNN hyperparameters, window size (Wsize), and anomaly classification factor (ε), the ABC algorithm helps to find the best structure of the model, resulting in a higher F1 score compared to the OeSNN-UAD (unoptimized OeSNN). Moreover, the results show that OeSNN-ABC improves the precision and recall and achieves a balance between accuracy and recall compared to other existing anomaly detection methods, especially on the NAB dataset and specific categories of the Yahoo Webscope dataset (A1 and A2 benchmarks). This implies that the optimization process enables the model to identify a greater number of true anomalies (higher recall) while reducing the number of false positives (higher precision). Furthermore, the results of the statistical tests (t-test and Wilcoxon signed-rank test) demonstrated that the performance differences between the proposed method and unoptimized methods, as well as OeSNN when optimized by other metaheuristic algorithms, were statistically significant.

Although the proposed method has shown a good performance, there are some limitations, which are provided as follows:

• The current study has considered only two hyperparameters: window size (Wsize) and anomaly classification factor (ε).

• The study only used benchmark datasets (NAB and Yahoo Webscope) to validate the performance of the proposed method.

• The evaluations of the proposed method did not consider the measurement of execution time.