Systematic literature review on intrusion detection systems: Research trends, algorithms, methods, datasets, and limitations

2.1 Search strategy

The three databases under consideration (Scopus, IEEE, and WoS) each underwent a thorough bibliographic search for academic papers written in English. All scientific articles production from 2018 to 2023 were included in this search.

This search used a Boolean query to link the keywords trustworthy, using two operator (AND) and (OR) as follows: “Intrusion detection system OR IDS” (AND) “machine learning OR deep learning OR classification algorithms” (AND) “optimisation algorithm OR optimization algorithm”).

2.2 Inclusion and exclusion criteria

The most crucial aspect of this conducted systematic review of the literature is the criteria taken into consideration for the inclusion/selection of studies (Figure 1). And for this, the following parameters were taken into account:

The papers had to be written in English, published in a journal or conference proceedings, and they had to take into account one or more reliable elements.

1. Each of the components required to be considerably connected to reliable ML or DL in order to be integrated into various ML techniques/methods for the intrusion detection domain.

2. Highly relevant articles that were published from 2018 to 2023.

On the other hand, papers beyond the purview of this investigation were omitted based on the following exclusion criteria:

1. Papers that were not from peer reviewed publication forums.

2. Papers that were not written in English.

2.3 Study selection

This study adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement for conducting a systematic literature review. The methodology involved several phases, with the initial phase focusing on the removal of duplicate papers. To achieve this, the titles and abstracts of the contributions were screened using the Mendeley program. All authors participated in this process, leading to the exclusion of numerous unrelated works. Disagreements and discrepancies among the authors were resolved by the corresponding author. The third step entailed a detailed examination of the full texts, during which articles failing to meet the previously established inclusion criteria (as outlined in Section 2.2) were eliminated. Only studies meeting the stipulated requirements were incorporated into this research. The initial search yielded 696 entries, of which 261 originated from Scopus, 175 from IEEE, and 260 from WoS. After removing approximately 217 duplicates and carefully scrutinizing the remaining entries, 393 studies remained. According to the inclusion criteria, studies were deemed relevant and subsequently included in the final collection of publications. The analysis of these gathered articles is subject to various bibliometric techniques, which are discussed in Section 3. The bibliometric analysis conducted in the systematic literature review involved the extraction and analysis of key information from the 393 papers meeting the inclusion criteria. The analysis aimed to understand publication trends, frequently adopted algorithms, datasets, limitations, and future challenges in the field of IDS. Key information extracted from the papers included: (1) Publication trends in the field of IDS from 2018 to 2023, indicating the number of published papers each year. (2) Frequently adopted ML, DL, and optimization algorithms for intrusion detection. (3) Utilized benchmark datasets for evaluating IDS. (4) Limitations and future challenges identified in the reviewed studies. To assess the significance and impact of the identified studies, bibliometric analysis techniques were used to visualize research themes, prominent authors, and frequently studied algorithms like CNN, SVM, XGBoost, and GA. Additionally, the structured taxonomy of the review techniques were categorized into four broad categories: ML, DL, optimization algorithms, and datasets, with detailed sub-categorizations to summarize advancements within each domain. The bibliometric analysis provided insights into the relative effectiveness and limitations of different algorithms and datasets, consolidating scattered developments into an organized synthesis to benefit researchers and practitioners working on ML- and DL-driven IDS.

3.1 Annual scientific production

In the previous 10 years, reliable ML in intrusion detection has advanced. In particular, the annual scientific output depicted in Figure 2 provides an explanation for the emergence of earlier theoretical and practical investigations on reliable ML. The annual scientific output for intrusion detection is depicted in Figure 2. The number of papers published in 2018 and 2019 reached approximately 23 papers, it can be seen that the quantity of publications has significantly expanded in recent years. There was an increase in the number of articles published in 2020 and 2021. In 2022, the number of articles grew even further, reaching a notable high of 138 papers. This pattern persisted in 2023, where 95 papers were published. The increase in research output suggests a growing interest and emphasis on the development and improvement of IDS over the years. Furthermore, specific authors have contributed significantly to this field, with some focusing on optimization and feature selection based on intrusion detection, while others have concentrated on IDS for Internet of Things (IoT) based on DL. These authors have achieved high accuracies in their research, indicating the advancement and effectiveness of the techniques employed in IDS. Overall, the observed publication trends demonstrate a substantial growth in research output in the field of IDS from 2018 to 2023, reflecting an increasing focus on enhancing the reliability and effectiveness of intrusion detection techniques. Figure 3 shows authors’ production over time, where Motwakel [4,5,6,7,8,9,10] published seven papers in 2022 and 2023 and he focused in his research on optimization and feature selection based on intrusion detection. In his research, the highest accuracy he reached was 99.87 using sand paper optimization. As for Al_Qaness [11,12,13,14,15], he published five papers in 2021–2023) and he focused on IDS for IOT based on DL. In his research, he reached the highest accuracy of 99.997 using swarm intelligence optimization. Bacanin [16,17,18,19,20] has published five papers from 2020 to 2023 and he focused on optimization algorithms and feature selection based on intrusion detection in his research, the highest accuracy he reached was 99.6878 using GOA and MPO. Chen [21,22] published five papers from 2020 to 2023 and he focused on intrusion detection using hybrid algorithms, the highest accuracy he reached was 99.44 using COBYLA optimization. Dahou [11,12,13,14,15] published five papers, one paper in 2021, two papers in 2022, and two papers in 2023 and he focused on IOT IDS using DL and optimization in his research, the highest accuracy he reached was 99.997 using swarm intelligence optimization. Fang [23,24,25,26,27] published five papers in 2018, 2020, and 2022 and he focused on optimization algorithm for feature selection of network intrusion detection in his research, the highest accuracy he reached was 97.89 using WOA and OPS optimization. Hilal [4,28,29,30,31] published five in 2022 and 2023, he focused on DL algorithms optimization based on intrusion detection in his research, the highest accuracy he reached was 99.77 using optimization IFSO-FS. Zivkovic [16,17,18,19,20] published five papers, three papers in 2022 and two papers in 2023, he focused on intrusion detection for optimization algorithms and feature selection in his research, the highest accuracy he reached was 99.6878 using GOA and MPO. Dahou [32,33,34,35] published four papers in 2020, 2022, and 2023) and he focused on IDS for optimization algorithms in his research, the highest accuracy he reached was 100 using Artificial organism algorithm (AOA) optimization. In addition to the mentioned authors, Al-Janabi [36,37,38,39] has contributed significantly by publishing four papers, with one in 2020 and three in 2021. His research was focused on IDS, optimization algorithms, and feature selection. Remarkably, his work achieved the highest accuracy of 100% using NTLBO optimization. Table 1 presents a comprehensive overview of the most influential authors in the field. Each of these authors has demonstrated exceptional achievements by reaching the highest accuracy through the utilization of classification and optimization algorithms.

Figure 2

Annual scientific production.

Figure 3

Authors’ production over time.

3.2 Three-field chart

A three-field chart is used to display data with three parameters. In this representation, the left field corresponds to the Research Title (RT), the middle field represents the Journal in which the Research is published or source (SO), and the right field contains the Researcher's Name (RN). Figure 4, is utilized to examine the relationships between these three parameters. According to the study, the RT on the left side is most frequently cited by Scopus, IEEE Xplore (IEEE), and WoS, as observed in the middle field (SO) of Figure 4. Furthermore, among the Research Titles (TI) that focus on the subject of reliable and understandable ML, the Scopus journal stands out as the most prominent. Additionally, as indicated in the corresponding box (TI), when considering all keywords, the journals listed in the middle field (SO) most frequently match the most popular keywords, which include “IEEE Access,” “Sensors,” “Cluster Computing,” “Neural Computing and Application,” and “Soft Computing.”

Figure 4

Three-field plot: left (TI), middle (SO), and right (AU).

3.3 Word cloud

This study has effectively identified the most popular and crucial keywords from previous research using a word cloud. In order to provide a comprehensive overview of these keywords and reorganize the information, Figure 5 presents these essential keywords extracted from the results of previous studies. In Figure 5, the keywords are displayed in different sizes, with the size indicating their frequency in the literature. Larger keywords are more prevalent, while smaller keywords occur less frequently. Based on the term frequency illustrated in Figure 5, “ID,” “DL,” and “ML” emerge as some of the most frequently discussed topics in the field of trustworthy ML, with “DL” having the highest frequency. The image also highlights the significance of “IDS” and “Intrusion Detection System” as critical topics in this area. Additionally, other related terms with relatively high frequencies include “classification,” “optimization,” and “feature selection,” emphasizing the importance of considering these factors when designing and implementing ML systems. Figure 5 also showcases various specific applications of ML across a range of algorithms, such as GA, SVM, and PSO. Furthermore, it includes several methodologies related to ML, including “Internet of Things,” “network security,” and “artificial intelligence.” The word cloud of trustworthy ML articles reveals the broad and diverse nature of this field, covering a wide range of subjects.

Figure 5

Word cloud.

3.4 Co-occurrence

Another method employed in bibliometric analysis is the co-occurrence network, which involves studying common words in earlier research. This semantic network offers valuable insights into the conceptual framework of a specialized field, benefiting practitioners, policymakers, and academics. Figure 6 presents information specifically related to a co-occurrence network based on the titles of reliable ML articles.

Figure 6

Co-occurrence network.

The network is composed of nodes, representing individual words in the titles, and edges connecting the nodes indicate how frequently these words appear together in a title. Figure 6 displays various nodes, including the clusters to which they belong and their proximity, a metric for gauging how well-connected a node is to other nodes in the network. Evidently, the nodes are divided into eight distinct clusters, with each cluster comprising words associated with a specific theme or idea related to trustworthy ML. For instance, Cluster 1 features terms like “support vector machine,” “DT,” “Extreme learning machine,” “cyber security,” “genetic algorithm,” and “bat algorithm.” These terms suggest a connection to the establishment of dependable ML systems in optimization. Cluster 2, on the other hand, includes words like “neural network,” “artificial intelligence,” “feature extraction,” “random forest,” “anomaly detection,” and “network security,” signifying a focus on IDS. Similarly, other clusters are linked to subjects such as clustering, the IoT, the NSL-KDD dataset, ML, and IDS. The closeness of a node within the network measures its centrality, and nodes with higher closeness values are more central to the topic of trustworthy ML, signifying their importance within the network. This figure offers an overview of the relationships between different concepts and words associated with trustworthy ML, as derived from the titles of papers in the field. This information is valuable for understanding the current state of research in this area and for identifying areas where further investigation is warranted.

4.1 ML

Arthur Samuel originally described ML in 1959 as a branch of research that allows computers to learn without requiring them to first be programmed. This section describes network ID strategies with a focus on ML algorithms used to create security tools. In recent years, ML has gained increasing importance in IDS for computer networks [40,41,42]. The foundation of this lies in the model for training and prediction, which has the capability to quickly identify both attacks and typical cases [32]. The feature selection process can be considered as data preprocessing for ML algorithms. Intrusion detection can involve two types of classification: two-class, where intrusions are detected based on class labels, and multi-class, which categorizes attacks into different classes. ML techniques like Random forest (RF), SVM, ELM, and Naive Bayes classifiers can be applied in this field, as well as methods such as Self-Organizing Maps, Fuzzy clustering, and K-Means clustering. Figure 8 show the classification of ML algorithms.

Figure 8

ML algorithms.

In the reviewed literature, several ML and DL algorithms have emerged as the most frequently used in intrusion detection research. Specifically, CNN, SVM, DTs, and GA have been prominently featured in the analyzed studies. Comparatively, these algorithms have demonstrated varying levels of popularity and effectiveness in the context of intrusion detection: (1) CNN: CNN has gained significant popularity and has been widely utilized in intrusion detection research due to its effectiveness in learning hierarchical representations of data, particularly in image-based intrusion detection scenarios. (2) SVM: SVM has also been frequently used and is known for its effectiveness in binary classification tasks, making it a popular choice for intrusion detection applications. (3) DTs: DTs have been commonly employed for their interpretability and ease of understanding, making them popular in certain intrusion detection contexts, especially when explainability is a priority. (4) GA: While GAs have been utilized, they may not be as prevalent as CNN, SVM, and DT in intrusion detection research. However, they offer the advantage of optimization and search capabilities, which can be beneficial in certain scenarios. In terms of effectiveness, the reviewed literature may provide insights into the comparative performance of these algorithms in specific intrusion detection contexts, such as their accuracy, precision, recall, and F1 scores. Additionally, the specific datasets and features used in the studies may influence the relative effectiveness of these algorithms. Overall, while CNN, SVM, and DT have emerged as popular and effective choices in intrusion detection research, the comparative effectiveness of these algorithms may vary depending on the specific context, dataset, and evaluation metrics used in the reviewed studies.

4.2 DL

DL, a subcategory of ML, consists of multiple hidden layers and finds applications in various domains, including image processing and natural language processing. It excels in understanding the meaning of vast multidimensional data, performing feature selection, classification, and uncovering data correlations, particularly in speech recognition and language processing [69] (Figure 9).

Figure 9

Deep learning algorithms.

The utilization of ML and DL techniques has significantly evolved in the development of IDS from 2018 to 2023. A systematic review of 393 studies revealed that ML and DL techniques have demonstrated significant potential in enhancing the reliability and effectiveness of IDS. The review identified frequently used algorithms, with CNN, SVM, DTs, and GA emerging as the top methods. Tables 2 and 3 and and 4 in the review indicate that SVM, DT, and ELM algorithms exhibit superior performance, particularly with the KDD Cup 1999 and NF-Bot datasets, both for multi-class and binary classification, as assessed by accuracy. In the realm of DL algorithms, Table 8 in the review showcases improved outcomes with the CNN algorithm and the NLS-KDD L dataset compared to Table 9, which demonstrates lower results with the RNN algorithm and the CICIDS2017 dataset, once again, gauged by accuracy. Overall, the utilization of ML and DL techniques has evolved significantly in the development of IDS, with CNN, SVM, DT, and GA emerging as the top methods. These techniques have demonstrated significant potential in enhancing the reliability and effectiveness of IDS.

4.3 Optimization (OP) algorithms

OP algorithms [76] is often the most efficient and accurate method for solving problems, although its definition can vary by context. In mathematics, it involves exploring the behavior of a problem by adjusting values within a specified range to either minimize or maximize a function. Optimization processes hold a significant role in DL, where various optimization functions are employed to minimize or maximize error functions. These functions have been developed in diverse environments. Figure 10 presents the most important OP algorithms used in research.

Figure 10

Optimization algorithms.

OP algorithms have been extensively integrated into intrusion detection research, with several studies focusing on OP algorithms for feature selection and DL algorithms based on intrusion detection. The review identified GA, ACO, and GWO as significant OP algorithms, consistently delivering high results across Tables 10, 12, and 14 . For instance, Fang published five papers between 2018 and 2022, focusing on OP algorithms for feature selection of network intrusion detection. The highest accuracy he reached is 97.89 using WOA and OPS optimization. Similarly, Zivkovic published five papers between 2018 and 2023, focusing on intrusion detection for OP algorithms and feature selection. The highest he reached is 99.6878 using GOA, MPO optimization. The integration of OP algorithms has contributed significantly to the overall performance of IDS. For instance, the KDD Cup 1999 dataset, which is one of the most widely utilized in the field of intrusion detection, has been used in comparison to the PSO algorithm, with Table 11 indicating a marginal difference of 0.1. Overall, OP algorithms have been extensively integrated into intrusion detection research, with GA, ACO, and GWO emerging as significant OP algorithms. These algorithms have contributed significantly to the overall performance of IDS, enhancing their reliability and effectiveness.

4.4 Dataset

The datasets encompass fields containing both unprocessed and processed data extracted from underlying network traffic [90]. These data are typically generated through studies aimed at identifying network intrusions. An intentional effort is made to manipulate the data, creating adversarial examples capable of deceiving classifiers and detection systems. When creating adversarial instances that alter the source data in network security applications, caution is essential, as highlighted by [90]. The most prominent datasets utilized in research include KDD Cup99, NSL-KDD, CICIDS 2017, UNSW-NB15, AWID, Kaggle, and TON-IOT. Table 15 provides an overview of the most crucial datasets commonly used in the field of intrusion detection [37–39,91–97].

The most commonly used datasets in IDS are as follows:

1. KDD Cup 1999: This dataset stands as one of the most widely utilized in the field of intrusion detection. It comprises a substantial and diverse collection of network traffic data, encompassing potential attacks.

2. NSL-KDD: An enhanced iteration of the KDD Cup 1999 dataset, it offers a more demanding and realistic environment for testing intrusion detection models.

3. UNSW-NB15: This dataset consists of samples of network traffic extracted from real-world internet environments, providing a formidable challenge for detecting advanced attacks.

4. CICIDS2017: This dataset encompasses a diverse range of data reflecting different attack scenarios, serving as an invaluable resource for evaluating the performance of intrusion detection models.

The reviewed studies have frequently utilized several benchmark datasets for evaluating IDS. The most commonly used datasets include: (1) KDD Cup 1999: This dataset is one of the most widely utilized in the field of intrusion detection. It comprises a substantial and diverse collection of network traffic data, encompassing potential attacks. (2) NSL-KDD: An enhanced iteration of the KDD Cup 1999 dataset, it offers a more demanding and realistic environment for testing intrusion detection models. (3) UNSW-NB15: This dataset consists of samples of network traffic extracted from real-world internet environments, providing a formidable challenge for detecting advanced attacks. (4) CICIDS2017: This dataset encompasses a diverse range of data reflecting different attack scenarios, serving as an invaluable resource for evaluating the performance of intrusion detection models. The utilization of these benchmark datasets has influenced the comparability of research outcomes by providing a standardized basis for evaluating the performance of IDS. Researchers can compare the effectiveness of different algorithms and techniques using these commonly accepted benchmark datasets, thereby facilitating the assessment of the reliability and generalizability of intrusion detection models.