Applications of integrating artificial intelligence and big data: A comprehensive analysis

1 Introduction

Artificial intelligence (AI) represents a paradigm in which computers exhibit intelligence similar to that of human and animal cognition. On the other hand, big data encompasses vast amounts of data that surpass the processing capabilities of traditional data systems, highlighting new approaches to data collection, analysis, and management. The combination of AI and big data is revolutionizing numerous sectors, including energy [1], security [2], health [3], agriculture [4], biotechnology [5], and education [6]. By merging data analytics and machine learning, this integration enhances operational efficiency, improves customer experiences, and drives innovation. It expands the range of possibilities and prompts us to reconsider the role of technology in decision-making and operational workflows.

Despite the significant advancements brought by AI and big data, several issues persist in the context of their integration. In terms of data quality, poor quality can lead to incorrect insights, and integrating data from various sources remains a significant technical hurdle. As for the computational resources, handling the vast volume of data requires substantial computational resources. Such issues cause negative implications for organizations to efficiently process and analyze large datasets.

Technically, AI systems can be biased according to the training data. Hence, ensuring fairness and mitigating bias in AI applications is an ongoing challenge. The use of large datasets involves sensitive personal information, raising concerns about privacy and security. Hence, ensuring data is handled in compliance with regulations is crucial. The deployment of AI and big data technologies also brings ethical and social implications, such as job displacement and impacts on social fairness.

As we explore the numerous applications of AI and big data, it becomes clear that their influence goes beyond traditional frameworks. From using predictive analytics in healthcare to creating personalized retail experiences, the combination of these technologies is reshaping decision-making and efficiency [7]. This research article aims to examine the roles of AI and big data in different industries, highlighting their potential for transformation and the opportunities they bring. The versatility of AI and big data allows for their application in a range of industries, each with its own unique challenges and opportunities for innovation.

Big data provides valuable insights into patterns and trends within large datasets. We recognize the important role of AI in improving big data analytics through parallel processing, which enables faster and more cost-effective analysis [8]. This shift from traditional analytical methods emphasizes the need for advanced tools capable of handling the complexity and scale of big data. Despite these advancements, the challenge of processing diverse and extensive datasets remains, requiring sophisticated AI interventions to reduce human errors and misconceptions in data analysis [9].

Our research is designed to explore the realm of big data and AI applications, with a specific focus on the importance of high-quality data. By carefully examining a wide range of original research databases, such as Web of Science, Scopus, and IEEE, and analyzing 105 papers, our study primarily investigates how the integration of big data and AI is shaping various sectors. By using bibliographic coupling, we identify clusters of interconnected research based on shared references. This method allows us to examine the challenges, innovations, and potential solutions within the domains of big data and AI. Our analysis aims to draw connections, highlight sector-specific challenges, and emphasize the global demand for big data and AI technologies. Through a cross-sector investigation, we aim to showcase the practical usefulness and transformative impact of these technologies across different fields.

Our proposed review aims to systematically explore the wide range of AI and big data applications across different industries, providing a comprehensive analysis of their transformative potential. However, we aim to address the following:

1. Identify and map out the diverse applications of AI and big data integration across industries, providing a wider view of their impact.

2. Analyze key innovations within these integrations, highlighting how AI enhances big data analytics to drive significant advancements.

3. Evaluate the challenges associated with integrating these technologies and propose potential solutions.

4. Identify gaps in existing research to provide a foundation for future studies to build upon.

Our findings reveal better understandings on how AI and big data integration can lead to notable advancements in sectors such as maritime computing and networking, healthcare, education, energy, and agriculture. These results mark a significant shift toward more sophisticated data-driven decision-making and operational processes.

The methodology of our research is designed to ensure a comprehensive and systematic review of the literature, enhancing the reproducibility of our research. The following steps were followed while reviewing and studying the 105 peer-reviewed papers.

2 Literature review

Big data and AI are essential in making decisions based on data and have gained a lot of attention in recent years. Munim et al. [10] conducted a bibliometric review that specifically looked at the use of big data and AI technologies in the maritime sector. They gathered data from the Web of Science website and analyzed it using the Bibliometric method in R software. Through bibliographic coupling, they found four main areas of research: digital transformation in the maritime sector, the application of big data from automatic identification systems, energy management, and predictive analytics. Figure 1 illustrates the main applications clusters of AI and bid data in maritime.

Figure 1

The big data and AI applications clusters and their respective subclusters in maritime [10].

Hussain and Manhas [11] emphasized the importance of top management’s commitment to facilitating improvements in procedures, the active involvement of organizational personnel, data availability, and collaboration with universities and technology partners specializing in machine learning systems and big data technology. They highlighted the benefits of employing AI in data analysis and emphasized the projected growth of the AI industry and its significance in the future of enterprises.

Yang [12] studied the application of AI and big data in computer networking technology. The article examines the different features of big data and the technologies used for its analysis. It also compares various machine learning frameworks. Importantly, the article emphasizes the benefits of using an AI platform in a computer network to handle large amounts of data while also ensuring data confidentiality. It discusses the use of AI in network security management, specifically in improving intrusion prevention, antispam, and firewall technologies.

According to Huerta et al. [13], big data has both challenged and revolutionized major markets, presenting unique opportunities for innovation. To fully harness the potential of AI for data-driven exploration, the authors emphasized the importance of establishing a robust mathematical framework, fostering interdisciplinary collaboration to integrate various frameworks, identifying correlations between AI data and models, and deploying AI, representations, and information on open-source platforms. The study addressed various software and hardware challenges in the convergence of AI, offering recommendations to enhance the effectiveness of using HPC tools for AI research.

Masood and Mokmin [14] examined the use of the case-based reasoning (CBR) AI algorithm, which is commonly employed in big data applications. They evaluated its suitability and effectiveness in real-time scenarios and subsequently created an intelligent tutoring framework based on CBR. This framework offers students personalized recommendations for learning materials based on real-time data from their profiles. The program was extensively evaluated in a real-world setting and demonstrated the ability to accurately assess case similarities. Students who received personalized learning resources exhibited improved performance compared to those who received nonpersonalized instruction.

On the other hand, Khan and Alotaibi [15] conducted a comprehensive study by analyzing numerous scientific papers collected from various sources, including science databases and Google Scholar. They identified key terms such as m-health, big data, and AI within these papers. The articles covered a wide range of topics related to m-health and its evolution over time. This article explores the various applications of AI and big data in m-health, emphasizing the development of algorithms for data collection and improving adaptability. In addition, the article proposes an m-health infrastructure that focuses on AI and big data analytics, utilizing technologies such as the Internet of things (IoT), web servers, and principles of big data storage and statistical analysis tools. Table 1 refers to the applications of mobile sensors in healthcare.

According to Pinto dos Santos and Baeßler [16], the inherent complexity of information stored in electronic health archives and patient information systems presents a significant challenge to the development of AI technologies for healthcare and radiology. The authors discovered that the quality of the data used to train and validate algorithms is crucial for the progress of AI in radiology and healthcare. They emphasized the prevalence of unstructured narrative texts in radiology documentation and emphasized the significance of standardized, organized reports to facilitate the integration and performance of big data and AI technologies, especially in tasks such as pattern or disease recognition.

The research presented by Dlamini et al. [17] highlighted the significant impact of next-generation sequencing (NGS) platforms in the field of oncology. These platforms have completely transformed the way novel biomarkers that affect drug efficacy and therapeutic targets are identified, thereby improving treatment systems. NGS has proven to be extremely valuable in risk prediction, early disease detection, diagnosis, medical imaging, and various other areas, generating vast datasets that require sophisticated bioinformatics tools for the analysis and retrieval of clinically relevant information. The authors advocate for the integration of AI into NGS to analyze these datasets and produce high-resolution images, ultimately enhancing cancer detection, diagnosis accuracy, and precision in treatment. Generally, the research highlighted the main general areas that AI and big data covers in healthcare sector as illustrated in Figure 2.

Figure 2

Overview of AI and big data utilization in different healthcare sectors.

Hariri et al. [18] examined the existence of variability in various stages of big data learning, such as data acquisition, definition volatility, and multimodality, which could undermine the effectiveness and accuracy of results. They explored different approaches and methods, such as Bayesian theory, belief function theory, and probability theory, to quantify uncertainty in big data. They also discussed different analysis methods for big data, such as parallelization, sampling, and granular computation, which can assist in precise and efficient processing. Furthermore, they investigated AI techniques like deep learning and transfer learning, which have the potential to enhance data analysis and transfer learning processes.

Arora et al. [19] discussed the complexities of collecting credit card payments and the technologies involved in this process. They offered a detailed overview of fraudulent practices in credit card transactions and developed a predictive framework to evaluate the probability of credit card delinquency. Machine learning algorithms assist in identifying deception in large datasets by categorizing transactions as either normal or fraudulent. If a fraudulent transaction occurs, a notification can be sent to the appropriate financial institution to void the transaction. In addition, OECD [20] identified the significant potential of big data and AI in risk classification, a crucial process in the insurance sector. They outlined various policy domains in which regulators in the insurance industry could consider taking action concerning big data and AI. They proposed that the insurance industry should be empowered to utilize big data and AI, keeping up with the latest technological advancements. They also suggested monitoring technological progress and implementing it in the insurance sector, while promoting international cooperation.

The potential of AI technology in various aspects of big data analytics is significant. Shen [21] highlighted the potential application of AI for big data analytics within computer networking technology. One key benefit of AI is its ability to automatically and in real time protect the integrity and reliability of big data used for analytics. Examples of such AI applications in big data networking include autonomous intrusion detection, artificial immunity, data information management, and AI-enabled firewalls for networking. These measures ensure that the underlying networking technology supporting the entire process of big data collection and storage remains secure and reliable. Shen also pointed to the increasing prevalence of the IoT and cloud computing as emerging technologies, which lead to a higher volume of data production and storage, thus emphasizing the importance of maintaining data integrity for reliable big data insights.

The healthcare industry is known for being an early adopter of new technologies, using each advancement to improve patient care outcomes. Benke and Benke [22] examined how AI and big data technologies contribute to the field of public health. According to their research, AI, when combined with access to vast healthcare data and patient imaging, has the potential to accelerate the diagnosis process for early and rapid detection of various health conditions. The continuous learning ability of AI ensures more reliable diagnosis processes and establishes parameters that can only be achieved with the help of big data. However, the researchers also recognize the ethical concerns surrounding the use of big data and autonomous technologies in diagnosing and treating patients. Nonetheless, the integration of AI and big data holds great promise in public health, allowing for faster identification of disease outbreaks and improved accuracy in diagnosis.

Oliveira [23] investigated the use and potential of big data technologies and AI in the field of biotechnology. The authors highlighted the synergistic use of AI and big data technology to support various biotechnology applications, such as drug discovery, drug recycling, drug safety, functional and structural genomics, and pharmacogenetics. In addition, Oliveira found that AI and big data can enhance three aspects of data management in biotechnology operations: process optimization, data integration, and data exploitation. The research identified a growing incorporation of new information technology in medical science, improved data sharing among researchers, and an increasing number of biotechnology journal databases, which collectively provide a vast global dataset for biotechnology research. Furthermore, the study suggests that AI with big data can significantly impact experiment design in biotechnology, as machine learning can offer unprecedented insights. For example, machine learning and AI can improve cell classification and other biological classifications, thereby enhancing experiment design processes.

Gao et al. [24] evaluated the e-learning and e-education industry, proposing a potential enhancement through a big data-driven approach. Their research focuses on using key technologies in AI and big data to identify applications in the education sector. The study suggests that AI and big data technology can greatly improve online education systems, addressing concerns about productivity levels in the industry. To achieve this, the researchers advocate for incorporating technologies such as man–machine cooperation, index update, edge computing, and fuzzy systems to develop intelligent algorithms in e-learning platforms. By utilizing machine learning and AI tools, these algorithms can continually evolve, enhancing the overall e-learning experience. Specifically, they propose using these algorithms for cross-dimensional data mining to understand students’ learning objectives, pace, areas of difficulty, and content effectiveness. In addition, the research suggests developing algorithmic interventions to assist students in areas where they struggle. For example, in language learning classes, a factorization machine can intelligently recommend movies in the target language, tailored to the student’s proficiency level, to effectively supplement their learning experience.

Teófilo and Antonio [25] analyzed the potential application of technology for accurate text analytics, emphasizing the crucial convergence of big data technology and AI in this field. Their research indicates that the combination of AI and big data analytics is essential for the advancement of natural language processing (NLP). NLP, a significant application of AI in big data, allows computer systems to understand human language and interpret the correct meaning of written or spoken text, taking into account human nuances in speech. This application is widely used to effectively understand historical texts and develop intelligent assistants capable of genuine interaction in human language.

On the other hand, Li et al. [26] investigated big data analysis (BDA) using extensive data from IoT in smart cities, with the goal of improving both the efficiency and security of data processing. By employing deep learning algorithms alongside BDA, the study introduces a distributed parallelism strategy for convolutional neural networks. The IoT–BDA system for smart city digital twins (DTs) is shaped by DT and multi-hop transmission technology based on DL. Performance simulations reveal that energy efficiency in model data transmission correlates with increasing collected energy in an inverted U-shaped relationship. Optimal power diversion factors are crucial for IoT–BDA’s signal transmission energy efficiency. The system’s prediction accuracy surpasses other DL models by at least 2.24%, achieving 97.80%. Data transmission analysis indicates minimal delays at millisecond levels under specific conditions. In conclusion, enhancing smart cities’ IoT–BDA systems with deep learning minimizes data delays, enhances forecast accuracy, and provides tangible benefits. Future research should explore deep learning’s potential for real-time analytics and refine IoT transmission schemes for large-scale communication in smart cities.

With an increasing focus on environmentally friendly energy sources, energy harvesting emerges as a sustainable solution that extends the lifespan of IoT devices. Ouafiq et al. [27] presented the latest advancements in IoT energy harvesting and its applications in smart agriculture. The authors introduce an integrated architecture that incorporates big data technologies, IoT elements, and knowledge systems to revolutionize farm infrastructure. Addressing the challenges of processing small files in big data settings without computational delays, their solution is based on a structured big data framework that features a data lake abstraction layer, ensuring data quality through strategic migration. Their analysis compares various machine learning algorithms for smart farming analytics, with the ID3 algorithm demonstrating superior accuracy. They propose the smart farming–oriented big data architecture, a multilayered platform that enhances farming data management, flexibility, and speed. Looking ahead, they aim to refine data ingestion workflows and prioritize agricultural sustainability, energy efficiency, and increased productivity.

Dubey et al. [28] investigated the impact of AI-driven big data analytics capability (AI-BDAC) on humanitarian relief efforts. As organizations strive to balance agility and resilience during crises, there is an ongoing debate in the academic realm regarding the suitability of traditional commercial supply chain theories, such as resource-based or dynamic capability views, for understanding humanitarian supply chains. Recognizing this research gap and considering the skepticism surrounding the adoption of AI-BDAC in humanitarian decision-making, this study adopts a practice-based view. This perspective examines practices that are common and replicable in enhancing performance. Using partial least squares analysis on 171 responses from international nongovernmental organizations, our findings reveal that AI-BDAC significantly influences the agility, resilience, and overall performance of humanitarian supply chains. Furthermore, it reduces information complexity between agility, resilience, and performance. This research underscores the distinctions between commercial and humanitarian supply chains, suggesting that traditional commercial-focused theoretical lenses might not adequately capture the nuances of the latter. The practice-based view emerges as a more appropriate lens, shedding light on the intricate interplay between AI analytics, humanitarian chain agility, and resilience. The results emphasize the need for further exploration into the differences between strategic resources and practices in humanitarian scenarios, promoting a deeper understanding of the complex challenges faced by humanitarian supply chains.

Popov et al. [29] explored the use of big data analytics and machine learning to assess the quality of railway tracks, specifically focusing on high-speed UK lines with more than 15 years of track geometry data. Traditional methods prove inadequate for analyzing such extensive datasets. Therefore, an artificial neural network (ANN) was used to process the data, effectively identifying track segments that showed improvement or deterioration between inspections. The ANN achieved an impressive ∼98% training accuracy and reliably identified track segments with significant measurement changes. The findings revealed that large-scale tamping maintenance may unintentionally include sections of healthy track, potentially reducing their lifespan. Surprisingly, around 50% of the tamped track did not show significant improvement based on various geometric features. On the other hand, local tamping with a sprinter tamper proved more effective in addressing defects. The study revealed that high-output tamping, which covers longer track sections, was less efficient than local tamping targeting specific defects. Large-scale tamping may have a negative impact on high-quality tracks, resulting in no noticeable improvements or even potential deterioration. This research highlights the potential of AI in improving the efficiency of railway maintenance.

Stergiou and Psannis discussed the challenges of managing and analyzing big data from industrial IoT in cloud environments [30]. They explore machine learning in cloud infrastructures, AI techniques for big data analytics in the cloud, and federated learning cloud systems. One of their main focuses is reinforcement learning, which improves the energy efficiency of resource allocation in cloud data centers. They propose an architecture called the energy-efficient industrial IoT-based big data management framework (EEIBDM), which integrates features from different cloud providers and situates it outside each user in the cloud. By combining IoT data with reinforcement and federated learning, they create a DT – a virtual representation of industrial IoT data, such as machine statistics and room temperatures. In addition, they present an algorithm to measure the energy consumption of the infrastructure using the EEIBDM framework. This study utilizes reinforcement learning to optimize cloud interactions for resource allocation. Their algorithm estimates CPU energy consumption through EEIBDM evaluation. Future plans involve improving the system’s security and privacy, to make it suitable for energy-efficient and secure data management in hospitals, schools, and legal repositories, using innovative reinforcement and federated learning techniques.

Wind energy forecasting plays a crucial role in improving energy efficiency, and the integration of big data and AI holds great promise in this field. Despite the abundance of literature, there is a lack of comprehensive overviews. Zhao et al. [31] aimed to fill this gap by providing an analysis of big data and AI applications in wind energy forecasting over the past two decades. Through literature reviews and scientometrics, the article categorizes and organizes various types of data, analysis techniques, and forecasting methods. By examining current research trends, the study identifies key developments and future directions in wind energy forecasting using big data and AI. The primary objective is to identify opportunities, challenges, and the evolving trajectory of big data and AI in wind energy. The analysis identifies four main forecasting models in wind energy forecasting, each with its unique requirements and benefits. The study emphasizes intelligent and hybrid methodologies that combine big data analytics to achieve superior forecasting results. Overall, this work offers a comprehensive view of the past, present, and potential future of big data and AI applications in wind energy forecasting, detailing processes such as data sourcing, preprocessing, feature engineering, and algorithmic enhancements.

Hassani and Emmanuel [32] explored the transformative role of ChatGPT in data science, discussing its potential benefits and challenges. ChatGPT assists data scientists by automating various stages of the workflow, including data preprocessing, model training, and result interpretation. It provides valuable insights by analyzing unstructured data and has a flexible architecture that can handle diverse language tasks and generate synthetic data. However, concerns about bias, plagiarism, and the interpretability of ChatGPT outputs are significant. While ChatGPT streamlines many processes, its results may be difficult to interpret, which could hinder decision-making in certain applications. Nevertheless, the general consensus suggests that its advantages outweigh these concerns. ChatGPT shows promise in enhancing data science productivity and accuracy, and it is poised to become an indispensable tool for intelligence augmentation in the field. Embracing ChatGPT requires not only leveraging its capabilities but also understanding its ethical implications, ensuring that its usage aligns with integrity and awareness of its potential strengths and pitfalls. In summary, ChatGPT has a profound impact on data science, and its ongoing development indicates a promising future where data scientists can optimize their tasks while remaining mindful of ethical considerations.

Agustí and Manuel [33] conducted a bibliometric analysis to track the evolution and trajectory of AI and big data. Initially focused on predicting and classifying companies based on risks, the research direction has shifted toward exploring business intelligence opportunities in the face of increased company digitalization. Present trends emphasize stakeholder-centric perspectives, including internal accounting information, decision-making processes, and external disclosures such as those found on social media platforms. Emerging areas like blockchain are also expected to receive more attention in the near future. However, these technological advancements signal significant transformations in the accounting and auditing sectors. As these technologies become integrated into these fields, professionals will require a different skill set than what is currently emphasized in business schools. The future accountant may resemble more of a data analyst, while auditors will increasingly rely on computer-assisted tools, highlighting the need for a broader understanding of these technologies among practitioners.

AI and big data are transforming people’s lifestyles and cognitive processes, leading to a shift toward online education in college ideological and political education (IPE). Du et al. [34] introduce an AI-driven online education algorithm designed to improve the management and delivery of IPE in universities. The back propagation neural network model is used to extract student learning features, overcoming the limitations of traditional feature extraction in machine learning. Combined with a human behavior recognition model, this allows teachers to gain real-time insights into students’ engagement with IPE through their listening devices. Experimental evaluations of the proposed algorithm resulted in recognition accuracies ranging from 95.61 to 100% for various student samples. Compared to similar recognition algorithms, this model demonstrates an accuracy and precision of 97.83 and 97.82%, respectively. Grounded in AI and big data, the human behavior recognition model provides real-time data on students’ learning statuses, significantly enhancing online teaching management. Overall, this study highlights the potential of integrating AI and big data to reshape and optimize online IPE teaching methodologies and management practices.

3 Applications of AI and big data integration

The current state of the art examines the crucial roles played by big data and AI in various sectors, emphasizing their importance in decision-making processes driven by data. One sector that stands out for experiencing significant advancements thanks to these technologies is the maritime industry. In this industry, several areas have undergone remarkable transformations, such as digital overhauls, the utilization of data from automatic identification systems, efficient energy management, and the implementation of predictive analytics. However, the integration of machine learning techniques requires comprehensive strategies. These strategies encompass firm commitment from top management, active participation across organizational levels, ensuring data accessibility, and fostering collaborations with specialized entities in machine learning and big data.

Furthermore, the healthcare sector emerges as a primary beneficiary of technological innovations. By integrating AI with voluminous healthcare data, faster diagnostic processes become possible, facilitating the early detection of various medical conditions. The self-learning capabilities of AI further enhance this process, refining diagnostic parameters based on insights derived from big data. In the biotechnology sector, the combined capabilities of AI and big data have the potential to revolutionize experiment design, offering unprecedented insights. Finally, AI demonstrates significant potential in big data analytics, particularly in ensuring real-time data integrity. This is evident in applications such as intrusion detection, AI-powered firewalls, and overall data management, especially with the proliferation of the IoT and cloud computing. Figure 3 provides a summary of the diverse applications resulting from the integration of AI and big data.

Figure 3

Applications of AI and big data integration.

The integration of big data and AI technologies across various sectors represents a substantial move toward more efficient, accurate, and secure operations. In this section, we explore the application and advantages of integrating these technologies in different domains, identifying crucial areas of impact and future potential.

In the maritime sector, the introduction of digital transformation through big data and AI technologies represents a significant advancement toward optimizing operations. Key areas of research in this field include the utilization of big data from automatic identification systems, energy management, and predictive analytics. These factors collectively enhance safety in navigation, operational efficiency, and environmental sustainability within the maritime industry. At the enterprise level, the strategic importance of AI in data analysis is emphasized, highlighting its role in improving procedures and the need for commitment from top management. The successful integration of machine learning technologies requires active participation from organizational personnel, data availability, and strong collaborations with universities and technology partners. This comprehensive approach not only facilitates current operational improvements but also prepares for the projected development and future significance of AI in enterprise settings.

In the field of computing and networking, the integration of AI and big data plays a crucial role in analyzing network attributes, employing advanced technologies for data analysis, and fortifying network security. AI contributes to managing network security through functions such as intrusion prevention, antispam measures, and firewall technologies. This ensures the integrity and security of data flows across networks. Meanwhile, big data learning focuses on addressing the variability in data phases, methodologies for measuring uncertainty, and the application of AI techniques such as deep learning and transfer learning to enhance data analysis. This domain highlights the importance of employing advanced analytical techniques to handle the complexities of big data, thereby improving the quality and insights derived from extensive datasets. Moreover, data science has made significant advancements in enhancing data processing efficiency. AI algorithms like ChatGPT have revolutionized customer service by offering automated yet personalized responses. In addition, text analysis tools facilitate the extraction of actionable insights from unstructured data, which can inform business strategies across various sectors.

In health research, the use of AI and big data to detect cancer cells highlights the important role of technology in improving the accuracy of cancer detection. This accuracy in medical diagnostics indicates a larger trend towards employing AI for more precise and early disease detection, which has the potential to improve patient outcomes. On the other hand, biotechnology benefits from big data in drug discovery by efficiently searching through extensive compound libraries to find potential drug candidates, a task that would otherwise be time-consuming and costly.

In the field of education, AI and big data are transforming traditional learning environments into dynamic and personalized experiences. By incorporating technologies such as machine collaboration, edge computing, and fuzzy systems, AI and big data can strengthen e-learning systems by creating intelligent algorithms that enhance the learning process. These advancements allow for advanced data mining across multiple dimensions, enabling educators to gain deeper insights into student learning patterns, identify areas of difficulty, and customize content accordingly. Furthermore, AI algorithms have proven to be effective in building intelligent tutoring frameworks that provide personalized learning materials based on real-time student data. This approach not only improves learning outcomes but also addresses the individual needs of students, demonstrating the transformative potential of AI and big data in educational methods and results.

The use of AI and big data in the energy sector for calculating energy efficiency and making predictions has significant consequences. By improving energy efficiency, costs can be lowered and the environmental impact can be mitigated. Furthermore, accurate predictions assist in effectively managing the balance between energy supply and demand, which is crucial for ensuring the stability of the grid. Considering that every industry depends on energy, advancements in this sector can have a domino effect, potentially amplifying the advantages of AI and big data in different sectors.

In the agricultural sector, the integration of AI, big data, and IoT technologies is bringing about the era of “smart farming.” This ground-breaking approach greatly improves productivity, sustainability, and operational efficiency. These advancements strengthen data management capabilities, allowing for more precise analysis and better-informed decision-making. As a result, higher crop yields, more efficient use of water resources, and a reduction in environmental impact can be achieved. The inclusion of machine learning algorithms within this framework has shown significant improvement in the accuracy of analytics related to smart farming activities. This highlights the transformative potential of these advanced technologies in shaping the future of agriculture, promoting greater efficiency and sustainability.

To provide a clearer picture of the practical applications and outcomes of AI and big data integration, we present several case studies across different sectors:

• AI in cancer detection (Case study: IBM Watson for Oncology)

  IBM Watson for Oncology utilizes AI to provide oncologists with evidence-based treatment options [35]. By analyzing huge amounts of medical literature and patient data, Watson helps identify personalized treatment plans for cancer patients. A study conducted at Manipal Hospitals in India demonstrated that Watson’s recommendations were matched with the oncologists’ treatment plans in over 90% of the cases, showing the practical effectiveness of AI in enhancing cancer treatment.

• Precision farming (Case study: John Deere’s Precision Agriculture)

  John Deere has implemented AI and big data technologies in its precision agriculture solutions [36]. By using sensors and GPS technology, John Deere’s equipment can gather data on soil conditions, crop health, and weather patterns. These data were analyzed to optimize planting, fertilization, and irrigation processes. Farmers using these technologies have reported significant increases in crop yields and reductions in resource usage, demonstrating the benefits of AI and big data in agriculture.

• Fraud detection (Case study: JPMorgan Chase’s AI Fraud Detection System)

  JPMorgan Chase has developed an AI-driven fraud detection system called COiN (Contract Intelligence) [37]. This system uses machine learning algorithms to analyze vast amounts of transaction data in real time, identifying suspicious activities and preventing fraudulent transactions. COiN has significantly reduced the time required to review documents and has improved the accuracy of fraud detection, resulting in substantial cost saving.

• Personalized customer experience (Case study: Amazon’s Recommendation System)

  Amazon utilizes AI to power its recommendation system, which suggests products to customers based on their browsing and purchasing history [38]. This system analyzes massive datasets to identify patterns and preferences, offering personalized recommendations that enhance the customer shopping experience. As a result, Amazon has seen increased customer engagement and higher sales, illustrating the effectiveness of AI in driving retail success.

In conclusion, these case studies provide concrete examples of how AI and big data integration can lead to significant advancements and practical benefits across various industries. While the benefits of integrating AI and big data differ among various sectors, they share common themes of efficiency, personalization, and decision-making support. Whether in healthcare, education, energy, or management, the impact can vary from individual enhancements to systemic transformations. Each application has the potential not only to revolutionize its own field but also to create ripple effects across other sectors. Through comparative analysis, it becomes clear that AI and big data possess transformative power across diverse fields, each presenting its own set of challenges and opportunities.

4 Challenges and limitations

While the integration of AI and big data offers significant advantages, it is important to acknowledge the challenges and limitations that might affect their implementation. These challenges can be categorized into technological, ethical, and organizational issues.

5 Discussion and future trends

The introduction of AI and big data ushers in a transformative era across various industry sectors. This integration revolutionizes operational efficiencies, cost structures, and decision-making processes. Specifically, implementing AI in the credit card industry for fraud detection provides notable financial savings and boosts consumer confidence by minimizing losses. Similarly, utilizing AI in the insurance sector for risk classification is set to streamline operations, resulting in more precise premium calculations and potentially reducing costs for end-users.

Technological advancements and research in AI, particularly its application in medical research for cancer detection, are leading the way in pioneering early diagnostics and improving patient care outcomes. The focus on big data learning, along with methodologies such as deep learning and transfer learning, demonstrates a substantial increase in technological research. This paves the way for more advanced AI applications in various fields. Collaborative efforts among industries, universities, and technology partners, particularly emphasized in the enterprise sector, play a crucial role in driving further research and technological breakthroughs.

While the extensive integration of AI and big data offers numerous benefits, it also presents complex social and ethical challenges. Concerns regarding data privacy are particularly notable in sectors like computer networking, where ensuring data confidentiality is crucial. The importance of robust regulatory frameworks is emphasized in sensitive areas such as credit card processing and insurance, where ethical AI usage and the protection of consumer rights are of utmost importance. In addition, the focus on the active participation of organizational personnel in AI initiatives highlights the ongoing need for training and skill development, ensuring that the workforce remains proficient in navigating the rapidly changing technological landscape.

The integration of AI and big data analytics drives advancements and transformations across various sectors, ranging from biotechnology to energy management. This integration unlocks personalized solutions, optimizes operations, and enhances decision-making processes. In biotechnology, the fusion of AI with BDA promises to revolutionize personalized medicine by allowing treatments and medications to be precisely tailored to individual genetic profiles. This approach holds immense potential to improve patient outcomes by directing therapies based on the unique genetic makeup of each individual. In addition, AI’s role in expediting drug discovery processes represents a significant paradigm shift, potentially reducing the lengthy timelines typically associated with bringing new drugs to market. By analyzing vast datasets more efficiently than ever before, AI can swiftly identify promising drug candidates and therapeutic targets, accelerating the transition from laboratory research to patient care.

In the field of IoT, the future envisions smart cities where AI-integrated devices manage traffic flows, optimize energy usage, and enhance public services through real-time data analysis. This smart infrastructure aims to create more sustainable and efficient environments. In addition, IoT devices are expected to become more autonomous, with systems capable of adapting to user behaviors and environmental changes without human intervention, ushering in a new era of intelligent automation in everyday life. Similarly, agriculture is expected to heavily rely on AI-driven solutions to manage crops and optimize resources such as water and fertilizers based on real-time environmental data. This approach holds the promise of increasing yields while reducing negative environmental impacts. Furthermore, the emergence of fully automated harvesting processes, powered by AI and robotics, could address labor shortages and improve efficiency, thus transforming agricultural practices.

In the field of data science, automated analytics and advanced predictive models are poised to revolutionize how businesses handle their data. AI offers the potential to autonomously analyze data, uncovering insights without the risk of human error, thus greatly enhancing data analytics. With the assistance of AI, predictive models could accurately forecast market trends, consumer behavior, and even global events, providing invaluable decision-making tools for businesses and governance. In the realm of management, AI-driven decision automation and real-time business insights have the power to free up human managers to focus on strategic initiatives. By automating routine decisions and providing instant insights into operations, AI tools have the ability to improve efficiency and responsiveness within organizational structures.

The future of education is expected to be highly personalized and immersive. AI’s capacity to customize educational content to suit the individual needs and learning pace of students has the potential to revolutionize the learning experience. Integration with virtual reality technologies holds the potential of creating more immersive educational experiences, captivating students in a dynamic, AI-enhanced learning environment. Conversely, the health sector is on the verge of transformation with AI-driven remote monitoring and virtual health assistants, facilitating continuous patient care and decreasing the necessity for hospital visits. These technologies could also forecast health issues in advance, providing a proactive approach to healthcare management.

In conclusion, the integration of AI and BDA is expected to promote innovation and efficiency across various domains, shaping a future where technology not only resolves intricate issues but also fulfills needs and optimizes resources with unparalleled precision. Table 2 provides a summary of the forthcoming trends in the integration of AI and big data across different domains.

6 Conclusions

This study provides a comprehensive analysis of the integration of AI and big data technologies across various industries, highlighting their transformative potential and addressing associated challenges. By examining 105 peer-reviewed articles, we have identified key applications, innovations, and obstacles in the adoption of these technologies.

Our research contributes to the understanding of how AI and big data can be integrated to enhance operational efficiency, precision, and security across diverse sectors. It provides a framework for analyzing the combined impact of these technologies and offers insights into the mechanisms through which they drive innovation and efficiency. Practically, the findings highlighted the significant advancements AI and big data can bring to industries. Our analysis demonstrated how these technologies can improve decision-making processes, optimize resource usage, and enhance overall operational performance.

Our research makes several important contributions. We have provided a detailed mapping of the diverse applications of AI and big data integration across various industries, offering a wide view of their impact. We also offer an in-depth analysis of key innovations within AI and big data integrations, highlighting significant advancements in different sectors. We have thoroughly evaluated the challenges associated with AI and big data integration, proposing potential solutions to address these issues. In addition, we have discussed the ethical and social implications of AI and big data technologies, providing guidelines for responsible deployment.

The practical advantages of integrating AI and big data are pivotal. In healthcare, AI-driven diagnostics and big data analytics have led to more accurate disease detection and personalized treatment plans, significantly improving patient outcomes. In agriculture, smart farming technologies optimize resource usage, increase crop yields, and promote sustainability. The energy sector benefits from AI and big data through enhanced efficiency and predictive maintenance, reducing operational costs and integrating renewable energy sources. These examples demonstrate the great benefits that AI and big data integration can bring to real-world applications, driving innovation and efficiency across various industries.

The integration of AI and big data algorithms in decision-making processes not only offers numerous benefits but also presents potential risks, biases, and unintended consequences. Understanding these challenges is crucial for the responsible and ethical deployment of these technologies. Table 3 presents the potential risks, biases, and the ethical and social implications of the integration of AI and big data.

Future research should explore the interplay between AI, big data, and other emerging technologies such as block chain, IoT, and edge computing. Investigating such integration can reveal new opportunities for integration and innovation. Conducting long-term studies to assess the sustained impact of AI and big data integration on various industries will provide valuable insights into their effectiveness and scalability over time.

The cross-sector analysis presented in this research not only highlights the usefulness and transformative impact of AI and big data technologies but also emphasizes the shared characteristics of efficiency, personalization, and decision-making support across different fields. While each sector presents its own challenges and opportunities, the overall benefits of integrating AI and big data suggest a future where these technologies bring about significant change, improving industry operations and services. This research contributes to a better understanding of how the application of AI and big data can drive advancements in different dynamic fields, setting the foundation for future innovations and the ongoing evolution of industry practices in the digital age.