Digital forensics architecture for real-time automated evidence collection and centralization: Leveraging security lake and modern data architecture

1 Introduction

Nowadays, cloud computing has become more commonly used within organizations over the years due to the numerous services provided to the user. As the utilization of cloud resources continues to grow, it also draws the attention of more advanced adversaries to exploit its potential. Cybercriminals have recognized the significant value proposition that cloud platforms offer, enabling them to scale up their operations and execute sophisticated attacks. A survey conducted between March 2022 and March 2023 among cybersecurity leaders of worldwide organizations revealed that, on average, 66% of organizations worldwide fell victim to a ransomware attack [1]. The estimated cost of cybercrime is expected to increase from 2023 to 2028, reaching a total of 5.7 trillion USD (+69.94%) [2]. Attackers can gain access to critical areas, such as cloud computing services, by using stolen credentials. This unauthorized access allows them to compromise the confidentiality, integrity, and availability of these services. In response to such crimes involving the cloud, investigators must conduct forensic investigations within the cloud environment. Cloud forensics, a subset of network forensics, becomes essential in these scenarios. By identifying, gathering, preserving, examining, and reporting digital evidence, cloud forensics applies scientific principles, practices, and procedures to reconstruct the events. The field of forensics is rapidly evolving, extending traditional digital forensics to encompass cloud-based environments and services [3]. According to the definition provided in [4], cloud forensics refers to the implementation of digital forensic science techniques within the context of cloud computing environments.

Digital forensics investigations require the collection and analysis of a large amount of data from various sources, including cloud infrastructure [5]. Analyzing this data is crucial for detecting attack tactics and techniques, identifying attack behaviors, determining the types of attacks, and uncovering issues related to malware and vulnerabilities [6]. To meet the increasing demand for effective data collection and analysis, investigators must leverage new technologies capable of gathering and analyzing information from multiple sources to uncover insights that were previously unattainable. The coming of the data lake concept has empowered security organizations with the ability to aggregate all data, making it readily accessible at any given time [7]. However, collecting evidence, especially logs from cloud infrastructure, poses significant challenges due to the exponential growth of digital data generated by organizations. Investigators need to be able to efficiently collect, store, and analyze this data while maintaining its integrity in digital forensics investigations [8]. The difficulties arise because security evidence is allocated across different servers in various regions, available in heterogeneous formats from different cloud layers, and must maintain data integrity in a multi-tenant environment. To overcome these challenges, previous research studies [9,10,11] have introduced different models for evidence collection and ensuring the integrity of the evidence. Other studies [12,13,14] focus more on analysis, emphasizing data collection techniques that enhance efficiency and ease of analysis, particularly for small companies. However, none of these studies address the issue of evidence collection in the context of organizations operating across multiple accounts and regions and how to centralize all the ingested data and keep it in one format. As organizations expand and require scalable infrastructure across various accounts and regions, investigators face increasing complexity in collecting evidence. The challenges come from the necessity to centralize all security evidence resources shared among accounts in different regions. The process of centralizing evidence involves copying data from various accounts with different formats and consolidating it, which is time-consuming. Time is a critical factor in digital forensics, adding pressure to the investigation process. Additionally, maintaining the integrity of evidence when resources are shared among multiple accounts, spread across various locations, and accessible by various parties without breaching the confidentiality of others, presents further challenges.

Furthermore, the unprecedented growth of data volumes from terabytes to petabytes, and even exabytes, further compounds this challenge [15]. Traditional on-premises data analytics approaches are inadequate for handling these volumes due to their limited scalability and high costs. In general, investigators must have the capability to readily retrieve and examine various forms of data, including structured, semi-structured, unstructured, and real-time streaming data, To conduct thorough and effective analyses, they also need to get and move data between their centralized data lakes and the surrounding purpose-built data services in a seamless, secure, and compliant way, to get insights with speed and agility.

Currently, cloud providers offer services that can be utilized to develop advanced systems for forensic investigators. In this article, we introduce a system that automates the collection and storage of evidence, providing investigators with a robust and secure platform for conducting thorough investigations efficiently and promptly. This system will leverage various cloud services provided by the AWS environment. Initially, the system will utilize services to gather logs from different accounts. Subsequently, it will employ the capabilities of a security lake and modern data lake architecture to serve as a centralized repository for security-related data and analytics, along with a design for scalable and flexible data storage, processing, and analysis.

The proposed Digital Forensics Architecture for Real-Time Automated Evidence Collection and Centralization by using Security Lake and Modern Data Architecture makes significant contributions to digital forensics investigation by gathering security evidence from multiple accounts and regions automated in real time. The main contributions of this work are listed below:

1. The system enables investigators to quickly gather, store, and analyze relevant data from multiple accounts within the organization and centralize it all in real time.

2. The system reduces the complexity associated with accessing data from individual accounts separately.

3. Centralized storage eliminates the need for separate storage systems for individual data sources or accounts, resulting in reduced storage costs and simplified data management.

4. Enhanced security because the components offer secure data storage, encryption, control access, and permission.

5. It gives seamless scalability, accommodating the growth of data over time without significant infrastructure investments. This scalability reduces costs associated with infrastructure expansion and ensures the system can manage increasing data volumes efficiently.

This article is organized into seven sections. The second section discusses related works. The third section provides preliminary knowledge about Amazon cloud services. The fourth section describes the proposed architecture and its components. The fifth section presents the implementation and evaluation of the proposed architecture. In Section 6, the results are analyzed and the limitation is presented. Finally, in Section 7, the conclusion is presented.

2 Related works

In recent years, researchers have directed their focus toward digital forensics within the cloud environment to ensure security against cyberattacks. Numerous significant research works have been conducted, with specific emphasis on digital forensics and data lake technology. Alex and Kishor attempted to address evidence collection in cloud forensics, aiming to tackle issues arising under the control of cloud service providers (CSPs) [16]. To overcome these challenges, Alex and Kishor proposed collecting all evidence outside the CSP’s control, specifically in the forensic monitoring plane. However, relying on a single forensic server as the storage location for all evidence can create a failure in a single point, leaving the evidence vulnerable to alteration and deletion if the forensic server is compromised by an attacker.

Alternatively, several researchers proposed an architecture that collects and preserves evidence for an Infrastructure-as-a-Service cloud in a blockchain, distributed among multiple peers [3,17] However, these approaches increase the complexity of data movement between analytical tools, requiring additional coordination and effort. It involves retrieving data from different peers, ensuring data consistency, and adapting it to the requirements of specific analytical tools, potentially hindering the efficiency and effectiveness of the investigation process. The model of Secure Logging as a Service (Secale’s) was introduced to establish a cloud forensics architecture that ensures the collection of various logs without loss of integrity [18]. To uphold integrity, a hash chain scheme was implemented, and periodic publication of proofs of past logs to cloud providers. However, the complexity of analyzing logs increases when they are collected in a centralized manner.

To address the problem of dependence on a CSP, the model of Forensic Acquisition and Analysis System (FAAS) was presented [19]. This agent-assisted system enables the control of all preserved evidence by an agent manager and coordinator, rather than the CSP. However, FAAS is not able to keep data provenance, which is a critical component in cloud forensics. Additionally, the involvement of multiple agents contributes to the complexity of the system. The model log aggregation was proposed for a digital forensic architecture, which includes several processes such as log extraction, acquisition, indexing, normalization, correlation, sequencing, and presentation [20]. All logs that were gathered were deposited into a central evidentiary log repository with the intent of carrying out additional analysis. Nonetheless, this centralized repository is vulnerable to attacks and can be compromised by attackers easily, as it lacks built-in security features such as access controls, encryption, and monitoring to protect against unauthorized access and suspicious activities. Once attackers gain access to the repository, they can modify or delete the logs, making it challenging to conduct accurate and reliable forensic investigations.

A forensic acquisition approach based on fuzzy-based data mining was proposed in Santra et al. [21]. The approach involved the use of a fuzzy expert system for monitoring, analyzing, and generating evidence for cloud logs. The CSP maintained control over the storage and management of all evidence. However, since the CSP cannot be entirely trusted in a cloud environment, the reliability of the evidence is reduced. To address the dynamic configuration challenges of cloud architecture, a proposal for an adaptive evidence-collection mechanism was introduced in Pasquale et al. [22]. This mechanism considers three distinct scenarios: vulnerable databases, security breaches, and cloud configuration. By considering these scenarios, the evidence-collection process is dynamically adjusted to adapt to the specific circumstances. While this approach demonstrates adaptiveness, it falls short in terms of providing comprehensive data provenance and ensuring evidence integrity.

Since cloud computing has spread, central logging is another potential approach. Because of virtualization, data distribution, and replication, identifying and gathering data in the cloud is not an easy operation. Investigators can quickly and effectively identify and gather evidence of artifacts utilizing a single storage place through central logging [23]. Several publications have also emphasized the significance of cloud computing which is forensics-enabled and the requirement for a distinct model that gathers forensics evidence. Kebande and Venter proposed Cloud Forensics Readiness Evidence Analysis System which is designed to leverage distributed hardware clusters within the cloud environment utilizing the MapReduce framework. This System is composed of various modules, such as a forensic Database that operates through the MapReduce process. These modules include Forensics logs, hypervisor error logs virtual images, which are then stored in the forensics database [24]. Patrascu and Patriciu proposed cloud architecture with forensics capabilities to monitor various activities within a cloud environment. They introduced a cloud forensics module designed to collect all forensic evidence and logs from virtual machines, through interaction with the underlying hypervisor [25].

Despite recent research efforts, there remains a significant gap in addressing the challenges of evidence collection for organizations operating across multiple accounts and regions. Specifically, no studies have thoroughly examined how to automate the collection of security evidence in real time, centralize data lakes, and standardize ingested data from diverse sources. Additionally, the seamless movement of data between lakes and the surrounding purpose-built data services has not been adequately explored. Therefore, the architecture for automating collection and centralization while maintaining the integrity of evidence in complex environments remains a major concern in cloud forensics.

3 Preliminary knowledge to Amazon

4 Proposed architecture and methodology

In this section, we introduce the architectures of the system we have implemented after considering the above-explained problems. We have implemented a system to gather data from multiple accounts and regions in real time and store the evidence in a modern architected data lake to ensure the investigator can analyze these data with more flexibility. We also implement access controls and permissions to keep the evidence more secure by controlling who accesses the data as shown in Figure 1.

Figure 1

Architecture of the proposed system.

The collection of evidence from cloud-based systems, as followed in this architectural framework, depends on the following workflow steps shown in Figure 2.

Figure 2

Process of evidence collection from cloud-based environment.

The architecture of the proposed forensic system consists of the following main components:

5 Implementation and evaluation

In this section, we will explain how the proposed system is implemented to collect and store evidence in centralized lakes automatically in real time while maintaining the integrity of evidence. This is achieved by combining security lake and modern architecture with other various services from the AWS cloud environment and building the modern data lake architecture with its key elements. We will also discuss how it was tested with a university system in AWS and integrated with third-party tools for advanced analysis. The implementation process involves several steps, each of which is detailed below.

Step 1: Create a delegate account which is the forensics account to manage the security lake and centralized lakes and add group users with permissions access.

Step 2: Identify security events and log sources to ingest from all supported and integrate services and enable them and specify the accounts and regions from which data should be collected.

• CloudTrail: collect log API calls and user activities across all AWS accounts.

• Security Hub: Security Hub is activated to aggregate security findings from various AWS services.

• VPC Flow Logs: VPC Flow Logs are enabled to capture information about IP traffic to and from network interfaces within the VPC.

• Route S3: Logging for DNS queries is enabled in Route 53.

• To configure log source collection.

Step 3: Create an IAM role that gives security lake permission to gather data from all sources and add them to data lake.

Step 4: Set up storage preferences for the data and specify rollup Regions for data consolidation.

Step 5: Add subscribers to the users outside of the group in forensics account who want to access the lakes for security with less privilege.

Step 6: Establishing a modern data architecture.

• This process entails cataloging the data using AWS Glue, following a series of outlined steps. First, a data catalog is created by subjecting the data sources to a crawling process, which automatically generates a catalog containing metadata that describes the data. This metadata facilitates easy exploration and querying.

• Second, a schema is defined for the data once the catalog is created. This schema establishes the structure of the data, simplifying the analysis process.

• Third, a Glue job is generated to extract data from the data sources, transform it, and load it into the desired data store. Glue data connectors are utilized to establish connectivity with various data sources, including Amazon S3, Amazon RDS, Amazon Redshift, and other data stores, and then scheduled the Glue job to ensure that the data remain consistently up-to-date and readily available for analysis purposes.

• Finally, implementing data governance and compliance in a modern data architecture is crucial. It involves defining data governance policies that cover data handling, access control, data quality, data retention, and compliance requirements, ensuring alignment with relevant regulatory and industry standards. Robust access controls and authorization are implemented to restrict access to specific data assets within the data lake, utilizing AWS IAM for user roles, permissions, and authentication, as shown in Figure 3.

Figure 3

JSON file for policy.

The principle of least privilege should be followed to grant access based on job roles. Furthermore, data encryption mechanisms are deployed to protect data at rest and in transit, leveraging AWS services such as Amazon S3 Server-Side Encryption and AWS KMS for encryption key management, with a focus on sensitive data to prevent unauthorized access.

To evaluate the system, we conducted tests using a university system built on an AWS environment. This system comprises a management account, which serves as the root account, and utilizes the AWS Organizations service to create different organizational units (OUs) for various college and department accounts. Additionally, we created an account named the forensics account to collect and centralize all the lakes for all chosen accounts. We set up the experiment by selecting accounts from this organization across different regions to utilize the proposed system for automated real-time collection of security events across multiple accounts and centralization in lakes within the forensics account. To facilitate easy and rapid analysis of the data stored in these lakes, we utilized Amazon Athena and AWS Lake Formation to view and query the evidence. Furthermore, we integrated these lakes with Wazuh, an open-source Security Information and Event Management (SIEM) solution, to perform more advanced analysis.

6 Results and discussion

After testing the proposed system, we proceed to analyze the performance of the forensic architecture. In this section, we will first introduce the results obtained from implementing the system within the university. Then, we will compare our proposed approach with the previous centralized log collection approach.

Once the system was implemented and the regions and accounts were selected from which to collect security evidence, along with specifying the types of logs to be collected and enabling the services accordingly, the results, as shown in Figure 4, that eight lakes were created centralized within a single account. Additionally, one lake was created for the chosen roll-up region to consolidate all selected regions and accounts inside it. This is considered a significant advantage because it allows for one lake designated for each region, facilitating localized data management and analysis. Simultaneously, the consolidated lake encompassing all regions enables broader analysis and correlation across the entire environment. This approach offers flexibility and scalability in managing and analyzing security evidence.

Figure 4

Lakes for each chosen region.

The analysis of creation time indicates that the initialization of lakes happened simultaneously, which is nearly identical to real-time as shown in Figure 5. In digital forensics, where timeliness and accuracy are crucial, this demonstrates the system’s capability of collecting evidence from multiple accounts and regions in real time. This capability is highly advantageous in forensic investigations, as it enables prompt collection, storage, and analysis of evidence, empowering investigators to quickly respond to security incidents or potential threats.

Figure 5

Corresponding time to create the lakes.

If we investigate deeper into each lake, we will find folders corresponding to each service enabled for log collection, along with folders for each selected account. Specifically, folders are created within the chosen region’s bucket for each service and each account that has the service enabled. For example, in this environment, CloudTrail and Security Hub findings are enabled. Figure 6 is shown in two folders: one for Security Hub and another for CloudTrail. Inside these folders, the accounts we selected to collect evidence from are included, and all the evidence is ingested inside the account folder in partial hourly intervals, as shown in Figure 7.

Figure 6

Folder for each service.

Figure 7

Evidence hourly partition.

If we go through all account folders, we will find that all evidence is automatically collected in real time (within about 2 min) from each account and sent to the respective S3 bucket assigned to each account, in addition to the roll-up area. This evidence is collected and organized in a unified format from all accounts, as shown in Figures 8 and 9. Although the data is collected from different sources and layers within the cloud, this approach solves the problem of having evidence available in heterogeneous formats from various cloud layers. By organizing all collected evidence into a unified format, investigators can quickly and efficiently analyze the data without needing to convert or normalize it from multiple formats. This reduces the time and effort required for data preparation.

Figure 8

Evidence collected with services.

Figure 9

Evidence collected from services in minutes.

The data and evidence have been successfully ingested into the S3 data buckets. To establish a data catalog for each service, AWS Glue is utilized. To extract valuable insights from the data stored in the S3 buckets, a crawler is created and configured. Before setting up the crawler, it is important to create an AWS IAM role that has both S3 Full Access and Glue Service Role policies enabled.

Once the crawler is launched and subsequently stopped, tables are added to the database based on the raw data captured during the crawling process, as illustrated in Figure 10. The raw data are then cataloged within AWS Glue. At this point, investigators can effectively query or analyze the data using tools such as Amazon Athena, Amazon SageMaker, and Amazon Recognition which significantly enhance the capabilities of digital forensics by automating analysis, identifying patterns, and extracting valuable insights from forensic data.

Figure 10

Tables for data.

For integration with third-party services, it was integrated with the Wazuh which is SIEM solution as shown in Figure 11.

Figure 11

Integrated with the Wazuh.

The above results show that the proposed forensics system has the capability to collect and centralize evidence in real time by using security lake and modern data architecture. Furthermore, the integration of the system assists investigators in analyzing and answering investigation questions. This characteristic is the main strength of this system compared to other frameworks. Table 1 depicts the comparison of the proposed system and other frameworks based on these characteristics.

7 Conclusion

The main goal of this research was to develop a system for cloud forensics investigation that provides investigators with the ability to collect security evidence from multiple accounts and regions within a cloud-based system, store data in centralized lakes, maintain its integrity, and make analysis easier for investigators. The proposed system, based on combining Amazon Security Lake and modern data architecture, serves as a centralized repository that preserves evidence by ingesting events and logs from various regions and accounts. Additionally, these lakes can integrate with third-party tools such as Wazuh, Splunk, and Athena, facilitating advanced analysis and reporting for investigators. The automation in collecting and centralizing data, along with seamless real-time analysis, strengthens this system compared to others in cloud digital forensics. This system was evaluated using an organization built on AWS with multiple OUs and one dedicated account for forensics. The proposed system successfully created all the lakes, collected evidence from multiple accounts and regions in real time, and centralized it in one account. Future research can explore building this system in different cloud environments and integrating it with SIEM solutions for enhanced cloud security monitoring.