Human-AI Teaming in a Digital Twin Model for Virtual Product Development

Introduction

The concept of Digital Twin Models (DTMs) has significantly developed in recent years, especially with the advances in the Internet of Things (IoT) and the rise of Industry 4.0. The model mirrors the attributes, states, and behaviors of the investigated physical entities in the virtual space [1, 2, 3]. Intuitively, DTMs allow for efficient storage and analysis of product-related data throughout the entire Product Lifecycle (PLC), from the early design conceptualization stage to further use and recycling. Among these, the product development stage is crucial in the product and the DT lifecycles. In fact, during the conceptualization and development phases, what is known as the ‘digital prototype’ or ‘digital model’ (e. g., [1, 4, 5]) of a product is created, which sets the foundation of the considered DTM.

During the development phase of PLC and the early stage of building the investigated DT (A) data from previous generations is utilized to identify product faults and integrate market preferences, shaping the new generation of the product. This involves analyzing vast volumes of data aggregated from earlier DTs, encompassing horizontal and vertical data related to the product in previous generations [6, 7, 8]. (B) The data obtained during the new generation of the product is used to evaluate the system’s configuration for further product development. This data may include simulated data or real data collected from product prototypes. In this regard, well-established models assess the behaviors of the system based on the design expectations and objectives and identify the different positive and negative behaviors, specifically the Unpredicted Undesirables (UUs) [9], to make an informed decision. In fact, through iterative simulations and analysis, the number of errors should be decreased to optimize and refine the virtual product and its prototype.

Nonetheless, despite the pivotal role of the product development phase, most existing studies on DTs have focused predominantly on the manufacturing process and the post-production phase of the products [10, 11]. Consequently, the role of product development within this context of DTM and, importantly, how current Artificial Intelligence (AI) methods can be exploited in the process remain underexplored. Thus, in contrast to previous studies that have mainly focused on the capabilities of DTMs at the actual production stage, we here investigate the product As-Designed.

To this end, the current work focuses on the initiation and modeling phases of the DT lifecycle corresponding to the product development phase of the PLC (see, e. g., [12] for a comparison). The aim is to enhance virtual product development as the basis for building an AI-based DTM by using an AI and human experts teaming (referred to as human-AI teaming). Such a human-in-the-loop procedure can utilize the power of both aspects and enhance the design and other conceptual stages of product development. This can effectively minimize the expensive costs of trial and error with physical product developments by detecting potential sources of defects and validating the design parameters for product quality assurance.

The paper is organized as follows: Section 2 discusses the general specification of the DTMs and how AI models have been explored in related work. Later, Section 3 discusses the investigated AI-driven DTM. Moreover, Section 4 is devoted to the investigated use case, where we demonstrate the applicability of the proposed approach using an ongoing project on Light Detection and Ranging (LiDAR) systems. For this, we assess the performance of LiDAR, focusing on its angular resolution, and explore how a human-AI teaming can consider design precision and optimize the system’s performance (for the specific tasks of object detection). Finally, Section 5 concludes the paper with a short discussion on future work.

Related Work

Our work at the Institute of Product Development and Equipment Construction (IPeG) at Leibniz University Hannover focuses on product development. It investigates how AI models can assist engineers during this process. Previous work includes the concept of design for additive manufacturing [13, 14, 15] and in conjunction with technical inheritance [8, 16], as well as DTM for distributed systems [17]. In the following, we discuss the state-ofthe-art DTMs, DT throughout the PLC, and AI’s role in DTMs.

Digital Twin Models

The concept of the DTMs was first introduced by Grieves [1, 2, 3] within the context of product lifecycle management in engineering. In general, a standard DTM consists of three key components: (A) the real space, which consists of the actual or intended physical entity, also known as the Physical Twin (PT); (B) the virtual space that hosts the corresponding digital counterparts, referred to as Digital Twins (DTs), (C) the digital thread, which bridges the physical and virtual spaces by establishing a communication between and within the two spaces [1, 4, 5, 18].

A distinguishing characteristic of DTMs is their ability to establish bidirectional communication between and within physical and virtual spaces [1, 19]. In a DTM, data flows from the physical environment to the virtual space, enabling continuous processing and extraction of critical insights. The obtained information is then transferred back to the real space to optimize real-world operations. This ongoing exchange of data and information forms a digital thread throughout the PLC, which supports seamless decision-making and operational enhancements (see, e. g., [19, 20, 21] for a complete discussion).

Recent work by Stark, Anderl et al. [4], conducted under the Wissenschaftlichen Gesellschaft für Produktentwicklung (WiGeP) association in Germany, categorizes DTs (the corresponding part of the virtual space) into two distinct parts of Digital Master (DM) and Digital Shadow (DS), where each part relates to different attributes of the framework (see Figure 1 which will be discussed further below). The DM encompasses meta-data, design parameters, and simulation data and models for product development, primarily generated during the early stages. On the other hand, the DS includes the data and all the related processing and is continuously updated with data collected from the physical object throughout the operation or maintenance phase of the lifecycle [22, 23]. The studies emphasize that a DT will be completed after the actual production of the product, where the corresponding DSs are established.

Figure 1

The investigated DTM

Furthermore, the communication between DM and DS is likewise established using different tools, such as Simulation Data Management Digital Twin (SDM-DT) [24, 25], and Virtual Part Inspection (VPI) [26], discussed in previous related studies. Data collection enables DS to provide up-to-date information that enhances simulation models, monitoring, and predictive analysis.

In another study, Eigner et al. [19] discussed the concepts of digital models and digital twins in DTMs (also see [20, 27]). The digital model counts as the basis for implementing DTs, which will be defined during the early “Concept and Design” phase of the PLC and further updated and enriched throughout the PLC. The digital twins are later defined as the instances from the digital model that occur in the production phase.

The Investigated AI-Driven DTM

As discussed, our primary focus is to enhance product design and development within the context of DTMs. We build upon the model proposed by Stark, Anderl et al. [4] and assume a model encompassing DM and different DS(s) in the virtual space. Importantly, we exploit AI algorithms to enable optimal product design and shorter development cycles to expand the creative work of the engineer. To ensure such a process, an efficient human-AI teaming is investigated. AI and advanced ML algorithms provide solutions or recommendations, and later, the engineer realizes and executes validation and design decisions. The engineer can extend and build on the proposed decisions by the model. If such a teaming is efficient, humans and AI will complement each other as, for instance, the AI can quickly process huge databases on a scale, and humans can understand common sense. The teaming is, therefore, based on a mutual understanding, which further promotes XAI.

The structure of the investigated DTM is illustrated in Figure 1. To explain the details of the model, we begin with the product development phase in the virtual space. In this phase, the DT is conceptualized, even before the physical product exists or when it is based on a previous generation model. Thus, the new generation of the product evolves from an initial concept to an optimized design before actual manufacturing begins. This phase is characterized by gathering information and requirements for virtual product development, as assigned to the DM (see, e. g., [26]). Once the virtual product design is finalized, it is transferred to the real space for manufacturing, represented as a PT in Figure 1. The DT entirely takes shape at the end of this phase, when the physical product is manufactured, and data from the physical product is fed back into the virtual space.

Throughout the DTM, particularly during the operational phase of the product, vast amounts of data are generated from various sources based on the product’s performance in the market. This data is continuously sampled from the real space and transferred to the virtual space. The data is processed in DS(s), and as the physical and virtual systems are twinned, changes such as part replacements or state transitions can occur in both systems. Information is often transferred back to the physical space through actuators (e. g., [9]). Ultimately, the collected data, aggregated throughout the entire PLC, is utilized in the Disposal phase, e. g., failure prediction using AI and ML models.

In virtual product development, three main components are involved: Design optimization, Product Data Management (PDM), and Simulation Data Management (SDM). The former focuses on the requirements of each iteration step to achieve the optimal product design, while the latter two components are responsible for managing simulated data. In this framework, advancements in ML algorithms can facilitate the process by reducing iteration cycles, enhancing the creativity of design engineers, and improving accuracy. For example, generative models can be utilized to create new CAD models [46], thus opening new possibilities for design innovation and creativity.

Besides, simulation techniques are commonly used to predict the expected behaviors of the PT. By simulating system behavior, design engineers can access valuable data sets that are further leveraged by analytical AI-based approaches for evaluating the model (e. g., by exploiting optimization methods). Such approaches focus on discovering new insights, patterns, relationships, or dependencies in data to support data-driven decision-making.

Figure 2 details the structure of the optimization cycle for virtual product development. In this procedure, the engineers will realize the first concept and definition of the product and later use them for simulation. If a previous generation of the product exists, then all the relevant data (from the aggregated DTs of the previous generations) will be exploited to establish the new model. In the next step, simulation techniques and tools (such as CAD, CARLA, etc.) will be utilized to generate data corresponding to the product’s current stage. The data will be further processed and used by the investigated human-AI teaming.

Figure 2

The diagram demonstrates the virtual product development process from design and simulation to validation and design optimization.

Here, we exploit deep learning algorithms to leverage their hierarchical learning representation capabilities and extract the desired features from the data. The choice of the corresponding ML algorithm depends on the details of the product, the acquired data, and the task we are investigating. It could be in the form of deep probabilistic models such as Variational Autoencoders (VAEs) [47] or conventional deep neural networks such as Convolutional Neural Networks (CNNs) [48]. For our use case in this study, a novel version of the CNNs is employed, which will be discussed further in the next section. The analysis part outlined in Figure 2 demonstrates the steps the investigated human-AI teaming takes to transform raw data into valuable knowledge. We exemplified the steps based on the considered use case and the methods that were applied for object detection.

After training, the corresponding results will be assessed and evaluated given benchmark metric values, and potential suggestions for design optimization will be obtained. Such analysis will be conducted by the human-AI teaming, where the design engineer can decide on further post-processing methods and modification of the design parameters for the desired product. Given the new parameters, a product redesign will be conducted, which is usually an enhanced version of the previous step. The loop will continue till the desired design is achieved. In the next section, we showcase this procedure using our use case of LiDAR systems.

The above procedure demonstrates the use of AI in product design. Applying such teaming can lead to an XAI, which provides several uncovered opportunities in production and engineering. For instance, an AI system is used to enhance the design process by analyzing a database of existing products. In this scenario, the final product will still be developed by an engineer, but as discussed, the process will be enhanced by AI.

Another example where XAI can be beneficial is the analysis of prediction errors of AI systems when applied, e. g., to monitor a manufacturing system. XAI can uncover spurious correlations picked up during the training of the AI system [49]. This means that an AI system might never have learned what a damaged product looks like but picked up features that co-occur, such as different reflection patterns of light sources on the product’s surface. A change in the illumination conditions could lead to significant errors in this case, which is usually not intended; XAI could help identify such issues efficiently.

The Investigated Use Case

The following use case details the underlying concept of building a DTM for Li-DAR systems, emphasizing how human-AI teaming can be integrated into the process. LiDAR is an advanced sensing technology that employs modulated lasers to measure the distance by calculating the laser’s Time of Flight (TOF). This enables the creation of detailed 3D point cloud maps of the surroundings. Currently, the resolution requirements for LiDAR systems remain unstandardized. On the one hand, low point cloud density may lack the necessary detail to capture key object characteristics, potentially leading to detection failures. On the other hand, increasing point cloud density requires higher resolution detectors or sampling frequencies, which raises system costs and can introduce unnecessary redundancy.

For this use case, we are interested in assessing how a specific point cloud density, defined by a particular angular resolution and generated by the LiDAR sensor, affects object detection performance. This enables us to modify this parameter and find an optimal configuration that leads to systems with higher object detection accuracies. Intuitively, we aim to build the corresponding DTM of the new generation of LiDAR systems by initiating virtual product development and enhancing the product’s design. This corresponds to the ‘Initiation’ and ‘Modeling’ phases of the DT lifecycle [12], i. e., the creation of the corresponding DM of the model. To this end, the main point that will be discussed in the following is how the proposed human-AI teaming will be applied.As the first step, data from the product’s (in this case, LiDAR systems) previous generation should be collected and assessed. As mentioned, this denotes the horizontal and vertical data acquired by the aggregated DTs of the earlier generations. The corresponding DMs of the previous generations facilitate the product‘s modeling, and DSs allow for detecting possible points of enhancement, unpredicted undesirable behaviors, and optimizing the product’s next-generation design. Nonetheless, real data obtained from DSs often lacks insights into entirely new product designs, underscoring the importance of simulated data. While real-world data provides a valuable foundation for market research, defect diagnosis, and predictive analysis, its utilization is constrained by the high costs associated with physical prototyping during the product development phase.

We used CARLA [50], an open-source simulator that provides a controlled environment for research on autonomous driving. CARLA efficiently simulates the system and field environments based on different degrees of complexity. It further allows for data annotation by generating ground-truth bounding boxes.

The LiDAR configuration used in CAR-LA features a vertical Field of View (FOV) ranging from -30 ° to 10 °, along with a horizontal FOV of 360 °. It has a detection range of up to 150 meters and operates at a rotation frequency of 10 Hz. Furthermore, the number of channels and points per second can be adjusted to modify the angular resolution. The generated LiDAR data ranges from an angular resolution of 0.1 °x0.1 ° to 1.0 °x1.0 °. Each data set at the specified angular resolution has a 3-minute recording in a smalltown scene, totaling 3000 frames of Li-DAR point clouds, with 1600 frames used as the training set, 400 as the validation set, and 1000 as the test set [1].

The scene includes 30 vehicles (excluding the ego-vehicle), their speeds dynamically adjusted according to the area‘s speed limits, 13 bicycles moving at a constant speed of 4 m/s, and 40 pedestrians at 1 m/s. Given the data, we trained the considered algorithm (PV-RCNN discussed further below) with training times varying depending on the complexity of point cloud data, from 31 hours at 0.1 °x0.1 ° to 9 hours at 1.0 °x1.0 °. The experiments are conducted on a system equipped with one NVIDIA RTX 3060 GPU, 12 GB. Figure 3 illustrates a generated frame used for training, together with the corresponding point clouds.

Figure 3

A considered environment in CARLA, used for collecting point cloud data (left); The object detection using the trained PV-RCNN model. The figures correspond to the angular resolution 0.1 °x 0.1 ° (right)

PV-RCNN for LiDAR Systems

Developing an initial concept for an optimized product requires robust methods for data analysis and evaluation. This plays a pivotal role in assessing system’s performance, identifying potential defects, and ultimately optimizing product design. We here employed Deep Neural Networks (DNNs) to learn and predict the performance of the considered LiDAR system with a specific angular resolution. This allows us to evaluate the potential functionality of the model by testing and analyzing device performance (for the task of object detection). Nevertheless, the model exploits a black-box approach that further requires expert knowledge (throughout the entire process) for informed decision-making, forming the discussed human-AI teaming.

We employed the PointVoxel-RCNN (PV-RCNN) model, introduced by Shi et al. in [41], demonstrating state-of-the-art performance in 3D object detection for point cloud data. This model combines gridbased and point-based methods to leverage each approach’s advantages, addressing the inherent challenges in detecting objects within point cloud data. Gridbased methods are generally more computationally efficient but suffer from inevitable information loss. In contrast, while computationally more expensive, point-based methods can achieve superior spatial resolution and a larger receptive field through point set abstraction [42]. By integrating these two methods, PV-RCNN effectively balances computational efficiency and localization accuracy, resulting in improved performance for 3D object detection tasks.

The results are evaluated in terms of mean Average Precision (mAP) with 40 recall positions on test data and are further summarized in Table 1. In addition, Figures 4 and 5 provide the results across different angular resolutions and for the three distinct objects. We first match the predicted bounding boxes to their corresponding ground truth to calculate the corresponding values using a metric such as Intersection over Union (IoU). Predictions with an IoU above a predefined threshold are classified as true positives, while the remaining predictions are false positives. Precision and recall are then computed, where precision represents the proportion of correctly predicted bounding boxes among all predictions, and recall measures the proportion of ground truth instances successfully detected. To compute the Average Precision (AP) for a given class, precision values are averaged at 40 evenly spaced recall points. The mAP is then derived by averaging the AP values across all object classes, providing an overall performance metric for the detection model (i. e., the PV-RCNN model). Moreover, Figures 4 and 5 present the average confidence scores as an additional metric for analyzing the considered use case. These values were computed by segmenting the point cloud data into different density intervals. The confidence scores assigned by the PV-RCNN model to the predicted bounding boxes within each interval were then averaged. As observed, an increase in point cloud density generally leads to higher confidence scores, which highlights the impact of density thresholds on enhancing detection capabilities.

Figure 4

Illustration of the average confidence scores for detecting three different objects using PV-RCNN trained on data with an angular resolution of 0.1 ° x 0.1 °

Figure 5

Illustration of the average confidence scores for detecting cyclists using PV-RCNN trained on data at various angular resolutions.

Table 1

The corresponding mean average precision values with 40 recall positions obtained using the PV-RCNN model. Here ‘BBox’ denotes Bounding Box, ‘BEV’ denotes Bird’s Eye View, and ‘AOS’ denotes Average Orientation Similarity. The results correspond to the angular resolution 0.1 ° x 0.1 °. We used a threshold of 0.7 for cars and 0.5 for both pedestrians and cyclists

Conclusion and Future Work

Integrating DTMs with advances in AI and ML transforms product development and manufacturing by enabling the processing and analysis of vast, heterogeneous data from diverse sources. AI-driven DTMs provide sophisticated diagnostic and decision-making support in this context, enhancing product quality and production efficiency. These technologies make it possible to address core manufacturing challenges, such as maintaining high product quality and boosting customer satisfaction.

In this paper, we investigated virtual product development as the basis for building an AI-driven DTM, utilizing human-AI teaming to leverage the distinct strengths of both the AI model and the design engineer. This approach allows for optimized design processes and more informed decision-making. We further demonstrated the framework’s capabilities by thoroughly examining LiDAR systems.

Future work includes expanding the framework’s application to various use cases, further developing its versatility and utility. Additionally, exploring other deep learning algorithms, especially generative models capable of extracting structural information, holds promise for improving model performance in product design tasks. These enhancements could be particularly impactful for tasks in image analysis, such as those involved in CAD designs.