Architectural animation generation system based on AL-GAN algorithm

3.1 Establishment of attention-GAN model

AA can generate images through network data models by inputting lines from architectural images and finally form a continuous AA. However, in overall drawing, there are often cases of incomplete colors, missing details, inadequate analysis of buildings, and even inconsistent image colors. Therefore, in the construction and analysis of AA, a new network framework needs to be added to enhance the detailed description of AA [16]. Figure 1 is the framework structure of AL-GAN.

Figure 1

Framework structure of AL-GAN algorithm (Self-drawn by author).

In Figure 1, the input structure is included in the framework to complete the input and processing of AA data. The preprocessed data are input into the attention generation adversarial network to generate more complex AA image data. These data are then input into the LSTM-GAN, and the processed LSTM-GAN data are output to generate new animation data. In Figure 1, two GANs are present in AL-GAN concurrently. Attention-GAN completes the animation color supplementation operation for different architectural styles in the early stage, while LSTM-GAN unifies the consistency of AA [17].

Attention-GAN is an improved approach to data transformation, function training, and other operations based on ArchcolGAN. ArchcolGAN mainly processes the color and style of architectural images. Therefore, it can also color the building lines and colors. The original model has been modified by introducing a new attention mechanism. Figure 2 shows the framework of Attention-GAN.

Figure 2

Framework of attention-GAN model (Self-drawn by author).

In Figure 2, Attention-GAN includes a generator and a discriminator. The generator can output relatively realistic image data through network training. The discriminator performs discriminant analysis on the currently generated image data. The generator of Attention-GAN includes a decoder, converter, and encoder. The encoder has two convolutional layers that do not change the size of the image, which only increases the channels of the feature image. When the input convolution image is 256 × 256 × 1 and the convolution kernel size is 7 × 7, the overall number of convolutions is 64. To ensure consistent data entry and exit sizes in the image, it is necessary to input a value of 0 in 3 × 3 blocks for the upper, lower, left, and right data in the image data. The convolutional kernel can only move up or down one grid distance on the image data. All corresponding data on its channels need to complete the multiplication operation. The final movement of the convolutional kernel will form a complete feature image. At this point, normalization is performed on the 64 feature images obtained, and an activation function is used to complete the output of the convolutional layer [18].

When building the converter, because the encoder has already encoded the feature data of the image into smaller data, the operation of the converter requires a style transformation operation on the current image data. The converter in the network model requires two Dense Net-BC network structures, which are similar to deep learning in Figure 3.

Figure 3

Dense Net-BC network structure (Self-drawn by author).

In Figure 3, Dense Net-BC consists of multiple convolutional layers, which are mainly analyzed and processed by performing convolution operations on the image data. However, during data processing, Dense Net-BC adds up the dimension channels of all previous convolutional layers, performs convolution calculations, and inputs them into the next layer. Therefore, when processing data, Dense Net-BC has L ( L + 1 ) 2 convolutional connections, and L refers to the number of channels. This convolution operation can reduce the vanishing of model gradients, enhance the utilization of feature channels, and reduce the usage of parameters [19]. When designing Dense Net-BC, it is necessary to set the convolution kernel size of each convolutional layer to 3 × 3 and the number to 32, resulting in a feature map with a convolution size of 32 × 32 × 80.

The decoder layer of Attention-GAN mainly decodes various data images on the current feature image to complete the conversion from data to image. In the decoding layer structure, there are three reverse convolutional layers and one parallel convolutional layer. Each convolutional layer requires the addition of a reverse convolutional layer and a parallel convolutional layer. The reverse convolution layer expands the length, width, and height of the input feature map to N times by setting a step size. Therefore, it is necessary to add a parallel convolutional layer between the upsampling and downsampling connections in each layer. This reduces resolution when downsampling and increases resolution when upsampling, thereby enhancing the painting effect of the building image. This can make the contours and curves of the architectural image in decoding clear and reduce line loss caused by conversion and network training [20].

The Attention-GAN discriminator is mainly constructed by downsampling convolutional layers, using five downsampling convolutional layers. The convolutional kernel size is set to 4 × 4, with quantities of 1,024, 256, 64, 512, 128, and a step size of 2. By reducing the image resolution and then using the flattened function, the feature vector images with larger dimensions are flattened into low dimensional vectors. Since the research object of this study is an architectural model, it is necessary to determine whether there is a coloring error during the coloring process. When the coloring exceeds the architectural wireframe, it is judged that there is a coloring error problem.

For the calculation of the loss function of the network model, Attention-GAN is divided into two parts: the first one is the generator loss function, and the second one is the judgment loss function calculation. For the calculation of the two loss functions, the hinge loss function calculation method is used, represented by Eq. (1) [21].

where y ˆ represents the size of the predicted value. L ( y ˆ ) refers to the size of the hinge loss function. y = ± 1 . According to Eq. (1), the hinge loss function needs to make y ˆ infinitely close to the true value to make the loss function smaller. ∥ 0 , 1 − y ⋅ y ˆ ∥ 2 represents the absolute value of an element. The loss function of this generator is represented by Eq. (2) [22].

where L ( G ) refers to the loss function of the generator. E ( ⁎ ) represents the expected value of the function. c means the image after coloring. S data ( c ) is the image distribution of AA after coloring. E c − S data ( c ) refers to the expected value of the distribution function of the style change image after coloring. G ( c ) represents the colored image obtained by the generator. ∥ • ∥ 1 represents the sum of absolute values. The loss function of the discriminator is represented by Eq. (3) [23].

where L ( D ) represents the loss function of the discriminator. Sigmoid refers to the activation function.

3.2 LSTM-GAN model and animation system design

By building Attention-GAN, the analysis of the coloring effect in the system model can be completed. However, the generation of AA is not only about image coloring but also needs to analyze the generation effect and continuity of the image. Therefore, LSTM-GAN is added to AL-GAN to improve the image effect. Self-encoding structure, recurrent neural network (RNN), and GAN are added to LSTM-GAN.

The self-coding structure can efficiently express the features of input data while using unsupervised learning methods. Its structure includes an encoder and a decoder. The encoder hides variables from the input data, and the decoder programs the hidden data into their initial data form. The self-coding structure can simultaneously compress high-dimensional data, thus enabling operations such as dimensionality reduction of image data after using the encoder. Figure 4 shows the self-encoding structure.

Figure 4

Autoencoder structure (Self-drawn by author).

In Figure 4, in the self-coding structure, the input data are first encoded and then convolved. After multiple convolutions, the information conditions are established, and then reverse convolution is performed. After multiple convolutions, the data are output for decoding [24]. The RNN in LSTM-GAN is a structure that processes input data. Due to its ability to chain connect input image data, it is applied in LSTM-GAN, which has good robustness in modeling and data sequence processing. Figure 5 shows the RNN framework in the LSTM-GAN structure.

Figure 5

RNN framework in LSTM-GAN network structure (Self-drawn by author).

In Figure 5, when the image data sequence is input, the decoder in the network structure first processes the current data information. The processed data are transmitted to RNN and LSTM. After processing the image sequence, these data are input into the decoder. After completing the image data decoding process through the decoder, the processing and prediction of the current image data are completed [25]. The GAN in the model can connect RNN with the image generation network, as shown in Figure 6.

Figure 6

Structure of connected RNN and image generation network (Self-drawn by author).

In Figure 6, the structures of GAN and RNN are similar. However, it adds a new discriminator to the input and output image data to judge the authenticity of the currently generated image data, thereby improving the performance of the current network framework. The combination of RNN and adversarial generative networks can demonstrate better data learning performance, but it is prone to gradient explosion when dealing with long time series. Therefore, RNN is replaced with LSTM, which has the same structural framework as the Attention-GAN framework and a decoder. The decoder structure and construction process are the same as the above network structure. However, before converting the data, to maintain the characteristic of uniform image coloring, the network structure of this layer retains the dilated convolutional layer. This allows the network parameters to expand the receptive field without increasing, allowing its structure to receive more regional sensing. The normalization function of the decoding layer is selected as the IN-normalization function [26]. The activation function is the same as above. Due to the ability of LSTM to solve long-standing problems in data processing, the commonly used RNN has been replaced with LSTM. Figure 7 shows the structure of LSTM.

Figure 7

Long short-term neural network structure (Self-drawn by author).

In Figure 7, LSTM contains three structures: input gate, output gate, and forget gate. The activation function is mainly used to determine whether to delete or retain the cell state of the previous layer of input. The 0 means delete and 1 means retain. Eq. (4) is the expression of the current network model [27].

where W f , U f , b f represent the bias and coefficients in the current relationship. σ is the activation function. f t refers to the output quantity. h t − 1 represents the current hidden state. x t is the data input at the current time. The input gate structure is represented by Eq. (5) [28].

where W i , U i , b i are the bias and coefficients in the current relationship, and the other parameters are expressed the same as above. The new cell is represented by Eq. (6) [29].

where C ˘ t represents the current new cell information, and the expression of other parameters is the same as above. The rationale behind employing the hyperbolic tangent function (tanh) lies in its capacity for nonlinear activation, a property that facilitates the acquisition of the essential nonlinear characteristics necessary for neural networks to learn and simulate intricate function mappings. Additionally, it serves to circumvent the issue of gradient vanishing. When the old cell information is iterated into a new state, the iteration method is to select a part of the new cell state as the current new cell state through the input gate of the old cell state, represented by Eq. (7).

where C t represents the current new cell state. C t − 1 is the old cell state. C ˘ i refers to the cell state in the newly added input gate. The current cell state is determined by inputting the hidden cell state from the previous moment, represented by Eq. (8).

where o t represents the judgment condition for the activation function. W o , U o , b o are the coefficients and bias in the current relationship. The remaining parameters are the same as above. The new output unit obtained by transforming the cell state vector through the tanh layer is represented by Eq. (9) [30].

where h t represents the output at the current time. C t is the current state of the cell. The remaining parameters are the same as above. The decoder layer of this network structure can obtain feature images of different sizes by inputting data from different images. For example, when the input data size is 3 × 256 × 256, the output feature image size is 1,024 × 16 × 16. The current task of the network is to produce continuous AA. Therefore, when training the data, it is necessary to input continuous colored images together with another real image and line information that has been excavated and colored. The decoded data are separated by a decoder and then input together into LSTM to predict the fourth image feature map based on the information from three consecutive colored images.

The decoding layer structure of LSTM-GAN is similar to the decoding layer structure in Section 3.1. Its main task is also to transform multi-layer channel feature images into formats through convolution. Its structure also consists of four reverse convolutional layers and one parallel convolutional layer. Due to the feature images input into the decoder being more refined, a relatively small 3 × 3 convolution kernel is used in the two upper sampling groups, and the step size is decoded and set to 1 for feature decoding. In the remaining two sets of upper layer sampling convolution kernels, a size of 7 × 7 with the same step size is selected, and a feature image of 256 × 256 × 64 size is obtained through parallel convolution. Finally, the feature image of this size is compressed and output.

The LSTM-GAN discriminator uses two types of judgments: global and local judgments. Global judgment is employed to ascertain the global impact of the presently generated predicted image, to determine its continuity with the preceding image, and to identify any excessive inconsistency. Local judgment refers to randomly judging whether the generated effect image of the current image prediction is good and whether it meets the current expected judgment. The two judges are composed of downsampling layers that are utilized for the classification of feature images. The global discriminator consists of four sets of downsampling layers and one result sampling layer. The downsampling layer inputs the image data into the result sampling layer through dimensionality reduction processing. Afterwards, the feature image is transformed and its feature value is calculated through the result sampling layer. The structure of the local judge is the same as that of the global judge, except that the input feature image of the local judge is selected from the generated image. The loss function of LSTM-GAN consists of the adversarial loss function and reconstruction loss function. Eq. (10) is the adversarial loss function [31].

where L norm ( G , F ) represents the reconstruction loss function. x 1 , x 2 , x 3 are the image dataset input into the generator. x ˘ 1 , x ˘ 2 , x ˘ 3 refer to three image feature datasets with a size of 40 × 40. G represents the generator. F is a generator in Attention-GAN. The remaining parameters are the same as above. The adversarial loss function of LSTM-GAN is represented by Eq. (11).

where Y ˘ represents the dataset of the real image after coloring. Y ' is a true colored dataset with an image size of 40 × 40. GD refers to the global evaluator. LD represents a local discriminator. The loss function of the overall network is represented by Eq. (12) [32].

The parameters in Eq. (12) are consistent with the above. By constructing two network structures, the processing of AA images can be achieved. Inputting and processing image data can achieve the construction of the current AA system. When i = 1, 2, 3, 4 in Eqs. (10)–(12), at this point the formula represents the processed image feature dataset of size 40 × 40. The initial step involves the input of image data, which are then processed through a model. The resulting processed data are subsequently output to obtain the desired AA image.

During the model building process, the generator in GAN is responsible for generating data, while the discriminator is responsible for determining whether the data are true. At the same time, people are studying how to combine attention mechanisms to enhance the coloring effect of animated images. Some scholars have improved the model through the attention mechanism, forming Attention-GAN, and added feature image channels in Attention-GAN, using encoders to process the input building line images. There are also related studies that use converters to style transform the Dense-Net-BC network structure, converting feature data back to image data through the Attention-GAN decoder. In conclusion, the model incorporates a LSTM structure to enhance the smoothness and continuity of the image. Additionally, it utilizes unsupervised learning methods for data compression and dimensionality reduction. After inputting the architectural line image, the model first uses Attention-GAN to perform shading and supplementation operations, generating more complex AA image data. After generating animation image data, the data are input into LSTM-GAN, which processes it and outputs new animation data. Finally, the discriminator of LSTM-GAN determines whether the global effect of the generated predicted image is continuous with the above image and whether there is excessive inconsistency.

The real-time AA generation in practical system applications requires the model to be able to complete data processing and generation in a relatively short period of time. This requires the system to have a fast response time. Therefore, conventional standard hardware may not be able to meet this real-time requirement. Delay or lag may affect the smoothness and effectiveness of the animation. To realize efficient AA generation, high-performance graphics processing units (GPUs) or intelligent dedicated hardware are required.

To address the impact of dynamic automotive elements on the AA model, it is necessary to integrate more data, analyze time series, and process videos, and simulate dynamic changes. In addition, to adapt to constantly changing lighting conditions, the model should include lighting and shadow algorithms to ensure realistic animation lighting effects. Finally, the model needs to integrate information from multiple sources, enhance the attention mechanism to focus on key dynamic elements, and optimize the computational complexity for the generation of dynamic building animations.

4.1 Simulation results analysis for AA generation of models

Due to the high training requirements of the network model used in this study, corresponding training and testing sets were selected for testing its performance effects. In the course of the analysis, existing online datasets were utilized, and the research endeavored to generate real-world AA models through the use of algorithms and system models. The study used the NYU Depth V2 dataset, which was mainly derived from publicly available data information on the Internet, including some self-plotted images. The efficacy of AL-GAN in addressing diverse architectural styles was assessed through a comparative analysis of the algorithmic models’ animation generation outcomes under identical datasets and conditions. The input image size in the experiment was 256 × 256 × 3, with a step size of 1 and a learning rate of 0.0002 for the network model. The dataset included 1,000 feature images, evenly distributed into 500, which were divided into Dataset 1 and Dataset 2 for testing experiments. The study was conducted on two different configurations of GPU servers both using Windows 10 operating system. Both servers used Intel Xeon E5-2643 v3 processors and NVIDIA GeForce GTX 2080 graphics cards. Both systems used Python 3.5 programming language and TensorFlow framework in their testing environment. To test the testing effect of AL-GAN, the animation generation algorithm structures used in AL-GAN were compared, such as Line, Attention-GAN, LSTM-GAN, and ArchcolGAN for animation generation image processing time comparison in Figure 8.

Figure 8

Comparison of animation generation times for different dataset algorithms (Self-drawn by author). (a) Dataset 1. (b) Dataset 2.

In Figure 8, when comparing the animation generation times of the algorithms, ArchcolGAN and AL-GAN had similar animation generation times, both between 1 and 3 min. However, based on the overall data distribution, AL-GAN had a shorter animation generation time. Attention-GAN and LSTM-GAN had similar animation generation times, both between 2 and 4 min. Line had the longest animation generation time among these four algorithms, with a distribution of 5–7 min. The animation generation time of AL-GAN was shorter than the other four structures, indicating that it was more efficient in generating AA. Therefore, by combining models, the performance improvement of the current algorithm could be achieved. The figure shows the comparison results of different algorithms in animation generation time. The results show that AL-GAN has a shorter animation generation time than the other four structures, indicating its higher efficiency in generating AAs. To test the animation generation performance comparison of the current model, the more advanced models were compared. Visual geometry group (VGG), Dual generative adversarial network (DualGAN), and Cycle-consistent generative adversarial networks (CycleGAN) were tested at the same animation generation time in Figure 9.

Figure 9

Comparison of animation generation states using different algorithms (Self-drawn by author).

In Figure 9, at the same animation processing time, the animation generation state of AL-GAN was the best and closest to the real image, followed by CycleGAN and DualGAN. The animation image effect obtained from VGG training data simultaneously was the worst. Therefore, AL-GAN performed relatively well in processing image data and generating animations simultaneously. The figure shows a comparison of animation states generated using different algorithms under the same animation processing time. The animation generation state of AL-GAN is closest to real images, followed by CycleGAN and DualGAN, while VGG performs the worst. To test the animation generation performance of the current model, the above three models were compared with AL-GAN. The animation generation effect was tested under the same conditions without comparing the generation time, as shown in Figure 10.

Figure 10

Comparison of four algorithms for generating AAs (Self-drawn by author).

In Figure 10, when the same AA was processed by four different models, AL-GAN showed more pronounced detail processing in the image, resulting in a more vivid overall animation effect and richer overall line fullness. Among the other three algorithms, the line changes in CycleGAN were closer to those of AL-GAN, while the other two models had obvious shortcomings in processing line effects. As a result, AL-GAN was significantly better than the other three common animation processing models in terms of the effect processing of AA. The figure compares the effects of four different models on the same building animation. AL-GAN exhibits a more pronounced capacity for image detail processing, resulting in a more vivid overall animation effect and more intricate lines. To conduct ablation experiments on the current model, the network model used in the model was tested for animation effects, as shown in Figure 11.

Figure 11

Model ablation experiment test results (Self-drawn by author).

In Figure 11, different network models obtained different image animations when processing the same building image, and they had differences in the detail processing of AA. Line could only process the lines of the current image, but could not process the saturation of color details in the image. Although Attention-GAN, LSTM-GAN, and ArchcolGAN could handle image details, there were significant differences in processing effectiveness. AL-GAN was more realistic in the color processing of buildings, which had fuller and clearer line contours for AA. Therefore, AL-GAN was more suitable for generating animated images and had a better effect. The table lists the comparative results of the adaptability of four models in different scenarios. AL-GAN has the highest adaptability in the same scene, especially in park scenes where the adaptability reaches 99.99%. To test the adaptability of the current AL-GAN to different architectural styles, the three commonly used models were compared with AL-GAN for adaptability, as listed in Table 1.

Adaptability can reflect the degree of adaptation and matching of the current architectural image in this algorithm. The higher the adaptability, the better the animation generation effect of the scene, and the closer the animation is to the real image. In Table 1, among these four model comparisons, AL-GAN had the highest fitness in the same scenario, while the fitness of the model in the park scenario was 99.99%. This indicated that the animation effect processing results generated by the model in this scene were consistent with the actual values. VGG had the worst adaptability, with a 4.52% lower adaptability than AL-GAN in park scenes. At the same time, the test results of some scenarios have better testing accuracy. The use of the AL-GAN model can effectively improve the test results in different scenarios. Therefore, the research algorithm was more suitable for the current AA generation. To test the color recognition accuracy of the current algorithm, Figure 12 shows the accuracy of the four models mentioned above.

Figure 12

Comparison of color recognition accuracy between different algorithms in two datasets (Self-drawn by author). (a) Dataset 1. (b) Dataset 2.

In Figure 12, when comparing the accuracy of color recognition between two datasets, AL-GAN had a higher recognition accuracy. The accuracy curves of the four algorithms first increased with the increase in sample size and then tended to a relatively stable state. The accuracy of AL-GAN remained stable at around 94.25%, as the algorithm performance gradually stabilized. The recognition accuracy of the algorithm in Dataset 2 is higher, which may be due to the fact that the building images in Dataset 2 are more suitable for this algorithm. The highest accuracy comparison between AL-GAN and VGG was about 7.25%, which might be due to the significant difference in algorithm performance. The figure shows the comparison results of color recognition accuracy of four models in two datasets. AL-GAN has higher recognition accuracy, and as the sample size increases, the accuracy of the algorithm tends to stabilize.

4.2 Algorithm performance testing of AA generation algorithm

To compare the recognition errors of the same building data among several models, different models were compared, as shown in Table 2.

In Table 2, when comparing model errors, the AL-GAN model had the smallest error value under the same conditions. When comparing the root mean square error (RMSE), the difference between the AL-GAN error and the maximum model error was 6.19. When comparing the average percentage error, the difference between the AL-GAN error and the maximum model error was 6.63. When comparing the mean square error, the difference between the AL-GAN error and the maximum model error was 7.64. As a result, AL-GAN had a smaller error value and better model performance. RMSE is mainly caused by missing or blurred details in the generated images. Therefore, RMSE can be reduced by increasing the amount of training data during the model training process. Mean percentage error can be caused by color or luminance instability in the generated model. Therefore, errors can be reduced by introducing color regularization terms in GAN models or using augmented adversarial loss functions. The mean squared error may be caused by the model’s lack of ability to handle animation continuity. Therefore, introducing time constraints can reduce errors and make the animation generated by the model smoother and more consistent between consecutive frames. The table compares the error recognition results of several models on the same building data, and the AL-GAN model has the smallest error value under the same conditions, showing better model performance and stability. To draw parallels between the performance of disparate models in the generation of building images, a comparison was made between the similarity of the generated images. The closer the similarity was to 1, the closer the generation effect of the model was to the real situation. Figure 13 shows the obtained results.

Figure 13

Comparison of similarity among four models (Self-drawn by author). (a) Dataset 1. (b) Dataset 2.

From Figure 13(a), in the comparison of similarity, AL-GAN had a higher similarity. The similarity of VGG was relatively small, with a minimum similarity of −0.3 when the sample size was 50. This indicated that the model had poor performance in generating architectural images. From Figure 13(b), the image generation similarity of AL-GAN in dataset 2 was also high, which might be due to the better image generation performance of the current model used. To test the stability of the current model, the loss functions of the more advanced models were compared in Figure 14.

Figure 14

Comparison of loss functions among four algorithm models (Self-drawn by author).

In Figure 14, the loss functions of the four algorithms first decreased with the increase in iterations. After reaching a certain value, their loss function values began to stabilize. The loss function value of AL-GAN tended to stabilize at around 1.87. The loss function values of the other three models were higher than those of AL-GAN when they tended to stabilize. The AL-GAN loss function was 0.53 lower than DualGAN, 0.45 lower than CycleGAN, and 1.02 lower than VGG. Therefore, AL-GAN had better stability compared to the other three models. The loss function results of four algorithm models were compared in the figure, and the loss function value of AL-GAN tended to stabilize at a lower level, showing better stability than the other three models. The “perturbations” refer to the changes in model parameters during updates or the deviations between model predictions and actual values. The loss function values of all models decreased with the increase in iteration times, indicating that the model predictions gradually approached the true values. AL-GAN exhibited lower and more stable loss function values, indicating smaller prediction errors and better performance. At the same time, the high loss function values of DualGAN, CycleGAN, and VGG indicated that these models had large prediction errors and relatively poor performance. To test the effect situation of different model performances, the study compares the inference time, frame rate, throughput, and real-time factor of different algorithmic models. The inference time refers to the time taken by the model to make a prediction for a single input sample. For real-time applications, this time should be as short as possible. In video or animation processing, the frame rate is the number of frames per second that the model can process. A higher frame rate means smoother animation. Throughput refers to the amount of data the model can process per unit of time. The real-time factor is the ratio of the actual processing time to the ideal processing time (i.e., zero latency). A real-time factor less than or equal to 1 means that the system is able to process data in real-time. The results are obtained as shown in Table 3.

Table 3 shows that the AL-GAN model exhibits better model performance in different performance tests, with a minimum inference time of only 2.64 s and a maximum frame rate of 5,364 FPS for image processing. The throughput of the model can reach 3.54 GB/s, and the real-time factor percentage is closer to 1. The result indicates that the AL-GAN model has a better performance test effect.