Traditional landscape painting and art image restoration methods based on structural information guidance

3.1 Design and process analysis of image restoration algorithm combining edge restoration and GANs

The motivation behind this research is to address two critical issues in image restoration. First, traditional structural models have achieved significant success in edge-assisted image restoration, but their simplified edge-shape assumptions limit the restoration effects on atypical edges. Second, GANs show strength in image restoration, but there are shortcomings in image structure restoration. To overcome these limitations, this study proposes an image restoration algorithm that combines traditional methods and GANs. By introducing GANs for image structure estimation and utilizing edge information for assistance, the algorithm emphasizes the crucial role of edges throughout the entire restoration process. Additionally, the algorithm integrates the ideas of traditional structural information guidance and artistic image restoration methods, aiming to provide a more comprehensive and effective solution for the field of image restoration.

The process of the edge-assisted generative image inpainting algorithm represents a conditional GAN model, as described in [14]. The model takes damaged images and masks as conditions, comprising two fully convolutional networks generators: an edge restoration network and an edge-constrained image restoration network [15,16]. The edge restoration network focuses on repairing the edge information of the image, while the image restoration network combines the output of the edge restoration network with mask information to generate the final restored image. A discriminator is employed to distinguish generated images from real images. The training process aims to minimize the difference between generated and real images, improving the discriminator’s classification performance. The entire training process aims to restore missing or damaged images by introducing an edge information protection mechanism, preserving edge features in the generated images. The algorithm’s output is an image that is restored and preserves edge information, enhancing the recovery effects on damaged images by synthesizing the structural advantages of GANs [17,18].

To better train the GAN, the value range of the original image x is linearly scaled to the interval [ − 1 , 1 ] . A binary mask is used to indicate the damaged region Ω and the known region Ω ¯ . The mask resolution is the same as that of the original image, and the value m i at pixel position i of the mask is defined as follows:

The research focuses on two main objectives of using masks. First, masks serve as a mathematical representation, aiming in making image restoration tasks more user friendly by allowing selective restoration based on user preferences. Second, for the input requirements of CNNs, the damaged regions of an image are typically handled by filling them with pixel values to meet the network's regulations. For input in the form of image signals [19]. The architecture of CNN is shown in reference [20].

During this process, masks play a crucial role in identifying the damaged areas. By padding the damaged areas with a constant value of 1, representing white fill, the representation of the damaged image is effectively distinguished between known and damaged regions. This differentiation is achieved by introducing an additional input mask. After introducing the mask, the auxiliary algorithm can clearly identify and process the damaged parts, thereby successfully completing the image restoration task, as specifically shown in the following equation:

Here, ⊕ and ⊗ represent element-wise addition and multiplication at corresponding positions, m is the same value code as the image resolution. Simultaneously, the algorithm needs to extract edges e from the original image using an edge extractor, as specified in the following equation:

The processing strategy for edge information is the main focus, with a particular emphasis on selecting the holistically nested edge detector (HED) as the edge extractor. This extractor directly generates a real-valued edge map, unlike traditional binary edges, with values mapped to the [ − 1 , 1 ] range to maintain a data range similar to the original image. To adapt to the needs of image display, the edge map is adjusted to the [ 0 , 255 ] range and undergoes eight-bit quantization, where higher pixel values correspond to higher edge probabilities. It is important to note that the generated edge map is in real number form, not binary. Additionally, the concept of masks is introduced in the process of obtaining the damaged edge map. Through the mask’s effect, the expression of the damaged edge map can be derived from the original edge map

The approach in equation (4) is suitable for obtaining the damaged edge map during the training process. Simultaneously, using the damaged edge map E to fill the damaged regions of the edge map in the edge restoration network is done to meet the network’s requirements for a complete input [21]. This process involves using a real-numbered edge map, typically generated by an edge extractor and identifying damaged regions through a mask. The edge restoration network takes the edge map with damaged regions as input and, through its learning ability, fills these damaged regions by understanding and predicting the patterns of damage from the training dataset, ultimately generating effective restoration results. Let R _edg represent the edge restoration network, then the network’s output is as shown in the following equation:

In this process, the known condition m is introduced to the edge repair network, which assists the network in accurately distinguishing between damaged regions and known regions in the damaged image e _in [22]. The primary goal of the edge repair network is to generate an edge map within the damaged area, but it also outputs content in the known region, which may differ from the input edge map of the known region. Given that the known region of the damaged edge map is accurate, the final repaired edge map can be obtained by combining the output e _out of the edge repair network with the damaged image, as expressed in the following equation:

The repaired edge map plays the role of prior information in this algorithm and is introduced as input, delivered to the image restoration network with edge constraints, as specifically shown in the following equation:

Here, the known condition m is introduced to the image restoration network, assisting the network in accurately delineating the damaged and known areas in the damaged image y [23,24,25]. By combining the damaged areas in the output x _out of the image restoration network with the known areas of the damaged image y under edge constraints, the final restored image is obtained, as represented by the following equation:

In the final stage, a tensor with a channel number of 4 is generated by concatenating the repaired edge map and the restored image in the channel dimension. This tensor serves as input to the GAN for determining its authenticity [26]. In terms of sample classification, the concatenation of the original edge map e and the original image x is treated as “real” samples, while the concatenation of the repaired edge e and the repaired image is treated as “fake” samples for training and evaluating the performance of the GAN.

3.2 Embedding multi-scale attention dilated convolution (MADC) in landscape painting image restoration model

The image repair algorithm that fuses edge repair and generative adversarial networks can achieve accurate repair of missing parts of the image by combining edge information and the powerful generation capabilities of the generative adversarial network. The utilization of edge information makes the repair results more accurate and natural, while the introduction of generative adversarial networks enhances the robustness and generation capabilities of the repair model. This section proposes a landscape painting image restoration model embedding multi-scale attention dilation convolution. This model introduces a multi-scale attention mechanism in the repair process to better capture important information at different scales in the image, helping the repair results to be more detailed and accurate. At the same time, the use of dilated convolution also enhances the receptive field of the model, helping to better understand the semantics and structure of the image. Therefore, the study proposes an improved generative adversarial network model, the core feature of which is to embed a multi-scale attention dilation convolution structure [27,28]. The overall framework of the model includes an encoder–decoder network as the generator and two branches of discriminator – global and local. The generator’s task is to generate restoration images with natural coherence, and through adversarial training, the discriminators assist in generating more credible restoration images. The structural design of the model aims to more effectively meet the specific requirements of artistic image restoration. Figure 1 vividly presents the overall architecture of the model, emphasizing the crucial properties of embedding MADC.

Figure 1

An artistic image restoration model framework embedding MADCs.

In image restoration, a GAN structure has been proposed with a generator employing an encoder-decoder architecture. The feature encoding network of the generator consists of a series of 3 × 3 convolutional layers and multi-scale attention-dilated convolutions [29]. The first convolutional layer employs a 5 × 5 kernel to extract latent information features from the input image. The introduction of the MADC module, combining dilated convolutions and attention mechanisms, enlarges the receptive field and effectively extracts relevant features of interest. The combination of 3 × 3 convolutional layers and multiple attention-dilated convolution modules, as shown in Table 1, preserves more image feature details during the encoding process.

The structure of the decoding network includes a 3 × 3 convolutional layer, three upsampling layers, and a 3 × 3 convolutional layer reused. The last two upsampling layers use ReLU activation functions, while the Tanh activation function is applied after the final convolutional layer [30]. Through the combination of these layers, the decoding network maps the features extracted from the encoding network to the generated restored image. The MADC, serving as a plug-and-play optimization module, integrates features of hybrid dilated convolution blocks and coordinated attention layers, as depicted in Figure 2. Its primary advantages lie in achieving expanded effects without increasing computational complexity through the introduction of dilated convolution blocks and effectively suppressing interference from irrelevant information through the application of coordinated attention layers.

Figure 2

MADC model.

The dilated convolution block, as the core component of the MADC module, draws inspiration from the aesthetic background of traditional landscape paintings and artistic image restoration. In traditional painting, artists achieve precise expression of depth and details through meticulous depiction and the use of perspective techniques. Similarly, in the field of artistic image restoration, special attention to microscopic details in the area of interest is required to restore the natural realism of the image while avoiding the introduction of additional noise or artifacts during the restoration process. The design of the dilated convolution block aims to expand the receptive field while maintaining computational efficiency. By employing dilated convolutions, the receptive field of the convolutional kernel is effectively expanded, inspired by traditional art techniques that create a sense of depth through the drawing of points, lines, and surfaces. In the context of the MADC module, the introduction of the dilated convolution block can be understood as mimicking the attention to detail in traditional art, allowing the network to observe and understand the image structure more comprehensively, thereby producing a more natural and artistically appealing restoration effect.

The application of coordinate attention mechanisms in neural networks, especially in traditional landscape paintings and artistic image restoration, holds significant significance. Traditional landscape paintings emphasize the expression of depth through specific painting techniques, which is equally important in artistic image restoration, particularly in the preservation of key details. The coordinate attention mechanism simulates the way traditional art focuses on details in the landscape by sensitivity to positional information. This enables the neural network to more accurately understand the relationship between positions and channels, enhancing its ability to capture contextual information in the image.

3.3 Design of the network structure and loss function for the artistic image restoration model guided by structural information

Compared with ordinary image restoration, the restoration of artistic images pays more attention to protecting the artwork itself and also needs to consider the unique structure and style of the artwork. By designing a specific network structure and loss function, the research can better retain the original artistic style and structural features when repairing artistic images. Relatively speaking, the solution provided is more objective and reliable and can improve the efficiency and accuracy of repair. The proposed artistic image restoration model guided by structural information consists of an encoder, an AOT module, a multi-scale feature fusion module, and a decoder. The input comprises the image to be restored and the structural information of the artistic image generated by the HED model based on CNNs. The encoder extracts texture features (F) and structural features (F′), while the multi-scale feature fusion module fills local damaged areas at multiple scales through dilated convolutional layers. The AOT block effectively captures global features, enabling the generator to acquire meaningful guidance from distant image contexts and interesting patterns. The decoder generates naturally plausible restored images through multiple layers of transposed convolution, synthesizing global and local information to enhance the restoration effect, as shown in Figure 3.

Figure 3

Image repair structure diagram based on structure generation and texture filling.

To better address the restoration of missing regions, a multi-scale feature fusion module is introduced in the study. This module aims to comprehensively consider missing areas in complex situations, synthesizing existing regions from different scales and perspectives to better understand which areas are beneficial for filling holes. Unlike the dilated convolution blocks in Section 3, the multi-scale feature fusion module captures features at different semantic levels through the encoding of two branches: texture features (F) and structural features (F′). It models long-range spatial dependencies by enhancing the correlation between features at different levels, ensuring the overall consistency of the restoration results. It is noteworthy that this module performs well in handling more challenging scenarios, such as scale variations, striking a balance between improved accuracy and reduced complexity (Figure 4).

Figure 4

Multi-scale feature fusion structure diagram.

To obtain multi-scale semantic information, the study introduces four sets of dilated convolution layers with different dilation rates. The purpose of this design is to capture semantic information at different scales for a more comprehensive understanding of the image structure and context. Through equation (9), these dilated convolution layers reconstruct feature maps, thus modeling long-range spatial dependencies at multiple scales.

Here, Conv _k represents a dilated convolution layer with a dilation rate of k, and k ∈ { 1 , 2 , 4 , 8 } . In the study, to more effectively gather multi-scale semantic features, a feature value weight mapping generator is introduced. The design of this generator includes two convolution layers with kernel sizes of 3 and 1, respectively, followed by ReLU non-linear activation functions, generating a total of 4 output channels. The core objective of this generator is to predict the weight distribution of multi-scale semantic features, guiding the image restoration process more effectively. The specific expression of the weight map generated by the generator, calculated pixel-wise, is detailed in the following equation:

Soft max in equation (10) refers to the Soft max operation performed on the channel level.

Slice in equation (11) means the Slice operation on the channel level. Finally, by combining and weighting the multi-scale semantic features through element-wise addition, a detailed feature map is formed, as shown in the following equation:

Here, F c represents the refined feature map. To improve the training efficiency of the generator and generate restored images of higher quality perceived by the human eye, five main loss functions were studied, weighted, and summed to form the overall loss function of the generator, as shown in the following equation:

Here, L _rec, L _sty, L _cont, and L _tv represent the reconstruction loss function, style loss function, content loss function, and the total encoding loss function, respectively. λ _adv represents the adversarial loss function. Meanwhile, λ _rec, λ _cont, λ _sty, λ _tv, and λ _adv are hyperparameters for each loss function. The reconstruction loss function is defined in the following equation:

Equation (14) refers to I ^out as the image to be restored and I ^gt as the ground truth image. The content loss function is computed as shown in the following equation:

Here, Φ l represents the feature map extracted by the l -th network pooling layer. h l , w l , c l represent the channel height, length, and width, respectively. The style loss function is given by the following equation:

Here, M G r a m ( I ) = Φ l ( I ) − Φ l ( I ) is the style image. The total variation loss function is expressed by the following equation:

Here, m , n correspond to the pixel position information of the output image. The adversarial loss function is defined in the following equation:

Here, p _data (I ^gt) represents the distribution of real images, while p _data (I ^out) represents the distribution map of restored images.

4.1 Validation of multi-feature fusion image restoration techniques and image quality detection model performance

The study utilizes the Chinese-Landscape-Painting dataset to evaluate model performance, comprising 2,192 images of traditional landscape paintings. To address the limited nature of current art image datasets and ensure comprehensive model validation, an innovative image segmentation method was employed to augment the dataset. Through this approach, the research team successfully segmented each landscape painting image into multiple sub-images, obtaining a more diverse set of training samples. This augmentation method not only enhanced the model’s generalization ability across various scenarios but also provided more comprehensive and specific artistic image information for training and validation, further enhancing the credibility and practicality of the study.

As shown in Figure 5, in order to expand the data set to verify the effectiveness of the model, the study used traditional Chinese high-definition landscape paintings and cut them into multiple images with rich content. Finally, 21,000 images were collected, and the size of each image was 256 × 256 × 3. In order to increase the randomness and uncertainty of the damaged area, the position of the mask in the image is random, and the size of the damaged area is also random.

Figure 5

Masked data samples at different scales: (a) 0–20%, (b) 20–30%, (c) 30–40%, and (d) 40–50%.

As shown in Figure 6, the study conducted image restoration under different mask proportions. Observations revealed that under mask proportions of 0–20%, the restored results were nearly indistinguishable from the real images, maintaining structural consistency and reasonable textures. However, as the damaged area increased, a noticeable loss of semantic information occurred, leading to discontinuous lines and blurred details, resulting in a decrease in consistency with real images. This trend is less pronounced in Figure 6(a) and (b) but becomes more evident in Figure 6(c) and (d).

Figure 6

Image repair results for irregular areas: (a) 0–20%, (b) 20–30%, (c) 30–40%, and (d) 40–50%.

As shown in Table 2, the proposed image restoration method demonstrates superior quality in pixel, structural, and perceptual aspects compared to existing methods. Particularly noteworthy is its excellent performance in filling irregular holes under different mask scales. This outstanding result is attributed to the innovative multi-feature fusion technique, skillfully integrating local information of structure and texture features while maintaining efficiency in considering the overall context of the image. The introduction of skip connections in the U-Net network and deep semantic information in the decoder successfully combines local and global feature information, resulting in higher-quality restored images.

In Figure 7(a), the comparative results of SROCC for various models indicate that the proposed image detection model achieved an SROCC of 0.97. This is an improvement of 0.20 compared to the JNB model, 0.07 compared to the CPBD model, and 0.06 compared to the MLV model. Figure 7(b) displays the PLCC comparison results, with the proposed image detection model having a PLCC of 0.97. This is higher by 0.17 compared to the JNB model, 0.08 compared to the CPBD model, and 0.02 compared to the MLV model, suggesting superior detection performance of the model.

Figure 7

PLCC and SROCC comparison results of different algorithms: (a) PLCC of each method and (b) SROCC of each method.

4.2 Comprehensive evaluation and comparative analysis of performance for Chinese landscape painting image restoration algorithms

Figure 8 illustrates the error distribution in the edge restoration task between the proposed method and the Canny algorithm. As shown in Figure 8(a), the errors of the research method are mainly within the range of 0.000–0.015. In contrast, Figure 8(b) demonstrates that the errors of the Canny algorithm are concentrated between 0.020 and 0.051. This clearly indicates the more significant performance advantage of the proposed method in edge restoration, with lower error levels compared to the traditional Canny algorithm.

Figure 8

Scatter plot of matching errors between different methods: (a) our method and (b) Canny algorithm.

Figure 9(a) presents the PRSN values of various comparative algorithms, with the proposed graphic processing algorithm achieving the highest PRSN of 36.7 dB. Compared to mean-filter-based image processing algorithms, it is higher by 6.3, 9.1 dB higher than wavelet denoising-based algorithms, and 15.8 dB higher than median-filter-based algorithms. In Figure 9(b), SSIM values of different algorithms are displayed, with the proposed graphic processing algorithm having the highest SSIM of 0.85. It surpasses mean-filter-based algorithms by 0.06, wavelet denoising-based algorithms by 0.12, and median-filter-based algorithms by 0.11.

Figure 9

Comparison results of image processing algorithms PRSN and SSIM values: (a) the PRSN results for each algorithm and (b) the SSIM results for each algorithm.

Figure 10(a) illustrates the structural similarity results of algorithms in the landscape painting image restoration task. It distinctly shows that the proposed model excels in image restoration, with an average increase of about 9.3% in the structural similarity index compared to other algorithms. Importantly, its fluctuation is relatively small, indicating the algorithm’s ability to preserve more detailed structural information, ensuring stability and reliability in the restoration effect. In Figure 10(b), the information entropy results show that the proposed model performs exceptionally well in this metric. It averages about a 3% increase compared to other algorithms, emphasizing its authenticity in landscape painting image restoration.

Figure 10

Different algorithms have similar repair effects and structures: (a) structural similarity and (b) information entropy.

Table 3 shows that the image restoration algorithm proposed in the study is significantly better than other algorithms in terms of signal-to-noise ratio, average gradient, structural similarity, and information entropy. Among the repair effects of the four groups of pictures, the signal-to-noise ratio of the image repair algorithm proposed in the study is more than 20, indicating that the ratio between the signal and noise of the repaired image is very high, showing the clarity of the repair effect and good image quality. The structural similarity indicators all reach above 0.9, which shows that the repaired image has a high structural similarity with the original image, and the repair algorithm successfully retains more structural features and detailed information about the image. At the same time, the information entropy exceeds 15, which indicates that the repaired image has a higher amount of information, higher information richness, and a more natural and accurate repair effect.