Super-resolution reconstruction method of the optical synthetic aperture image using generative adversarial network

Jing Chen; Aileen Tian; Ding Chen; Meng Guo; Dan He; Yuwen Liu

doi:10.1515/phys-2023-0194

Article Open Access

Super-resolution reconstruction method of the optical synthetic aperture image using generative adversarial network

Jing Chen , Aileen Tian , Ding Chen , Meng Guo , Dan He and Yuwen Liu

Published/Copyright: April 4, 2024

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Open Physics Volume 22 Issue 1

Abstract

In order to solve the contradiction between large aperture elements and high-resolution images, in this study, we propose an improved image-resolution method based on generative adversarial network (GAN). First, we analyze the imaging principle of the optical synthetic aperture. Further, we improve a super-resolution GAN; especially, this network uses a multi-scale convolutional cascade to obtain global features of the image, and a multi-scale receptive field block and residual in residual dense block are built to obtain image details. In addition, this study uses the Mish function as the activation function of the discriminator to solve the problems of neuron extreme, gradient explosion, and poor generalization ability of the model. Through simulation, the results show that the proposed method can achieve a peak signal-to-noise ratio (PSNR) of 30 dB compared with traditional image super-resolution reconstruction methods for synthetic aperture image. The method proposed has an improvement of 2 dB in the PSNR and 0.016 in structure similarity index measure compared with the original super-resolution GAN. Therefore, this method can effectively reduce the image distortion and improve the quality of image reconstruction.

Keywords: super-resolution reconstruction; generative adversarial networks; multi-scale convolution cascade; feature extraction; optical synthetic aperture

1 Introduction

In general, for single aperture telescopes, higher spatial resolution requires a larger aperture [1]. Therefore, higher spatial resolution of the optical system must use entrance pupil with the large diameter. For an optical synthetic aperture by combining sub optical systems, it is equivalent to a single large aperture and is widely used in various systems of different modes such as holography, astronomy, remote sensing imaging, etc. By increasing the effective aperture, spatial resolution can be improved. Accordingly, the system resolution is no longer limited by the size of a single aperture and the contradiction between large-aperture elements and high-resolution image can be effectively resolved. Unfortunately, due to the discrete distribution of sub apertures, the pupil function is no longer a connected domain, resulting in a significant decrease in the frequency response of the modulation transfer function (MTF). This means that image information, especially intermediate frequency information, is suppressed or lost. This reduces the image quality and causes the image to be blurred. It is therefore essential to use an effective image restoration method.

So far, many investigations have been carried out in the field. With the rapid development of artificial intelligence [2], super-resolution technology (SRT) has become the mainstream technologies of digital image processing technology [3]. Besides, the field of image SRT has very high guiding significance, great development potential, and application prospects. Currently, more and more scientific researchers are devoted to the super-resolution image reconstruction based on deep learning, in all kinds of fields such as military field, medical image diagnosis [4,5], and agricultural remote sensing monitoring [6]. Dong et al. proposed a super-resolution convolutional neural network (SRCNN) based on the classical convolutional neural network model [7]. This network uses a three-layer convolutional neural network to achieve end-to-end mapping between low-resolution image and high-resolution image. However, the convergence speed is slow and the training cost is high. Zhang et al. proposed a residual dense network that can fully fuse all hierarchical features in low-resolution images [8]. The algorithm achieved good reconstruction results, but the number of model parameters was too large and there was feature redundancy, and so on. Sha et al. built a parallel residual network by changing the residual network structure [9], which improved the feature extraction capability and accelerated the network training speed. Zhou et al. proposed the super-resolution reconstruction method based on recursive residual network [10]. This method relieves the pressure of deep network training by constructing a local residual network. Meanwhile, an adjustable gradient pruning method is used for optimization to prevent gradient dispersion during deep network training. Tang et al. used U-net convolutional network for image restoration of three-arm optical synthetic aperture. This network can achieve fast blind image restoration [11]. The U-net convolutional network can effectively restore the images formed by the system after training. It has the advantages of strong restoration ability and low dependence on the training set. However, there are some drawbacks such as complex models and low universality. Qiao et al. proposed a network model based on Fourier domain attention. It was applied to microscope images and extended the application field of SRT [12]. Hu et al. proposed a progressive generative adversarial network (GAN) for face super-resolution reconstruction [13], using a progressive generative method to ensure the stability of the training process by dividing the training into stages. Sun and Chen used a multi-scale residual network [14], which allowed the reconstructed images not to be limited to a fixed scale, and thus efficiently trained a network model for arbitrary multiples of the reconstruction. Chen et al. proposed an adaptive field of view multi-scale GAN image restoration method based on coordinate attention. By introducing a deformable encoder into the generator, the local visual consistency of image restoration has been improved. Simultaneously combining coordinate attention mechanism with convolutional layers expands the receptive field of deep networks and global perspectives [15]. Chen et al. proposed an improved image restoration network based on multi-scale feature modules and improved attention modules, which further enhances the semantic restoration ability of images. However, the restoration effect is poor for complex scene images [16]. Chen et al. proposed a lightweight method that combines group convolution and attention mechanism to solve the problem of information mobility between channels in traditional convolutional processing [17]. Ning et al. studied the image restoration problem of optical synthetic aperture systems using variational physical information networks, effectively improving the quality of image restoration [18]. Li et al. proposed a new optical guided super-resolution network for large-scale synthetic aperture radar images, which considers both low-frequency information from low resolution synthetic aperture radar images and feedback evaluation from high-resolution optical images [19]. At present, certain achievements and progress have been made in the research of image super-resolution reconstruction, but most of it is aimed at traditional natural images. When facing remote sensing images with richer details and complex features, the various current methods still have some shortcomings in the reconstruction process, such as deep network layers, unstable network, and slow convergence speed. This study proposes an improved super-resolution generative adversarial network (SRGAN) to address the issues of missing detail information, insufficient feature extraction, and blurred edges in the reconstructed remote sensing images.

In this study, we propose a super-resolution reconstruction network based on the integrated multi-scale receptive field block (RFB) to GAN, and it can effectively solve the problem of image blur in the optical synthetic aperture imaging system. Multi-scale refers to the sampling of signals at different scales. Different features can be observed at different scales. Multi-scale refers to scaling the output feature maps of different convolutional layers to a unified size, so that they contain both global and local detail information. The multi-scale convolution cascade, the multi-scale RFB, and the residual in residual dense block (RRDB) are used to enhance the ability of the generated network to extract detailed features. Besides, the Mish function is used in the discriminative network instead of the original LeakyReLU function, and it can solve the problems encountered in model trainings, such as neuron extremism, gradient explosion, and poor generalization ability. Finally, some images from the NWPU VHR-10 dataset were simulated using the proposed algorithm and the traditional super-resolution reconstruction algorithm, and the quality of image after reconstruction can be evaluated using the peak signal-to-noise ratio (PSNR) and the structure similarity index measure (SSIM). For this study, the main contributions are as follows:

In response to the problem of super-resolution reconstruction of remote sensing images, the residual module has been improved to more accurately identify the features of remote sensing images. Applying it to the generative network in adversarial networks can improve the operational efficiency of the network.
Improved the receptive field of the feature network by changing the activation function while further enhancing the model’s generalization ability by combining convolutional kernels of different scales, effectively preventing gradient vanishing and explosion.

2 Method

2.1 Imaging principle of the optical synthetic aperture

Generally, optical synthetic aperture is a sparse aperture array structure. It consists of multiple small aperture optical systems arranged in a certain way. Therefore, it is also known as sparse aperture imaging technology. According to the imaging theory of Fourier optics [20], the point spread function (PSF) indicates the imaging performance of the optical system in the spatial domain. Likely, the optical transfer function (OTF) and the MTF indicate the imaging performance in the frequency domain. The relationship between them is shown in Figure 1, where F is the Fourier transform.

Figure 1

Relationship between pupil function, OTF, PSF, and MTF.

For a single aperture system, the circular pupil function can be expressed as

(1) P ( x , y ) = circ x 2 + y 2 D / 2 = 1 , x 2 + y 2 ≤ D / 2 0 , others ,

where D is the diameter of the exit pupil, and circ is a circular domain function.

For a two-dimensional synthetic aperture imaging system, the pupil function P _array(x, y) is

(2) P array ( x , y ) = ∑ n = 1 N P sub ( x − x n , y − y n ) e i φ n ( x , y ) ,

where N is the number of sub apertures, and (x _n , y _n) is the center position of the pupil for the nth sub aperture, φ _n is the phase shift of the beam when it reaches the nth sub aperture, and P _sub is the pupil function of the sub aperture.

If Eq. (2) is transformed by the Fourier transform, then the amplitude diffusion function of the synthetic aperture system is given by

(3) A array ( u , v ) = A sub ( u , v ) ⋅ ∑ n = 1 N e − i 2 π u λ f x n + v λ f y n ,

where λ is the wavelength, f is the focal length, (u, v) are the pixel coordinates, and A _sub is the amplitude diffusion function of the pupil, and its expression is

(4) A sub = π D 2 4 λ f J 1 ( π D r / λ f ) π D r / λ f ,

where r = (u ² + v ²)^1/2, and J ₁ is the first order Bessel function.

The PSF_array of the synthetic aperture system is the square of the modulus of the amplitude diffusion function, and it is expressed by

(5) PSF array ( u , v ) = PSF sub ( u , v ) N + 2 ∑ n = 1 N ( N − 1 ) / 2 cos 2 π λ f ( Δ x n u ) + Δ y n v ,

where (Δx _n, Δy _n) represents the distance between the coordinates of the centers of two sub-apertures, and PSF_sub can be expressed as

(6) PSF sub = | A sub | 2 = D J 1 ( π D r / λ f ) 2 r 2 .

Similarly, PSF is transformed by Fourier transform, OTF can be represented as

(7) OTF = H ( f x , f y ) = Fourier { PSF arry ( u , v ) } ∫ ∫ PSF arry ( u , v ) d u d v ,

where f _x and f _y represent the spatial frequencies.

Using the autocorrelation of the pupil function to obtain the OTF, the MTF is the modulo of the OTF, and then it can be expressed as

(8) MTF array ( f x , f y ) = MTF sub ( f x , f y ) × δ ( f x , f y ) + MTF sub ( f x , f y ) × 1 N ∑ n = 1 N ( N − 1 ) / 2 δ f x ± Δ x n λ f , f y ± Δ y n λ f ,

where MTF_sub(f _x , f _y) is the MTF of a sub aperture, and * is a convolution.

(9) MTF sub ( ρ ) = 2 π arccos λ f D ρ − λ f D ρ 1 − λ f D ρ , ρ ≤ D λ f 0 , ρ > D λ f ,

and ρ = f x 2 + f y 2 .

As shown in Figure 1, the pupil function affects the imaging quality of the optical system. For a single aperture optical system, if the given aperture and wavelength, it will be a low-pass filter. The high-frequency component is cut off by optical systems. Generally, this affects the resolution of the image. However, the pupil function is related to the pupil structure. Therefore, the MTF can be further improved by optimizing the size and position of each sub aperture in the synthetic aperture system. Further, the resolution of the system can be improved. Figure 2 shows the PSF and MTF of single aperture and synthetic aperture systems of equal area. In Figure 2, the solid line represents the synthetic aperture systems, and the dashed line represents the single aperture system. According to the PSF of a single aperture system, the dashed line can be obtained as shown in Figure 2(a). According to the Rayleigh criterion, as the diameter of the aperture increases, the diameter of the Airy spot decreases, and the half width of the main peak in the dashed line of the graph narrows. Likely, as the aperture diameter decreases, the diameter of the Airy spot increases, and the half width of the main peak in the dashed line of the graph increases. It can be seen from Figure 2(a), the diameter of the Airy disk is narrow for the synthetic aperture system, but the secondary peak increases. The autocorrelation operation of the Pupil function is changed to OTF, and MTF is the modulus of OTF. When the MTF first reaches 0, the corresponding spatial frequency is the cutoff frequency. In Figure 2(b), the cut-off frequency of the MTF becomes higher. This preserves the high-frequency information of the image. As a result, the image resolution is enhanced.

Figure 2

Image principles: (a) PSF and (b) MTF.

Currently, synthetic aperture imaging systems are widely used in environmental resource monitoring, disaster monitoring, maritime management, and military applications. Their application scenarios dictate that there are many types of sub-aperture structures, such as circular structures, three-arm structures, and Golay structures. The schematic diagram of common structures is shown in Figure 3. For the Golay structure, these sub aperture systems are arranged on an equilateral triangular mesh with the sub aperture diameter as the side length. The diameter of sub apertures is related to the filling factor of sparse aperture array systems, and the distribution of MTF curves varies according to the filling factor. By comparing the MTF curves with different filling factors, the optimal filling factor can be selected to determine the diameter of the sub aperture. The position of the sub aperture is determined by its diameter and the diameter of the circumscribed circle (the fill factor is the ratio of the total area of each sub-aperture to its circumscribed area).

Figure 3

Schematic diagram of sub aperture structure: (a) circular structure, (b) three-arm structure, and (c) Golay structure.

The value of the fill factor indicates the degree of sparsity of the structure of the synthetic aperture array. Meanwhile, the transfer function and the SNR of the system will also decrease. Increasing the fill factor improves the SNR for a given system quality. Tables 1 and 2 show the maximum and minimum fill factors for circular, three-arm, and Golay structures at three aperture numbers.

Table 1

Maximum fill factor corresponding to the number of sub apertures

N		6	9	12
Maximum fill factor	Circular	66.67%	58.46%	50.73%
	Three-arm	34.76%	23.76%	18.05%
	Golay	36.49%	13.88%	8.95%

Table 2

Minimum fill factor corresponding to the number of sub apertures

N		6	9	12
Minimum fill factor	Circular	30.11%	21.34%	16.76%
	Three-arm	10.58%	6.61%	5.47%
	Golay	15.19%	5.21%	3.22%

According to Tables 1 and 2, regardless of the number of pore sizes, the maximum fill factor and minimum fill factor of the circular structure are higher than the three-arm structure and the Golay structure. In addition, the circular structure has a more uniform arrangement than the three-arm structure and the Golay structure, and has a larger assembly margin. In general, the circular structure is easy to process and adjust and has a wide range of applications. Therefore, the circular structure is chosen in our study.

In this study, the seven-hole circular structure of the synthetic aperture is investigated and the grey scale image of the synthetic aperture is reconstructed. According to the sub-aperture coordinates and the pupil function of the seven-hole circular structure, the pupil arrangement and MTF distribution of the seven-hole circular structure can be obtained, as shown in Figure 4.

Figure 4

Circular-shaped seven-aperture structure. (a) Pupil function; (b) PSF; and (c) MTF distribution.

Figure 4 shows that the main peak of the synthetic aperture MTF is obvious, the secondary peak is distributed according to the regular distribution, and the intermediate-frequency and low-frequency decrease rapidly. Therefore, the loss of intermediate-frequency and low-frequency information is the main cause of image blur.

Through analysis, it can be concluded that remote sensing images obtained using synthetic aperture systems have richer details and more complex features. Traditional image reconstruction methods suffer from problems such as deep network layers, unstable networks, and slow convergence speed. This article mainly addresses the problems of missing low-frequency detail information, insufficient feature extraction, and blurred edges in synthetic aperture imaging images.

The OTF expresses the spatial frequency response characteristics of the system. As shown in Figure 1, the OTF can be obtained after P(x, y) autocorrelation and normalization for an ideal optical system. According to the Fourier transform law of synthetic aperture imaging, synthetic aperture imaging can be simulated. As shown in Figure 5, the simulated image of the synthetic aperture corresponds to different fill factors.

Figure 5

Synthetic aperture simulation imaging: (a) original figure; (b) F = 0.35; (c) F = 0.45; (d) F = 0.55; and (e) F = 0.64.

When the fill factor number is small, the optical system receives less light, and this brings about a blurred image and a lack of detailed features such as texture. Figure 5 shows that the imaging law of the optical synthetic aperture is fully satisfied. As the fill factor increases, the image blur decreases, but the resolution of the outline features remains low. Table 3 shows the image quality evaluation after comparing the simulated image with the original image.

Table 3

Quality evaluation of simulation images of the ring with seven holes

Filling factor	PSNR	SSIM
F = 0.35	20.083	0.511
F = 0.45	20.546	0.516
F = 0.55	21.037	0.602
F = 0.64	21.173	0.687

Table 3 shows that the smaller the fill factor, the worse the image quality and the lower the resolution, so image restoration processing is particularly necessary. In this study, a simulation dataset of the annular seven-hole synthetic aperture system with a fill factor of 0.35 is constructed and the image restoration technique is studied.

2.2 Traditional SRGAN

The traditional SRGAN network model consists of two parts, the generator and the discriminator. The first part is the generator, and the network model of the generator is SRResNet. The network structure diagram is shown in Figure 6 [21,22]. The input is low-resolution image, which is passed through the convolution and activation layers to obtain the feature map. The residual blocks take the obtained feature map in turn through B residual blocks for deep extraction to obtain a new feature map. In this case, the residual block consists of the convolution layer, BN layer, and PReLu activation layer. Then, the new feature map is up-sampled to enlarge the image to obtain the super-resolution image. Finally, the multidimensional feature map obtained by up sampling is down-sampled by the convolution layer to output the three-dimensional super-resolution image.

Figure 6

Generator network.

The second part of the SRGAN network model is the discriminator. The network model structure diagram is shown in Figure 7 [21,22]. The super-resolution image and high-resolution image are, respectively, entered into the discriminator. First, the high-resolution image enters the activation layer after passing through the convolution layer. Second, the high-resolution image passes through N convolutional layers, BN layers, and activation layers, its feature map is mean-pooling. Furthermore, after two convolutional layers, the activation layer outputs the quality score of the high-resolution image.

Figure 7

Discriminator network.

The GAN is based on ideas from game theory and uses a generator G and a discriminator D for adversarial learning to generate sample targets. Figure 8 shows the training process for super-resolution reconstruction. The input low-resolution image is passed through the generator G to generate the super-resolution image, and the high-resolution image is sent to the discriminator D along with the super-resolution image, and the discriminator results are fed back to the generator network and the discriminator network to update both the generator G and the discriminator D. The generator will continue to learn and try to generate the same super-resolution image as the high-resolution image, and the discriminator will try to distinguish the super-resolution image from the high-resolution image to facilitate efficient learning of the generator. When the discriminator network D is unable to distinguish between the two, an equilibrium state is finally reached and the reconstructed super-resolution pixel image is obtained.

Figure 8

The process of SRGAN training for the reconstruction of the image.

2.3 Improved SRGAN

2.3.1 Generation network

To address the current problems of synthetic aperture imaging for the reconstruction of image intermediate-frequency detail information, in this chapter, a super-resolution reconstruction model with multi-scale RFB for GAN based on SRGAN is proposed. First, the multi-scale convolutional cascade is used to improve the acquisition of the global features of the image, and then the BN layer is removed to improve the reconstruction effect of the images and reduce the computational complexity. Second, in order to improve the reconstruction quality of the network and obtain more detailed texture information, the multi-scale RFB and RRDB are used to extract detailed features of the generated network. An improved generation network is shown in Figure 9. The RFB network structure is shown in Figure 10. The generative network consists of a global feature extraction module, a detail feature extraction module, and an image reconstruction module. The global feature extraction module aims to obtain more global features for the subsequent detail feature extraction. Then, detail feature extraction module aims to obtain more intermediate-frequency information. And the image reconstruction module aims to integrate the reconstructed feature information obtained from the previous part by multiple convolutions. In order to better adapt to the requirements of remote sensing high perception information, the serial convolution structure is constructed using convolution kernels of different scales (7 × 7, 5 × 5, and 3 × 3). And this structure was used as the global feature extraction module to obtain more global features.

Figure 9

Improved generation network.

Figure 10

RFB network structure.

In the module for the extraction of detail features, this study uses the serial connection of the RRDB and the residual of receptive field dense block (RRFDB). Each RRDB contains three residual stack blocks RDB. However, each RDB contains five layers of residual convolution. RRDB’s specific structure is shown in Figure 11. By removing the BN layer from RDB, the performance of the generated network can be improved. Meanwhile, the presence of BN layers can lead to the introduction of artefacts in the super-resolution image. This affects the reconstruction effect and limits the generalization ability of the network. By using RRDB to extract primary detail features, its dense multi-level network connection effectively expands the network capacity. And the residual scaling factor P is used to suppress the influence of unstable factors during the training phase.

Figure 11

RRDB.

The structure of the RRFDB is similar to the RRDB. The RDB module in the RRDB is replaced by the RFDB module. The specific network structure of the RRFDB is shown in Figure 12.

Figure 12

Residual of receptive field dense block.

2.3.2 Discriminator network

For discriminator networks, discriminators play a role in discriminating reconstructed images in SRGAN, and their performance is related to the quality of the final reconstructed image. In the traditional SRGAN, the LeakyReLU function is used as the activation function, and its functional expression is

(10) f ( x ) = α x , x < 0 x , x ≥ 0 ,

where α is usually around 0.01.

The LeakyReLU function is optimized based on the ReLU function. Compared with the ReLU function, there is also a slope in the part less than 0. The function of LeakyReLU solves the problem of gradient vanishing in the back propagation process of neural networks. Meanwhile, the LeakyReLU function has no upper limit, so there will be no systematic gradient vanishing in parts greater than 0. Unfortunately, the LeakyReLU function has no lower bound and is monotonously increasing. The left and right derivatives of the function at point 0 are not consistent. As a result, the function is not smooth and causes problems such as neuron extremism, gradient explosion, and poor model generalization. In view of the above shortcomings, this study chooses the Mish function as the activation function of the discriminator, and the expression of the Mish function is

(11) f ( x ) = x ⋅ tanh ( ln ( 1 + e x ) ) .

Compared to LeakyReLU, Mish has three advantages. First, the Mish function has a lower bound. If the activation function does not have a lower bound, the value of the pixel information in the feature map may become smaller and smaller after the activation function is entered. As a result, the network weight continues to increase in a negative direction, eventually leading to the extremes of neurons in the network. Second, when x < 0, the value of the Mish function is less than 0, and the function value first decreases and then increases. Therefore, if the input information is a small negative number, the information can be appropriately amplified by the activation function. And if the input information is a large negative number, the information can be appropriately reduced by the activation function. Further, the gradient explosion can be well avoided. Finally, the Mish function is continuous at point 0 and the left and right derivatives are equal. This feature makes the function smooth at point 0. As a result, the function avoids the singularity. Therefore, when neural networks propagate back, the function can still take derivatives at point 0. As a result, the model has a good generalization ability. In summary, in the discriminator, the Mish function is selected as the activation function and the BN layer is removed. The improved authentication network is shown in Figure 13.

Figure 13

Improved discriminator network.

2.3.3 Loss function

The network loss function is perceptual loss, consisting of two parts: content loss and adversarial loss.

(12) l SR = l X SR ⏟ content loss + 10 − 3 l Gen SR ︸ adversarial loss ︸ percepual loss (for VGG based content losses ) .

The calculation of adversarial losses is shown in Eq. (13)

(13) l Gen SR = ∑ n = 1 N − log D θ D ( G θ D ( I LR ) ) ,

where D θ D is the probability of successfully improving the resolution of the original pixel image (achieving super-resolution), G θ D ( I LR ) is the high resolution pixel image generated.

The content loss function includes pixel level errors and activation function losses, and their calculations are shown in Eqs (14) and (15), respectively

(14) l MSE SR = 1 r 2 W H ∑ x = 1 r W ∑ y = 1 r H ( I x , y HR − G θ D ( I LR ) x , y ) 2 ,

(15) l VGG / i , j SR = 1 W i , j H i , j ∑ x = 1 W i , j ∑ y = 1 H i , j ( Φ i , j ( I HR ) − Φ i , j ( G θ D ( I LR ) x , y ) ) 2 .

The improved image super-resolution reconstruction network pseudocode algorithm is shown in Table 4.

Table 4

Algorithm

Algorithm

input: HR: High resolution images

output: reconstruct super-resolution images

1: Read HR from folder

2: for epoch = 1, 2, …epochs do

3: HRs = Shuffle (HR) //Disrupt image order

4: for i = 1, 2,…,len(HRs)/batch_size do

5: HRb ← {HRs, i, batch_size} //Obtain partial images

6: HRc = Crop(HRb) //Crop images

7: LRc = Downsampling(HRc) //Downsampling to obtain low resolution images

8: SRc = Generator(LRc)

9: HRc_edge, SRc_edge ← {E(HRc),E(SRc)} //Calculate image edges

10: Loss ← {HRc,SRc,HRc_edge,SRc_edge} //Calculate loss function

11: Update(Generator,Discriminator) ← Loss //Update net parameters

12: end for

13: end for

14: return reconstruct super-resolution images

3 Results and discussion

The network was built based on Python 3.7 and Pytorch1.6 framework, and the training and testing were done under the configuration environment of Tesla K80 graphics card and NVIDIA GeForce GTX 1080Ti GPU.

3.1 Experimental dataset

Our study is the super-resolution reconstruction of remote sensing images, and NWPU VHR-10 is one of the mainstream datasets for remote sensing images, so we chose the NWPU VHR10 dataset for model training. The NWPU VHR-10 dataset contains 800 VHR optical remote sensing images with ten classes of geospatial objects. The dataset is divided into two sets: one is a positive image set, containing at least one target in an image, and contains 650 images. The other set is a negative image set containing 150 images and does not contain any targets. Based on the results of the synthetic aperture imaging system parameter simulation in Section 2.1, this work will construct the recovery dataset by randomly selecting some images from the NWPU VHR-10 remote sensing images.

3.2 Evaluation method for image super-resolution reconstruction

PSNR and SSIM are two image quality metrics commonly used in super-resolution reconstruction.

PSNR calculates the distortion of an image using the mean square error; the higher the PSNR value, the closer the distorted image is to the reference image. PSNR is currently the most widely used numerical evaluation method in the field of image processing, but its image quality evaluation results may differ from the visual perception of the human eye. PSNR is calculated as shown in Eqs. (16) and (17)

(16) MSE = 1 m n ∑ i = 0 m − 1 ∑ j = 0 n − 1 [ I ( i , j ) − K ( i , j ) ] 2 ,

(17) PSNR = 10 log 10 MAX I 2 MSE ,

where I and K are the original and input images, m and n are the image sizes, MSE is the mean square error, and MAX I 2 is the maximum possible pixel value of the image

SSIM is an image quality assessment criterion that is intuitive to the human eye. The quality of the distorted image is measured by calculating the difference between the reference image and the distorted image, and then using the calculated value as an indicator. The formula is given by Eq. (18)

(18) SSIM = ( 2 μ i μ k + c 1 ) ( 2 σ i k + c 2 ) ( μ i 2 + μ k 2 + c 1 ) ( μ i 2 + μ k 2 + c 2 ) ,

where μ is the mean value, σ is the variance, σ _ik is the covariance, and c ₁ and c ₂ are the constants; where c ₁ = (k ₁ G)², c ₂ = (k ₂ G)², k ₁ = 0.01, k ₂ = 0.03, and G is the grey level of the image, determined by the data type of the image. The grayscale image data type is unit8, so the value of G is 255, c ₁ = (k ₁ G)² = 6.5025 and c ₂ = (k ₂ G)² = 58.5225.

For reconstructed images, when the PSNR value is close to 40 dB, the restoration effect is excellent. When the PSNR value is greater than 30 dB but less than 40 dB, the image restoration effect is good [23]. SSIM evaluates images in terms of brightness, contrast, and structure. The closer the SSIM result is to 1, the more similar the images are and the better the recovery [24].

3.3 Experimental results

Common super-resolution reconstruction algorithms include nearest neighbor interpolation, bicubic interpolation [25,26], SRCNN [27], and SRGAN. This study tests the conventional super-resolution reconstruction algorithm and the improved SRGAN algorithm on the NWPU VHR-10 dataset. The above four methods are compared with the improved reconstruction algorithm proposed in this work.

To make the SRGAN network operation more stable, all use a multiplicity factor of 4 as the network input. The final recovery results of the five recovery algorithms are shown in Figure 14.

Figure 14

Comparison of restoration algorithms for four remote sensing images: (a) nearest neighbor interpolation, (b) bicubic interpolation, (c) SRCNN, (d) original SRGAN, and (e) improved algorithm.

Based on the comparison of the algorithm recovery effects shown in Figure 14, the image quality of the above algorithms was evaluated using PSNR and SSIM as shown in Table 5.

Table 5

Multiple reconstruction algorithms image quality evaluation results

Method	PSNR				SSIM
Method	①	②	③	④	①	②	③	④
Nearest neighbor interpolation	21.114	22.186	21.132	30.267	0.572	0.747	0.737	0.864
Bicubic interpolation	21.458	24.756	23.770	33.337	0.622	0.789	0.778	0.904
SRCNN	22.031	27.487	25.272	35.834	0.725	0.875	0.829	0.921
Original SRGAN	28.076	29.428	29.741	36.045	0.868	0.887	0.843	0.937
Improved SRGAN	30.258	31.897	31.765	38.898	0.915	0.932	0.895	0.953

According to the image quality evaluation results in Table 5, all the above reconstruction algorithms can improve the reconstruction effect of the simulated images. SRCNN learns the image features through a convolutional network, so the reconstruction effect is slightly better than the interpolated reconstruction algorithm. The improved algorithm improves the PSNR by 2 dB compared to the original SRGAN algorithm. Comparing the above five recovery algorithms, the improved SRGAN has the best recovery effect on the optical synthetic aperture with the PSNR value up to 30 dB.

3.4 Ablation experiments

In order to verify the effect of different numbers of residual groups (G) and residual blocks (B) combination methods on the reconstruction performance of the synthesized aperture images, ablation experiments were carried out in this work by selecting different combination methods. G is the number of groups, B is the number of residual modules in each group. And the experimental results are shown in Table 6.

Table 6

Impact of the number of residual groups (G) and residual blocks (B) on the performance of image super-resolution reconstruction

Parameter	G0B0	G2B1	G2B2	G2B3	G2B4	G2B5
PSNR (dB)	27.45	28.89	30.01	30.45	30.90	30.94
SSIM	0.7845	0.8247	0.8647	0.8815	0.8843	0.8840

As can be seen from Table 6, the PSNR values of the model (G2B4) in this article have increased by 3.45, 2.01, 0.89, and 0.45 dB compared to G0B0, G2B1, G2B2, and G2B3, respectively. The SSIM values increased by 0.0998, 0.0596, 0.0196, and 0.0028, respectively. For G2B5, the PSNR value presented after testing has increased by 0.04 dB compared to the model in this study, but the structural similarity SSIM index has decreased by 0.0003. Figure 15 shows the relationship between PSNR and SSIM values of models composed of different number of residual blocks.

Figure 15

The relationship between PSNR and SSIM.

As shown in Figure 15, it can be seen that when the combination of residual blocks is G2B1, G2B2, G2B3, and G2B4 (the model of this study), both the PSNR value and the SSIM value of the test gradually increase. However, when the combination of residual blocks is grouped as G2B5, the PSNR value is slightly higher than G2B4, but its structural similarity SSIM indicator decreases instead. This indicates that the model performance is saturated when the number of residual blocks reaches more than 8. At this time, if we continue to increase the number of residual blocks, it will greatly increase the overall training difficulty of the network, and the reconstruction performance of the model will not be effectively improved, so this study chooses the combination of G2B4 to achieve the best performance of the model in this work.

4 Conclusion

In this study, the simulation analysis reveals that the lack of intermediate frequency information inherent in the synthetic aperture imaging system is the main reason for the degradation of the imaging performance of the system. To solve this problem, we propose an improved SRGAN to restore the synthetic aperture image without changing the structure of the optical SAR imaging system. Through the calculation and structural analysis of the optical synthetic aperture system, the system parameters of this study were determined. At the same time, the synthetic aperture simulation dataset was constructed according to the system parameters. The improved algorithm in this study transforms the residual structure of SRGAN generated network. and enhances the global features of the image by using multiscale convolutional cascade. The details of the generated network are extracted using multiscale RFB and RRDB. As a result, the quality of network reconstruction is improved and more information about texture details is obtained. The stability of the algorithm is also improved and more high frequency details are obtained. In addition, this study uses the Mish function as the activation function of the discriminator, which solves the problems of neuron extremity, gradient explosion, and poor model generalization ability. The results show that the method proposed in this study can effectively improve the image quality and provide a reference for the development of the field of image super-resolution reconstruction. However, it is less effective in image recovery for complex scenes. Therefore, we will focus on the adaptability of optical synthetic aperture super-resolution reconstruction algorithms to different natural environments and their application in real scenes in the future.

Funding information: Natural Science Basic Research Plan in Shaanxi Province of China (grant No. 2022JM-399), Postdoctoral project in Shaanxi Province (grant No. 2023BSHEDZZ163), and Department of Science and Technology of Shaanxi Province (grant No. 2024SF-YBXM-449).
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Conflict of interest: The authors state no conflict of interest.

References

[1] Hao WQ, Liang ZC, Liu XY, Zhao R, Kong MM, Guan JF, et al. Imaging performance of fractally structured sparse aperture arrays. Acta Phys Sin. 2019;68(19):323–9. 10.7498/aps.68.20190818.Search in Google Scholar

[2] Jin J, Hou J, Chen J, Kwong S. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; 2020. p. 2260–9. 10.1109/CVPR42600.2020.00233.Search in Google Scholar

[3] Cheong JY, Park IK. Deep CNN-based super-resolution using external and internal examples. IEEE Signal Proc Lett. 2017;24(8):1252–6. 10.1109/lsp.2017.2721104.Search in Google Scholar

[4] Gao Y, Li H, Dong J, Feng G. A deep convolutional network for medical image. CAC IEEE. 2017;5310–5.10.1109/CAC.2017.8243724Search in Google Scholar

[5] Liu KW, Ma Y, Xiong HX, Yan ZJ, Zhou ZJ, Liu CY, et al. Medical-image super-resolution reconstruction method based on residual channel attention network. Laser Optoelectron Prog. 2020;57(2):021014. 10.3788/LOP57.021014.Search in Google Scholar

[6] Alvarado ST, Fornazari T, Cóstola A, Morellato L, Silva T. Drivers of fire occurrence in a mountainous Brazilian Cerrado savanna: Tracking long-term fire regimes using remote sensing. Ecol Indic. 2017;78:270–81. 10.1016/j.ecolind.2017.02.037.Search in Google Scholar

[7] Dong C, Loy CC, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell. 2016;38(2):295–307. 10.1109/TPAMI.2015.2439281.Search in Google Scholar PubMed

[8] Zhang YL, Tian YP, Kong Y, Zhong BN, Fu Y. Residual dense network for image super-resolution. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE; 2018. p. 2472–81.10.1109/CVPR.2018.00262Search in Google Scholar

[9] Sha F, Zandavi SM, Chung YY. Fast deep parallel residual network for accurate super resolution image processing. Expert Sys Appl. 2019;128:157–68. 10.1016/j.eswa.2019.03.032.Search in Google Scholar

[10] Zhou DW, Zhao LJ, Duan R, Chai XL. Image super-resolution based on recursive residual networks. Acta Autom Sin. 2019;45(6):1157–65. 10.16383/j.aas.c180334.Search in Google Scholar

[11] Tang J, Wang KQ, Zhang W, Wu XY, Liu GD, Di JL, et al. A deep learning-based image recovery method for optical synthetic aperture imaging systems. J Opt. 2020;40(21):66–74.Search in Google Scholar

[12] Qiao C, Li D, Guo Y, Liu C, Jiang T, Dai Q, et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods. 2021;18:194–202. 10.1038/s41592-020-01048-5.Search in Google Scholar PubMed

[13] Hu DM, Wang KH, Lin J. Progressive GAN for face image super-resolution. J Chin Computer Syst. 2021;42(9):1955–61.Search in Google Scholar

[14] Sun CW, Chen X. Multiscale feature fusion back-projection network for image super-resolution. Acta Autom Sin. 2021;47(7):1689–1700. 10.16383/j.aas.c200714.Search in Google Scholar

[15] Chen G, Kang P, Wu X, Yang Z, Liu W. Adaptive Visual Field Multi-scale Generative Adversarial Networks Image Inpainting Base on Coordinate-Attention. Neural Process Lett. 2023;55(2023):9949–67. 10.1007/s11063-023-11233-0.Search in Google Scholar

[16] Chen Y, Xia R, Yang K, Zou K. MFMAM: Image inpainting via multi-scale feature module with attention module. Computer Vis Image Underst. 2023;238(2024):103883. 10.1016/j.cviu.2023.103883.Search in Google Scholar

[17] Chen Y, Xia R, Yang K, Zou K. GCAM: lightweight image inpainting via group convolution and attention mechanism. Int J Mach Learn Cyber. 2023;1–11. 10.1007/s13042-023-01999-z.Search in Google Scholar

[18] Ning B, Hui M, Liu M, Dong L, Kong L, Zhao Y. Image restoration for optical synthetic aperture system via variational physics-informed network. Results Phys. 2023;52(2023):106878. 10.1016/j.rinp.2023.106878.Search in Google Scholar

[19] Li Y, Zhou L, Xu F, CHEN S. OGSRN: Optical-guided super-resolution network for SAR image. Chin J Aeronaut. 2022;35(5):204–19. 10.1016/j.cja.2021.08.036.Search in Google Scholar

[20] Liao YB, Ma XH. An introduction to fourier optics. Beijing: Tsinghua University Press; 2016.Search in Google Scholar

[21] Hu L, Wang ZG, Chen T, Zhang YM. An improved SRGAN super-resolution reconstruction algorithm for infrared images. Journal of System Simulation. 2021;33(9):2109–18. 10.16182/j.issn1004731x.joss.20-0450.Search in Google Scholar

[22] Cheng HX, Liu EH. Research on image super-resolution based on generative adversarial networks. Electronic Measurement. Technology. 2020;43(14):132–5. 10.19651/j.cnki.emt.1904062.Search in Google Scholar

[23] Li WT, Lin ZC, Jiang D, Li HD, Liu T. A peak signal-to-noise ratio-based algorithm design for image comparison of a test site monitoring and acquisition. China Sci Technol Inf. 2023;690(01):115–8.Search in Google Scholar

[24] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13(4):600–12. 10.1109/tip.2003.819861.Search in Google Scholar PubMed

[25] Chen YQ. Application of interpolation algorithm in image restoration. Digital Technol Appl. 2017;2:156–7. 10.19695/j.cnki.cn12-1369.2017.02.099.Search in Google Scholar

[26] Zeng QL, Hong ZW. Fish eye image correction based on Bicubic interpolation. J Jishou Univ (Nat Sci Ed). 2021;42(2):45–9. 10.13438/j.cnki.jdzk.2021.02.008.Search in Google Scholar

[27] Zhao ZH, Li DX. Research on face recognition technology based on improved SRCNN algorithm. Foreign Electron Meas Technol. 2020;39(12):74–9. 10.19652/j.cnki.femt.2002241.Search in Google Scholar

Received: 2023-05-22

Revised: 2024-01-02

Accepted: 2024-01-15

Published Online: 2024-04-04

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/phys-2023-0194

Keywords for this article

super-resolution reconstruction; generative adversarial networks; multi-scale convolution cascade; feature extraction; optical synthetic aperture

Creative Commons

BY 4.0