Neural network big data fusion in remote sensing image processing technology

2.1 Dataset selection

The data is derived from the UC Merced Land Use Dataset constructed by Newsam SD, Yang Y [15]. The dataset contains images of 21 land use categories, which are manually extracted from large images sourced from the USGS National Map Urban Area Imagery collection, focusing primarily on various urban areas. In the process of analyzing RS image processing technology, this article selected 400 RS data images that meet the research requirements based on parameters such as spatial resolution, temporal resolution, and spectral resolution of RS data, including 100 RS images of buildings, 100 storage tanks, 100 parking lots, and 100 farms, and they were all in Tagged Image File Format (TIFF). The size of these 400 RS images is 256 × 256, and the resolution of RS images is 72 dpi (dots per inch). At the same time, in order to ensure the reliability of subsequent image processing and analysis of experimental results, it is necessary to evaluate the image quality of these 400 images. This article improves image quality through methods such as image enhancement, gray scale processing, noise removal, and geometric transformation. First, image gray scale processing is performed by adjusting parameters such as brightness, contrast, and color saturation to enhance the details and visual effects of the image. Then, the nearest neighbor interpolation method and bilinear interpolation method are used for image geometric transformation, and finally, median filtering, Gaussian filtering, and mean filtering are used to smooth the image. In this study, histogram analysis is chosen to evaluate the quality of RS images.

In the evaluation process, this study analyzes the histogram of these 400 images, and determines whether there is overexposure or color imbalance in each RS image. At the same time, a RS image was randomly selected from 400 RS images of different categories to display its histogram, as shown in Figure 1.

Figure 1

Presentation of histogram of category 4 RS images [15].

After analyzing Figure 1, according to the pixel distribution of the histogram, it can be seen that the brightness distribution of the two images of the parking lot and farm is not close to the left or right side, which indicates that there is no overexposure or underexposure of brightness in these two images. The gray distribution of pixels of buildings and storage tanks has a certain distribution on the leftmost side of the histogram, but it is not mainly concentrated on the leftmost or right side. This indicates that the gray scale distribution of the image is relatively uniform or there is a certain degree of gray scale shift. Generally speaking, if the gray scale distribution of an image is mainly concentrated on the leftmost side (i.e., dark area), it indicates that the image is dark and lacks brightness. On the contrary, if it is mainly concentrated on the far right (i.e., bright area), it indicates that the image is slightly brighter and lacks contrast. When the gray scale distribution has a certain distribution on the far left side of the histogram, but is not mainly concentrated on the far left or right side, there may be the following situations: first, the gray scale distribution of the image is relatively uniform, and second, the image has a certain gray scale offset. Finally, after visual detection and histogram analysis of 400 RS images, color imbalance or overexposure in 400 images are determined. The quality assessment results of 400 images are shown in Table 1.

After analyzing the data in Table 1, it can be determined that among the 400 RS images, 2 RS images of storage tank have the problem of brightness overexposure or underexposure, and 3 RS images have the defect of color balance. Therefore, there are 95 RS images of storage tank that can be used in this study. Second, there are 2 RS images of urban buildings with brightness defects, so there are a total of 98 smoke sensing images of urban buildings that can be used. Then, there is 1 RS image of parking lot with brightness defect and 1 RS image with color balance defect, so there are 98 RS images of parking lot available. Finally, there are two RS images of farms with visual defects and one with color balance defects. A total of 97 RS images of the farm were therefore available.

At the same time, part of the RS image public dataset used in this study is displayed. Public RS image datasets are RS image datasets from a variety of publicly available data sources, which are usually released by government agencies, research institutions or commercial companies, etc. They usually contain various types of images, such as visible, infrared, hyperspectral, etc. These images can provide information about different aspects of the Earth’s surface, such as the natural environment, human activities, and climate change. The formats of these datasets also vary, and common formats include GeoTIFF, ENVI, HDF5, etc. The sources of public RS image datasets are mainly various satellite and airborne RS platforms, including but not limited to Landsat, Sentinel-2, Google Earth Engine, Kosatellite, and so on. These platforms regularly collect and process large amounts of RS imagery data and release them for public use. In addition, some commercial companies such as DigitalGlobe and Planet Labs also provide high-resolution RS image data, which can also be used for research and applications in the fields of machine learning and computer vision. Some of the public RS image datasets are shown in Figure 2.

Figure 2

Image display in the public dataset of RS images [15].

The first row in Figure 2 is the RS image of the storage tank, in which the boundary of the storage tank in image 2a and b is clear, so the difficulty of RS image processing is relatively low. In Figure 2c and d, the boundary of the storage tank is blurred or there are many similar objects, which can easily increase the processing difficulty of the RS image processing model. The eight RS images 2e, f, i, j, k, l, o, and p are easy to process, while the four RS images 2g, h, m, and n are difficult to process.

Then, the image processing difficulty of 388 available RS images was studied and plotted, as shown in Table 2.

Table 2 shows how many RS images are easier to process and how many are harder to process for four different types of RS images when RS image processing technology is used.

2.2 Preprocessing steps

In the process of RS image preprocessing, 388 RS images that can be used need to be converted into gray images, and the selection of gray images involves selecting different gray methods for different types of RS images. In this study, storage tank remote sensing image, urban building remote sensing image, parking lot remote sensing image and farm remote sensing image are used for gray processing. Image dimensions were converted from 300 × 300 to 250 × 250. In this study, four RS images of storage tanks, urban buildings, parking lots, and farms were randomly selected to display gray image conversion, mainly using four methods: system function operation gray, average gray, weighted average gray, and maximum gray, as shown in Figure 3.

Figure 3

Grayscale display of the RS images of storage tanks, city buildings, parking lots, and farms [15]

In the process of RS image preprocessing, there are several main purposes for converting RS image to gray level image. First, gray scale images have fewer color channels than RS images. This means that the computing tasks of RS image processing are more than that of gray scale image, which requires more server computing resources. Then, the conversion of gray image can further remove the influence of color information in the process of image processing. Gray scale image conversion is a commonly used method in image processing, which can convert color images into gray scale images for further processing and analysis. In gray scale image conversion, by contrast stretching, dense gray scale areas can be transformed into a wider range, compressing the gray scale range of areas of interest. Afterwards, by using the corresponding 0/1 of each pixel to generate the image, it helps to save space occupied by the data. For images with a concentrated gray scale range on the histogram, they may appear dark or bright, resulting in a low contrast visual effect. Histogram equalization can be performed to enhance the contrast. Second, after the image is converted from RS image to gray scale image, the feature dimension of the image would be reduced to a certain extent, which can improve the accuracy of image classification and detection of image processing technology. Finally, gray scale images have only one color channel, so the same number of gray scale images occupy less storage space than RS images in the system.

In this study, 388 RS images are gray-processed by using these four methods. In the process of gray-processing, a gray scale image that is most conducive to image processing would be selected from the gray scale images constructed by the four gray scale methods and put into the gray scale image sample dataset. A sample dataset with 388 gray scale RS images is presented.

This method of selecting the gray scale image that is most conducive to image processing is mainly obtained by comparing the contour similarity between the image and the original image. In this study, the image of the parking lot is used to illustrate. First, the original image of the parking lot and the target edge contour of the four gray scale methods need to be obtained. In the edge detection, the Prewitt edge detection operator is used. This is because the Prewitt operator utilizes the pixel averaging method when detecting edges, which can effectively reduce the impact of noise on edge detection results. In contrast, some other edge detection operators, such as the Roberts operator, have lower noise suppression performance than the Prewitt operator. At the same time, the Prewitt operator utilizes two directional templates to perform neighborhood convolution with the image in the image space to complete edge detection. These two directional templates detect horizontal edges and vertical edges. This method helps to detect edge information in images more comprehensively, as shown in Figure 4.

Figure 4

Prewitt edge detection display of the parking lot and four gray scale methods [15].

Figure 4 shows the effect of the original RS image [15], system function gray level image, average gray level image, maximum gray level image, and weighted average gray level image after using the Prewitt edge detection operator to continue edge detection, and the contours of the five images have little difference in intuitive visual performance. Therefore, this study introduces the Structural Similarity Index measure (SSIM) to calculate which of the four gray scale images has the better SSIM with the original image, so as to complete the selection of gray scale method.

In SSIM, it is proposed to extract the features of brightness, contrast, and structure from the image for comparison. Therefore, compared with traditional image error comparison methods, SSIM is a more suitable image similarity assessment method for human vision. The main formula used is shown in formula (1).

where a and b usually represent the original image and the contrast image in two 8 × 8 or 11 × 11 windows, u a and u b represent the average of a and b, u a 2 and u b 2 represent the variance of a and b, and γ ab is the covariance of a and b. At the same time, in the evaluation of SSIM calculation results, the closer the value is to 1, the two images are very similar, and the closer the value is to 0, the two images are very different.

Finally, by comparing the SSIM of the four gray scale images and the original image, the maximum gray scale image is more similar to the original image in the parking lot gray scale method. Therefore, the maximum gray image is selected from the four gray scale images of the parking lot to be added to the sample dataset.

In this study, the data of sample dataset is expanded with the help of image geometric transformation, which mainly enhances the generalization ability of image processing model, and also helps the image processing model to alleviate overfitting and improve the robustness of the model. In this study, one of the 95 RS gray scale images of storage tanks is randomly selected to show its geometric change results, as shown in Figure 5 [15].

Figure 5

Geometric transformation of the image [15].

In the RS gray image sample dataset, one RS gray image can be expanded to five RS gray images by geometric transformation. Two interpolation methods are used in this process, namely, nearest neighbor interpolation method and bilinear interpolation method. First, the original gray scale image is rotated 20° counterclockwise with the center as the origin, and a complete rotating image is generated by using the nearest neighbor interpolation method. Then, the center of the image is rotated 20° clockwise, and the full image is generated using the same nearest neighbor interpolation method. Next the image is rotated 20° counterclockwise with the center as the origin, and the complete image is generated using bilinear interpolation. The fourth image is rotated 20° counterclockwise from the center as the origin, and the nearest neighbor interpolation method is used to generate an image of the same size as the original image. The final fifth image is similarly rotated 20° counterclockwise, using nearest neighbor interpolation to produce a fully rotated gray scale image.

At the same time, image denoising is also required in the preprocessing stage of RS images. In this study, a RS gray image of a storage tank is selected to demonstrate the denoising effect of different Gaussian noise methods, as shown in Figure 6 [15].

Figure 6

Gaussian noise denoising display of the image [15].

Figure 6 shows the effect of median filtering, Gaussian filtering, and mean filtering on the noise removal of gray scale images containing Gaussian noise. Median filtering is a nonlinear filtering method, which can better preserve the edges and details of the image. Gaussian filtering is a linear filtering method, which can effectively reduce the high-frequency noise in the image, but would cause the loss of image details while processing [16]. Finally, the mean filtering is a relatively simple linear filtering method. Although it can remove noise from the image, it also makes the edges of the image more blurred.

In the image preprocessing stage, the last step is histogram equalization of gray level image. This study also selects a gray scale image for histogram equalization processing and display, as shown in Figure 7.

Figure 7

Histogram balanced display of gray image [15].

Figure 7 shows the histogram equalization operation of RS images [15] in the preprocessing stage of this study, where the horizontal coordinate generally ranges from 0 to 255, representing the brightness range of the image. The ordinate represents the normalized pixel frequency of the RS gray image, representing the proportion or frequency of pixels with a specific brightness value in the entire image. This histogram can also be called normalized histogram, which can reduce the impact of image size and the total number of image pixels, and help researchers observe the pixel distribution of the image.

In the preprocessing stage, atmospheric correction, geometric correction, and image clipping are carried out to further eliminate some factors affecting the image processing process. Atmospheric correction is usually performed using specialized software such as The Environment for Visualizing Images (ENVI) to analyze and manipulate RS images. ENVI is a complete RS image processing platform that integrates software processing technologies that cover input/output of image data, image calibration, image enhancement, correction, ortho correction, embedding, data fusion, as well as various transformations, information extraction, image classification, knowledge-based decision tree classification, integration with GIS, DEM and terrain information extraction, radar data processing, and 3D display analysis.

3.1 BD application pattern

Among the RS image processing technologies combining NN and BD, BD technology mainly provides a variety of optimization methods related to RS image data processing. The application mode of BD technology in RS image processing is shown in Figure 9.

Figure 9

Application model of BD technology in RS image processing technology.

Figure 9 shows the application mode of BD technology before and after the preprocessing stage of RS image processing technology combining NN and BD. In this process, BD technology is mainly used to provide five optimization methods for RS image processing. The five optimization methods are parallel processing, data fusion, efficient data storage and retrieval, real-time monitoring, and high-dimensional data analysis. When processing BD, parallel computing models can be adopted to fully utilize computing resources and improve data processing efficiency. For multi-source data, data fusion technology is used to integrate them together to obtain more comprehensive and accurate data. In terms of storage, distributed storage can be used to disperse data to different nodes, reducing storage pressure on individual nodes and improving system scalability. Meanwhile, for data that require frequent access, caching technology can be used to accelerate the speed of data reading and reduce the number of disk reads. Then, real-time monitoring is achieved through technologies such as real-time data collection and stream computing. Finally, by performing dimensionality reduction on high-dimensional data, we can better understand and analyze the essence and laws of the data. In the traditional RS image processing technology, parallel processing technology is rarely used in the process of processing a large number of RS images. In the image preprocessing stage of RS image processing technology combined with NN and BD, the same type of RS image can be preprocessed simultaneously through parallel data processing. This image processing mode can better speed up the execution of image processing algorithm and improve the processing efficiency of image processing. Data fusion: data fusion is to integrate RS image data from different types of sensors and different times, which can help staff in related fields to get more accurate data. Efficient data storage and retrieval: in addition, after the combination of RS image processing technology and BD technology, the RS image processing technology combined with NN and BD can disperse RS image data in multiple storage systems through distributed storage systems, which can improve the storage capacity of RS image data to a certain extent. On the other hand, specific RS data in the database can be retrieved through the index technology and distributed query in BD technology, which further reduces the requirements and costs of RS image processing technology. Real-time monitoring: In the RS image processing technology combined with NN and BD, the advantages brought by its combination with BD technology are also real-time monitoring function for the target. This RS image processing technology combined with NN and BD can not only help relevant departments of urban management to complete real-time monitoring of urban buildings and farms but also complete all-day monitoring and early warning of storage tanks storing dangerous goods in the storage field of dangerous goods, further reducing risks in all aspects. The last is high-dimensional data analysis.

High-dimensional data analysis: In the RS image processing technology combined with NN and BD, BD technology is generally used for high-dimensional analysis of RS image data. In high-dimensional analysis, the first thing people need to face is the selection of dimensionality reduction algorithm. This study mainly uses Autoencoder (AE) in NN technology to reduce the dimensionality of data in image processing. The AE dimension reduction operation process is shown in Figure 10.

Figure 10

Display of the operation process of dimensionality reduction of AE high dimensional data.

In the process of dimensionality reduction of RS image data using AE, sliding window filter is given, and feature extraction of data is carried out through NN algorithm. In this process, the weight of data is guaranteed not to change. At the same time, the algorithm in NN is used to process the non-linear mapping of the resulting data, and its formula is shown below [21].

where f represents the activation function, w represents a parameter, and n represents the number of training samples. At the same time, the parameters of the model in AE are constantly updated, that is, backpropagation is used, as shown in formula (3).

In the high-dimensional data analysis, the above steps are used to continuously train and update the data of multiple layers in AE to reduce the dimensionalization of high-dimensional data. Then, feature extraction is carried out on the data after dimensionality reduction. In this process, parallel processing is also used, that is, the image data is cut into smaller images for processing, and the final image data are obtained by merging the results. This kind of high dimensional data analysis work can reduce the complexity and number of tasks of computation.

3.2 Application scenario of ConvNext-CNN

In the RS image processing technology combined with NN and BD, NN mainly has three functions: RS image feature extraction, RS image target detection, and RS image change detection [22].

In the RS image classification process combining the RS image processing technology of NN and BD, the sample dataset is first divided into three main datasets: training data set, verification dataset, and test dataset [23]. In this study, we use the leave-out method to divide the dataset into two mutually exclusive sets, and the common ratio is 70% as the training set and 30% as the test set, which is required to maintain the consistency of the data distribution in the process of dividing the training set and the test set. According to the RS image preprocessing stage, the geometric transformation of 388 available RS image sample datasets is carried out to obtain 2,328 RS image data. The 2,328 RS image data are divided into three datasets, each of which contains 776 RS image data to complete the division of sample data [24].

Then, NN model needs to be established to process the sample data and complete the extraction of image features [25]. In this process, this study mainly uses the ConvNext-CNN model. The ConvNext CNN model is a CNN model used for image classification and semantic segmentation. It adopts spatial pyramid pooling technology. This technology changes the conventional operation of pooling layers in traditional CNN, capturing features at different scales by pyramid pooling inputs at different scales, and combining these features to improve the accuracy of CNN. It also incorporates the idea of residual networks. ConvNext CNN improves the performance of the network by combining shallower features with deeper features. Compared to traditional models, the ConvNext-CNN model emphasizes deeper understanding of data and feature capture to achieve higher accuracy. However, this requires more complex technical design and parameter adjustment, as well as higher requirements for computing resources and training time. First of all, it needs to train the model with the training sample dataset, and in the process of model training, the weight and bias of the model are adjusted by using backpropagation [26]. Then, the model hyperparameters are tuned in the validation dataset, which can further improve the classification performance and generalization ability of the model. Finally, the performance of the model is evaluated experimentally by using the test sample dataset, and the image classification model based on ConvNext-CNN is constructed.

In order to show the classification effect of this image classification model based on ConvNext-CNN, this study selects 6 RS images from the original 388 gray scale images for image detection and classification display, and the results are shown in Figure 11.

Figure 11

Display of image detection and classification effect of RS gray image.

In these six images, it can be clearly seen that there are two gray RS images of urban buildings, two gray RS images of storage tanks, and two gray RS images of parking lots. In the process of image target detection and image classification of two gray RS images of urban buildings, the red frame is used for marking, the yellow frame is used for processing two gray RS images of storage tanks, and the green frame is used for processing two gray RS images of parking lots. Through the visual detection of Figure 11, the RS images combined with NN and BD can be correctly classified, and most of the targets in the images can be well identified.

In the change detection of RS image, the main method is to compare the pixels of the target in the RS image of different time series in the same region. In this process, a group of RS images are first input into the model, and at the same time, the RS images are detected and classified. Then, a group of RS images in the same area with different input times are used for target detection and image classification, and then the two groups of RS images are compared. This kind of comparison mainly exists in the same type of RS images. If the pixel values in the RS images are compared, it can be considered that the RS images have changed.

The above is the complete operation flow of RS image processing technology combined with NN and BD technology. This RS image processing technology further improves the problems of large-scale RS image data processing difficulties, target recognition difficulties in complex RS image background, and low accuracy and efficiency of RS image processing. This image processing technology can not only be deeply applied in many fields such as urban planning, building monitoring, real-time control of dangerous storage tanks, farm land planning, etc., but also help people to explore the geological and environmental conditions of more regions [27].