Smear character recognition method of side-end power meter based on PCA image enhancement

2.1 Overall process

To achieve accurate and stable recognition of smear characters of power meters under complex lighting, improvements are mainly made in three aspects: image enhancement, binarization and character segmentation, and character recognition [17]. The overall process is shown in Figure 1. Among them, the fully convolutional network (FCN) [18] and the VGG [19] instrument character segmentation model were trained in advance, which are not marked in the flowchart. The major steps are as follows:

1. Camera images acquisition;

2. The fast adaptive image enhancement method based on PCA [20] fusion is used to correct uneven illumination, and then the binarization is carried out based on FCN and VGG instrument character segmentation method;

3. Remove small connected domains and noise by opening and closing operations;

4. Tilt correction;

5. Segmentation of single character;

6. Single characters are normalized and input to smear character recognition model for recognition.

Figure 1

Character recognition flow chart of power meter.

2.2 Fast adaptive image enhancement based on PCA fusion

The original RGB image is converted into HSV color space, and the V component is extracted and down-sampled. The illumination component of the image is extracted by multiscale Gaussian filtering and up-sampled to the size before down-sampling. Then, the illumination component is refined by guided filtering, and the illumination component is adaptively adjusted by different correction coefficients to obtain two images. The weights of the two images are calculated by PCA and fused to enhance the V component. Finally, the image is converted from HSV space to RGB space to obtain the final enhanced image. The specific process is shown in Figure 2.

Figure 2

Fast adaptive image enhancement algorithm flow based on PCA fusion.

The calculation amount of multiscale Gaussian filtering increases with the increase of the input image size [21], which affects the real-time performance of the algorithm. Down-sampling and up-sampling can well reduce the calculation amount, and guided filtering can refine the up-sampling results and retain the local characteristics of the illumination component. To avoid the small down-sampling size and excessive loss of image details, the linear interpolation method is used for down-sampling, and the down-sampling size is a quarter of the input size.

To balance the global and local characteristics of the extracted illumination components, the multiscale two-dimensional discrete Gaussian function is used to extract the illumination components in the scene, and each component is weighted. Finally, the estimated illumination components are obtained. The calculation is as follows:

where I v _ g i ( x , y ) denotes the illumination component extracted by the ith scale Gaussian function, θ i represents the corresponding weight coefficient, and N represents the number of scales. To balance the accuracy and calculation amount of illumination component extraction, N is set to 3, θ i is set to 1/3, and the parameters c of the three scales of Gaussian filtering are 4, 20, and 63, respectively.

To reduce the amount of fusion computation, a simple source image weighted fusion is adopted without multiscale decomposition to avoid the bias of fusion results to a certain source image. PCA is used to determine the weight coefficient of each source image as the key to image fusion. The specific process and result are shown in Figure 3, and it can be seen that the image process by fast adaptive image enhancement algorithm is significantly improved in terms of color, brightness, and detail.

Figure 3

Image fusion based on PCA.

2.3 FCN- and VGG-based smear character segmentation

Combining FCN and VGG16 network, the input is the original RGB image and the output is a binary map without the smear. The network consists of two parts: feature extraction and image recombination. The feature extraction is a down-sampling composed of five convolutional blocks, each convolutional block includes a convolutional layer and a maximum pooling layer, and the Relu activation function is chosen behind the convolutional layer to alleviate overfitting. The number of convolutional kernels in the convolutional layer is 256, the size is 3 × 3, the step size is 1, and the pad is 1. The pooling layer has a pooling window size of 2 × 2 and a step size of 2.

The image reconstruction is performed by five deconvolution blocks to reduce the extracted information to images. The deconvolution block includes a deconvolution layer and a connection layer, where the convolution kernel of the deconvolution layer is 4 × 4 with a step size of 2. The overall structure of the network is shown in Figure 4, and the mean square error (MSE) is used as the loss function, which is defined as follows:

where n is the number of samples, y i is the label, and y i ′ is the network prediction result.

Figure 4

The structure of a fully convolutional neural network.

3.1 Collection and calibration of data sets

The samples were collected in the instrument working environment. The light sources were placed in the left, right, and median positions of the instrument and combined with a 30 degree angle deflection between the camera and the instrument. A total of 7,600 color images were collected.

Label production is divided into two stages: preparation and production. In the preparation stage, the accurate position of the characters in the instrument image is mainly counted. The image with good illumination is selected, and the approximate region of the characters is manually calibrated. The minimum circumscribed rectangle is determined, and the angle θ between the long edge of the minimum circumscribed rectangle and the horizontal x-axis is calculated. The binary image is rotated and corrected to make all the characters on the same horizontal line. The center P1(x ₁, y ₁), P2(x ₂, y ₂), and P3(x ₃, y ₃) of the three calibration circles are determined by circle detection as the three points corresponding to the affine transformation in the label production stage. The position of each character is recorded by the vertical projection method, and the characters with complete segmentation and no shadow are selected as the standard single character, as shown in Figure 5.

Figure 5

Label preparation phase. (a) Original image, (b) binary graph, (c) calculate rotation angle, and (d) location of each character.

In the label production stage, Wolf algorithm is used to binarize the instrument image and detect the center of three calibration circles P1́ ( x 1 ' , y 1 ' ) , P2́ ( x 2 ' , y 2 ' ) , and P3́ ( x 3 ' , y 3 ' ) , combined with the center P1(x ₁, y ₁), P2(x ₂, y ₂), and P3(x ₃, y ₃) detected in the preparation stage, the affine transformation matrix A is calculated as shown in Eq. (3). Affine transformation of instrument image is obtained by affine transformation matrix, let P1, P2, and P3 coincide with P1́, P2́, and P3́, respectively. On each character position recorded in the preparation phase, the shadow character is replaced with its corresponding standard single character to complete the label production, as shown in Figure 6.

Figure 6

The label production phase. (a) Original image, (b) binary graph, (c) affine transformation result, and (d) label after character replacement.

To reduce the training time, the sample only represents the number of parts by the interceptor, and each sample is randomly rotated and shifted to expand three samples. Through this method, the total number of samples is 30,400, and 80% of the samples are randomly selected for training, and the remaining samples are used as testing samples.

3.2 Binary segmentation experiment

The experimental environment is Intel (R) Xeon (R) 2.4 GHz CPU E5-2640, GPU NVIDIA RTX 2080Ti, Python3.6 programming in TensorFlow framework. Without other pretraining models, batch size is 16, initial learning rate is 1 × 10⁻⁴, period is 100. Compared with local threshold binarization algorithms such as Bernsen [22] black, Sauvola [23] Bradley [24], Wellner [25], and Wolf et al. [26], IoU and SNR are used as evaluation indexes.

The running time, IoU, and SNR mean of each algorithm are presented in Table 1. Some of the operation results are shown in Figure 7. The uneven illumination has a great influence on the binarization effect, and the binarization effect of the traditional methods on the smear is very poor. Niblack algorithm generally divides the target area, but with a lot of noise, it cannot remove the smear character. Because most of the local region of the instrument image is the background, uneven illumination causes Bernsen, Sauvola and Bradley algorithm to divide part of the background into target pixels and do not eliminate the smear character. Wellner algorithm cannot segment strong light region and smear character. Wolf algorithm overcomes the influence of uneven illumination to some extent, but cannot smear character. The network designed in this article has a clear and no smear character on the binarization effect of uneven illumination and shadow images. The IoU and SNR values are higher than those of other methods. It is suitable for images with different light intensities and rotations. It has a strong generalization ability and fast processing speed, only slightly higher than Wellner.

Figure 7

Processing results of different algorithms. (a) Original image, (b) ideal binary label, (c) Bernsen algorithm, (d) Niblack algorithm, (e) Sauvola algorithm, (f) Bradley algorithm, (g) Wellner algorithm, (h) Wolf algorithm, and (i) the method of this article.

3.3 Character recognition experiment

Smear characters are segmented for character recognition [27]. Character segmentation includes tilt correction, affine transformation, vertical projection, and horizontal projection. The specific effect is shown in Figure 8. Because the horizontal projection intervals of numbers, decimals and “VAC” characters are different, and “VAC” characters remain unchanged, decimals can be judged directly by the horizontal projection interval, so only numbers are reserved in segmentation. The single character is segmented by this algorithm, as shown in Figure 9, where the smear character is basically eliminated.

Figure 8

Segmentation effect of the projection method.

Figure 9

Single-character samples segmented by the proposed algorithm.

The individual characters after sample segmentation are categorized to make numerical labels, which are input into the CRNN model for training and testing, respectively.

To verify the effectiveness of the algorithms in this article, HOG + SVM [28], KNN [29], and CRNN [30] algorithms are used for character recognition. The CRNN algorithm parameters are as follows: input image size is 32 × 32, batch size is 64, training cycle is 3000, and learning rate is 0.0001. SVM uses linear kernel function, and HOG feature unit size is 3 × 3.

The experimental results are presented in Table 2. Compared with the direct recognition of characters, the recognition rate of all algorithms is higher after the processing of the proposed character segmentation algorithm, and the recognition rate of CRNN algorithm is improved by 21.71%, indicating that the proposed binarization algorithm provides a better basis for character recognition.