Feature fusion-based text information mining method for natural scenes

2.1 Pre-processing

Texts are typically designed in natural scenes to attract attention, and some research [14,23] evaluated existing visual attention models for text detection to show that scene texts are salient. The saliency cues are adopted in our approach to prevent an exhaustive spatial and scale search over all possible regions. The augmented precision of the edge map consistently exerts a profound impact on the SWT outcome, as the SWT operator exhibits a linear correlation with the quantity of edge pixels. A multistage edge detection method is proposed to obtain an accurate edge map based on the gradient image with the assistance of the visual saliency regions in our work.

2.1.2 Obtaining the accurate edge map by using the multistage edge detection

The Canny edge detection method is characterized by its utilization of two thresholds to identify both strong and weak edges. This method is known to be less susceptible to being impacted by noise when compared to other edge detection techniques such as the Sobel and Prewitt methods. The Canny method operates by identifying local maxima in the gradient of an image, thereby detecting edges. However, it should be noted that this approach may also result in the detection of non-text edges. Efforts to minimize such false detections by adjusting the segment threshold may unfortunately result in the loss of true text edges.

Fortunately, the characters have obvious differences with the backgrounds in text regions. The strokes of the character have similar fixed widths, and the gradient distribution of the character has obvious regularity, (as shown in Figure 2). We can obtain the accurate text edges^[1] in the image while dismissing noisy and foliage edges with the following steps. First of all, we strengthen the regular boundary and weaken the irregular one, which is produced mainly by the non-text targets, (i.e., calculating the synthesis gradient value of every pixel with 3 × 3 neighborhoods), and then the edge filter mask is obtained by using binarization processing and morphological processing. At last, the accurate edge is obtained using the following equation:

(3) A ccurate _ edge ( i , j ) = O rig _ edge ( i , j ) × E dge _ filter ( i , j ) ,

where A ccurate _ edge ( i , j ) shows the pixel value of the accurate edge map at position ( i , j ) , O rig _ edge ( i , j ) and E dge _ filter ( i , j ) show the pixel value of the original Canny edge map and the edge filter mask at position ( i , j ) , respectively, i ∈ ( 1 , M ) , j ∈ ( 1 , N ) , and M and N show the row and column of the gray image, respectively.

Figure 2

Gradient direction images of the text and the non-text regions. (a) non-text region, (b) gradient direction image of the non-text region, (c) text region, and (d) gradient direction image of the text region.

2.2 Text candidates generation processing

The characters always have a nearly fixed stroke width in the text region. The SWT performs per-pixel computations to estimate the most probable stroke width, resulting in an output image that maintains the same dimensions as the input image. In the SWT image, each element contains the width of the stroke associated with the pixel. The SWT operator is linear with the number of edge pixels, and an accurate edge map often completely enhances SWT results. To describe the effectiveness of the applied method, we obtain accurate edges by using the multistage edge detection method proposed in our method. The SWT images are guided by the original edge map and the accurate edge map separately, and the SWT images are shown in Figure 3. Applying the SWT image guided by the accurate edge map, we can obtain better results in our work.

Figure 3

SWT images guided by different edge maps. (a) SWT guided by the original edge, and (b) SWT guided by the accurate edge.

In text regions, the stroke width of characters is typically consistent. The SWT algorithm computes the most probable stroke width for each pixel, producing an output image of equal size to the input image. Each element in the SWT image corresponds to the stroke width associated with the respective pixel. The computational complexity of the SWT operator is linear with the number of edge pixels present in the image. The accuracy of the SWT results can be significantly improved by incorporating an accurate edge map. To evaluate the performance of the proposed method, we employ a multistage edge detection technique to obtain accurate edges. The SWT images are generated using both the original edge map and the accurate edge map as guides. The results are depicted in Figure 3. It can be observed that the use of the accurate edge map as a guide for the SWT results in improved performance in our study.

2.3 False positives elimination processing

The SWT generates an output image, where each pixel corresponds to the width of the most probable stroke. While the SWT processing eliminates certain text regions that are unlikely to be text (e.g., by setting a maximum stroke width threshold of 350 pixels in our work, pixels with a stroke width exceeding this threshold are discarded), there may still exist non-text regions that persist and hinder subsequent processing.

2.4 Retrieving the missing text regions around the key regions

Despite the potential loss of text regions through prior processing, it is fortunate that text in natural scenes frequently appears in groups or lines. As a result, candidates in proximity to text regions are highly likely to be considered as text. To take advantage of the specific spatial distribution of text, we utilize context information to recover missing text regions.

In order to describe the algorithm conveniently, we illustrate this question aided by Figure 4. In our work, the key region is defined as the region of the remaining adjacent candidates in the horizontal direction. As shown in Figure 4, the words “AR PARK” highlighted with a yellow background are the key region, the red dashed box indicates the horizontal search area, and the yellow dashed box indicates the vertical search area. Meanwhile, we further imagine that the candidate regions are lost after a series of ahead processing, which are shown in the left and right parts of Figure 4.

Figure 4

Retrieving the missing text regions by using context information.

In our study, we leveraged context information to retrieve the missing text regions by utilizing key regions. To formulate context descriptors, we employ four distinct features, namely stroke width, common part, CC width, and CC height. We denote all the missing candidate regions after a series of ahead processing as C r = { c r 1 , c r 2 , … , c r i , … , c r M } , where M is the total number of missing candidate regions. The c r i is the i th missing candidate region, and we define the state of c r i as { c s i , c c i , c w i , c h i } , where the c s i is the stroke width and c c i is the common part with the key region (i.e., it is the common width with the key region in the vertical search area, and the common height in the horizontal search area). Note that c w i is the CC whole width and c h i is the whole height. Meanwhile, we denote the key region K r = { k r 1 , k r 2 , … , k r i , … , k r N } , where N is the total number of characters in the key region. The K s ave is the average stroke width of the characters in the key region, the K w ave is the average width, and the K h ave is the average height.

The missing text candidates will not be retrieved in the search area, if they meet any one of the following conditions:

Note that, the first row of equation (4) is one of the requirements to find the missing text regions in the vertical searching area, and the second row of equation (4) corresponds to the horizontal search area. Based on the pixel-wise annotation of the ICDAR 2003 training dataset,^[2] the parameters are empirically selected, and T 1 = T 2 = 0.5 and T 3 = T 4 = T 5 = 0.75 in our work.

2.5 Grouping characters into text line and further verification

The objective of this step is to group the set of characters identified in the previous steps into coherent lines of text. The characters within the same text line are anticipated to exhibit similar stroke width, character width, and character height. Our work operates under the assumption that the text lines in natural scenes tend to be horizontal or slightly tilted. To address this, we introduce the concept of an “influence field.” For example, the influence field for the character R in Figure 5(b) is depicted by the yellow background. The CC candidates within the influence field are compared to the reference character based on their CC width and height. If these CCs meet specified constraint conditions, they are merged together.

Figure 5

The results of grouping characters into text line. (a) Before grouping characters into text line, (b) the influence field of a character, and (c) after grouping characters into text line.

Note that, for the ith CC in the influence field, the c w i is the whole width and c h i is the whole height. The parameter c w seed is the width and c h seed is the height of the specific candidate. For every specific candidate, the morphology structure element is dynamically selected as 1 × ( T × c h seed ) . The morphology close processing is implemented between the specific candidate and the CCs in its influence field. For every CC that is taken into account, the text line is developed by merging the sub-images, which are secured by using morphology close processing. The before and after grouping characters in text lines are shown in Figure 5(a) and (c), respectively, and in this section, T = 1.5 and T 1 = T 2 = 0.65 .

In the task of text region recognition, the selection of appropriate descriptors is critical for effectively differentiating between text regions and background interferences. The SVM classifier utilized in the previous stage has demonstrated remarkable efficacy in eliminating background interferences. However, text lines in natural scene images often exhibit significant layout variations, making it challenging to train a comprehensive classifier using generic descriptors. As a result, some persistent background interferences may not be fully removed in previous processing steps. To address this challenge, a different approach is adopted. Guided by the work of Wang et al. [14], an unsupervised feature learning algorithm is utilized to automatically extract features from the training samples, and a convolutional neural networks (CNN) classifier is applied for candidate region classification. Although the CNN classifier presented in Wang et al. [14] demonstrates exceptional classification capabilities, it is not employed for text detection in the original image due to its time-consuming nature as it performs multi-scale evaluations of candidate regions. In light of these considerations, an off-the-shelf CNN classifier is utilized in this work for efficient verification of candidate text lines with smaller areas than the original image.

2.6 Segmenting text line into words

In the context of scene text detection, while word segmentation is not the primary concern, it is nonetheless necessary to segment the detected text lines into separate words for the purpose of evaluating performance using the strategies employed in the ICDAR 2005 and ICDAR 2011 robust reading competitions. To address this requirement, a heuristic rule has been developed to facilitate the segmentation process. This rule is based on the calculation of the average distance between adjacent CCs within a given text line. The minimum word spacing distance T is estimated by equation (10). A separation will occur if the distance between adjacent CCs is over T , and the intra-word characters would be separated from the inter-word characters.

where D ave is the average distance value. Based on the previous research, we empirically set α = 1.75 and β = 3 .

3.1 Experimental datasets

In order to comprehensively assess the performance of the proposed method in comparison with other state-of-the-art text detection methods, we carried out experiments on the ICDAR 2005 dataset^[3] and ICDAR 2011 dataset.^[4] The ICDAR 2005 dataset consists of ∼ 499 natural scene images that have been annotated with ground truth, with varying resolutions, and its TrialTest subset contains a total of 249 images. The ICDAR 2011 dataset was specifically collected for the text detection competition at the ICDAR 2011 conference, and it comprises 484 natural scene images that have been annotated with ground truth, with a corresponding test dataset of 255 images.

3.2 Evaluation protocol

The performance of our method is quantitatively measured by Precision (P), Recall (R), and f-measure (F). They are separately computed using the definitions provided in the studies by Lucas et al. [29], Lucas [30], and Shahab et al. [31]. The output is a set of rectangles designated with bounding boxes for detected texts, and a set of ground-truth rectangles are also provided in these datasets. The match m p between two rectangles is defined as the area of the intersection divided by the area of the minimum bounding box containing both rectangles. The number has the value one for identical rectangles and zero for rectangles that have no intersection. The closest match of each resulting rectangle with the set of truths is calculated.

The best match m ( r ; R ) for a rectangle r in a set of rectangles R is defined as follows:

The Precision and Recall are defined as follows:

where T and E represent the sets of ground truth and result rectangles, respectively. The f-measure, which is a single measure of algorithm performance, is a combination of the two measures, and it is defined as follows:

where the parameter α is the relative weight of the Precision and the Recall, and α = 0.5 in our work.

3.3 Evaluation of SWT results obtained by using the multistage edge detection

To prove the effectiveness of the multistage edge detection method, we adopted the ICDAR 2005 annotation test dataset^[5] as the baseline. Compared with the baseline in pixel level, the detection results are quantitatively measured by the Recall and the Precision. Note that, in this section, the two parameters are defined as follows:

where I mg _ dec is our segmentation result obtained by binarizing the SWT image and I mg _ gt is the corresponding ground truth. In order to prove the effectiveness of the multistage edge detection algorithm, we adopted the original Canny edge map and the accurate edge map to obtain the SWT image, respectively. As shown in Table 2, we can greatly increase the Precision with a little cost of the Recall reduction.

There are total 1.08 × 1 0 7 pixels annotated as texts in the ICDAR 2005 annotation test dataset. With the original Canny edge map for SWT processing, we can obtain 1.1 × 1 0 8 candidate text pixels, and there are 9.77 × 1 0 6 true text pixels in it. However, with the accurate edge map for SWT processing, we can obtain 5.0 × 1 0 7 candidate text pixels with 8.15 × 1 0 6 true text pixels in them. The aforementioned results show that we can greatly remove the false positive pixels with a little cost of the true positive pixels reduction. In addition, the average time consumption of performing SWT processing guided by the original edge maps is 16.6 s; however, the average time consumption is 10.8 s by using the accurate edge maps.^[6]

To further illustrate this problem, we first, we adopt the original Canny edge map and the accurate edge map to obtain the SWT image, respectively, and then the same exact processing steps as the subsequent pipelines will follow the SWT processing to detect the scene texts. For the original Canny edge map case, we can obtain the Recall 0.61, the Precision 0.66, and the f-measure 0.63. However, for the accurate edge map case, we can obtain the Recall 0.62, the Precision 0.73, and the f-measure 0.67. It is obvious that better text detection performance has been obtained by using the accurate edge map obtained by our multistage edge detection method.

3.4 Evaluation of retrieving the missing text regions

In order to further verify the effectiveness of retrieving the missing text regions by using the context information, a comparison experiment has been designed on the ICDAR 2005 TrialTest dataset. The results, which are obtained before retrieving and after retrieving the missing text regions, are compared with the ICDAR 2005 annotation test dataset at the pixel level, and they are quantitatively measured by the Recall and the Precision defined in equations (15) and (16). Before retrieving the missing text regions, we can obtain the Recall 0.66 and the Precision 0.53; however, we can obtain the Recall 0.72 and Precision 0.52 after retrieving the missing text regions. It is obvious that we can greatly improve the Recall rate with a little cost of the Precision reduction.

3.5 Comparison with other approaches

In order to verify the effectiveness of the text detection scheme proposed in this work, experiments have been designed and carried out on two public datasets, and the detected results are evaluated at the word level. The performance of our method is quantitatively measured by Precision (P), Recall (R), and f-measure (F), and they are computed by using the definitions provide in the studies by Lucas et al. [29], Lucas [30], and Shahab et al. [31]. It is imperative to acknowledge that the evaluation methodology employed in the ICDAR 2011 competition differs from that of the preceding ICDAR 2005 competition. The revised evaluation scheme was introduced by Wolf et al. [32]. The performances of the proposed method and other state-of-the-art methods on the ICDAR 2005 and the ICDAR 2011 databases are shown in Tables 3 and 4, and our results in each table are highlighted with bold font.

We can know from Tables 3 and 4, that our method achieved the Precision 0.73, Recall 0.62, and f-measure 0.67, respectively, on the ICDAR 2005 dataset. Meanwhile, a similar competitive performance is obtained on the ICDAR 2011 dataset, in which we achieved the Precision 0.69, Recall 0.58, and f-measure 0.63. Comparing our approach with other state-of-the-art methods listed in Tables 3 and 4, our approach has achieved competitive performances with the most state-of-the-art methods on the ICDAR 2005 and the ICDAR 2011 datasets. Specifically, as shown in Table 3, our results on all performance indicators are better than the method of Epshtein et al. [21]. The SWT processing was first proposed in the study by Epshtein et al. [21], and they had achieved great success in scene text detection.

Figure 6 shows some typical results obtained by our method, and the detected text regions are bounded by red rectangles. These results indicate that our system is robust against large variations in text font, color, size, and geometric distortion. In addition, one of the advantages of our method is that we can detect text regions with less than three characters (as shown in the bottom row of Figure 6), while these text regions are always lost by the other proposed methods (e.g., [34]) for the assumption that the text regions always have more than three characters.

Figure 6

Typical results obtained by our method on the two public datasets.

3.6 Weakness

Although some satisfactory detection results have been obtained by using our method, there are still some false positives that cannot be eliminated, because these text candidates are very like the genuine texts, e.g., the rectangular windows in Figure 7(a) and the wire fences in Figure 7(b) are very like the digital 1; The intra-word characters may be incorrectly separated from the inter-word characters (as shown in Figure 7(c)). Our method has difficulty in detecting some exaggerated art texts (as shown in Figure 7(d)), and the proposed method does not work well when the text regions have a poor resolution (as shown in Figure 7(e) and (f)). Meanwhile, the hand-designed heuristic rules in Section 2.3.1 may affect the robustness of the algorithm. In our future work, we will try to resolve these problems by improving the detection method.

Figure 7

Some poor results were obtained by our method. (a) false alarm due to rectangular windows, (b) false alarm derived from wire fences, (c) wrongly segmented texts, (d) missing texts from exaggerated art texts, (e) missing texts due to dark resolution, and (f) missing texts from glare regions.

In addition, we found in our experiments that, as a feature extraction method, the SWT transform is strongly influenced by the accuracy of edge extraction, while the traditional Sobel edge detection operator based on differential operations lacks robustness in dealing with noisy images. Accordingly, this poses difficulties for the subsequent elimination of false alarms and for improving the performance of the algorithm. In fact, deep learning has substantially improved the accuracy of edge detection in recent years, so obtaining high-precision text edges by deep learning and then obtaining stroke width features by SWT are expected to improve text detection performance.

In Table 1, the establishment of discriminative conditions for candidate regions is predicated upon their geometric features, yielding the algorithm with diminished robustness primarily owing to the empirical selection of hyperparameter thresholds. An Interval Type-3 Fuzzy System and a new online fractional-order learning algorithm are proposed in the literature [41], which has fewer rules compared to traditional methods and learning iteration results, which has better stability for prediction tasks. Based on this, we will try to adopt the work proposed in the literature [41] for fast classification of text candidate regions in our future work.