Cross-modal multi-label image classification modeling and recognition based on nonlinear

2.1 Cross-modal multi-label classification algorithm

By simulating the label dependencies, it has become a popular strategy in multi-label image recognition to predict those labels that co-occur in a picture. Previous research has concentrated on capturing label correlation but has failed to adequately fuse picture features and label embeddings, which has a substantial influence on the model’s converge efficiency and restricts future number of co image recognition average accuracy. A single semantic label is usually insufficient to characterize multimedia data (images, text, etc.). Cross-modal retrieval systems previously relied on single-label multimodal data sources; however, several multi-label data sets with multiple modalities have recently been introduced. Multi-label datasets consider introducing a natural many-to-many interaction across different methods; i.e., each data object from one mode is closely linked to several other data points to the other method. Such correlations should be able to include this in any multi-label merge retrieval system that learns the common cross-modal subspace. Suppose X represents the instance space and L represents the q-dimensional label set. Given a set of training sets, formula (1) is shown:

The goal is to learn a multi-label classifier h : X → 2 L . However, for convenience, learn a real numerical scoring function f : X × L → R . Given an instance x i and a set of associated labels Y i , for labels y ∈ Y i , f ( x i , y ) a greater posterior probability should be output. In other words, this is true for y 1 ∈ Y i and y 2 ∉ Y i , f ( x i , y 1 ) > f ( x i , y 2 ) any sum. Using the scoring function f ( . , . ) , the multi-label classifier can be obtained as shown in formula (2):

Here, T can be a constant (for example, 0.5), or a function of inferring thresholds from the training set, which can divide the tag space pairs into related and unrelated tag sets.

2.2 Transfer learning model

The multi-label transfer learning model includes text and visual parts, as shown in Figure 1.

Figure 1

Architecture of cross-modal multi-label classification model.

2.3 Nonlinear cross-modal label calibration algorithm

The given classifier outputs a test set instance x i ( 1 < i < m ) the posterior probability set of P = { p j ∣ p j ∈ R , 1 < j < q } using the threshold calibration function T; to get the set of predicted tags Y i = { y ∣ y ∈ L } , the formula of element Y is shown in Eq. (6):

When labeling the current sample, labels higher than the threshold are added to the set of relevant labels according to the posterior probability output by the image model. The following two methods are usually used to select the threshold value: one is the fixed threshold value, the fixed threshold calibration method, which usually uses the popular threshold constant T = 0.5; the other is the dynamic threshold, which is determined by minimizing the difference between the training set and the test label base, as shown in formula (7):

Here, LCard ( X ) is the label cardinality, which is the most natural way to describe the attributes of multi-label data set, namely, the average number of labels per instance, as shown in formula (8):

In this article, the cross-modal model is combined with global preference method and mean value method. Labels are calibrated according to the threshold function (fixed threshold 0.5) and the posterior probability of the output of the image model. If the relevant label set of a sample is empty, the posterior probability average of all labels output is calculated by the image model and text model, and the labels with the largest K average probability are taken as the relevant labels (e.g., K = 1).

3.1 The data set

Deep convolutional neural network-based multi-label classification models successfully fuse image representations with label co-occurrence embeddings, proving that the model’s convergence efficiency is significantly enhanced. Furthermore, picture recognition performance has improved as compared to prior systems. In most classes, the cross-modal multi-label image classification approach improves cross-modal retrieval efficiency and can reduce overfitting. The two cross-modal mapping systems’ efficiency and logic are also proven. In this study, the image Lef2016 multi-label classification task data set was used. The training set and test set contain 1,568 and 1,083 images, respectively, and provide corresponding explanatory text. This data set label adopts the category subset of the ImageCLEF2013 pattern recognition task, namely the remaining 30 classes after the composite image category (COMP) is removed. The class codes and class names are shown in Table 1.

The single-label medical image data were derived from two other medical image processing tasks, namely the pattern classification task of ImageCLEF2013 and the subgraph pattern classification task of ImageCLEF2016. The former training set and test set contain 1,796 and 1,568 samples (excluding COMP mode), respectively, while the latter contain 6,676 and 4,166 samples.

3.2 Data preprocessing

When the image is loaded, its size is modified to 224 × 224, and the Keras pretreatment tool is used to convert it into a four-dimensional tensor. The channel-first mode is adopted, that is, (N, 3,224,224), where N is the number of instances. In order to improve the interpretability of word vectors, as many biomedical image captions as possible are collected [20]. In addition to the illustrations provided by the ImageCLEF2016 training set and test set, all illustrations were extracted from 300,000 medical literatures in ImageCLEF2013 [21]. After serialization, the Word2Vec tool is used to train the text to obtain the word vector dictionary, and the word vector quick reference table is constructed according to the description text of ImageCLEF2016, which is used as the weight of the embedded layer of the convolutional neural network, and updated synchronously in the process of network training.

3.3 Evaluation indicators

The evaluation indexes are divided into two types: case-based and tag-based, and two kinds of case-based evaluation indexes are selected. Suppose the test set is shown in formula (9):

Hamming Loss (h-loss for short): Evaluating the misclassification times of instance label, the score ranges from 0 to 1, and 0 represent the best result. The calculation formula is shown in Eq. (10):

where Δ represents symmetry difference. Mathematically, the symmetry difference of two sets is a set composed of elements that belong to only one set but not the other [22]. Macro average F1 value:F1 value represents the harmonic average of precision and recall rate. The larger the F1 value is, the more effective the classification method is. For example, when the accuracy is fixed, the larger the recall rate is, the larger the F value is, and vice versa [23]. We can test whether there is an unbalanced label distribution that may cause overfitting of some labels. F1_Macro represents the arithmetic average of all label F1 values, and its calculation formula is shown in Eqs. (11)–(14):

3.4 Experimental results and discussion

3.4.2 Cross-modal label calibration

The time complexity of cross-modal retrieval can be divided into two parts, that is, the computation time of similarity matrix and the sorting time of the retrieved data. Therefore, Figure 2 only compares the retrieval cost of Minimizing_LCard with th_0.5 on the test set. From Figure 2, it is clear that the proposed approach is great for extracting subgraph pattern information from composite images of the biomedical literature, and it can better ease the problem of overfitting and has high cross-modal retrieval performance

Figure 2

Comparison of the retrieval cost of Minimizing_LCard and th_0.5 on the test set.

Traditional methods were considered, and a comparative analysis has been done to justify the originality of the work. Many methods for investigating label dependencies have been investigated in order to reduce and enhance the label prediction space. Deep convolution architecture is utilized to learn the approximate top-k ranking objective function for multi-label picture recognition. In certain cases, CNN has been used with RNN to represent label dependencies sequentially by embedding semantic labels into vectors The fixed threshold method selects labels with a fixed threshold value of 0.5. If labels are calibrated according to the mean value of the prediction probability output by the two modal models, this calibration method can obtain multi-label classification performance with certain potential, namely, Hamming loss of 0.0161 and macro average F1 value of 0.470 (as shown in Table 3). As can be seen from Table 3, the label calibration method TH_0.5, which combines the globally preferred fixed threshold method and the highest mean probability method, is adopted in this article. Both the Hamming loss and the macro average F1 value are better than the fixed threshold method threshold_0.5, and the macro average F1 value is higher than all other methods.

Table 3

Performance comparison of threshold calibration algorithms

Methods	10FCV		Test
	H-Loss	F1Macro	H-Loss	F1Macro
Minimizing_LCard	0.0267	0.348	0.0206	0.477
Threshold_0.5	0.0226	0.326	0.0161	0.470
Highest_Probability	0.0226	0.287	0.0150	0.438
TH_0.5	0.0224	0.333	0.0157	0.488

In Table 3, a comparative analysis has been done with various methods. This is clear that under various test methods, this method has proved better. The macro average F1 value increases from 0.20 to 0.488, which is about 52.5% higher. In this article, the method of minimizing the difference between the predicted tag set and the training tag set (minimizing the difference between the tag cardinality) is used to dynamically determine the threshold. However, the Minimizing_LCard method (Table 3 method Minimizing_LCard) obtains the dynamic threshold of 0.296. According to the strategy adopted in this article, the Hamming loss and macro average F1 are 0.0206 and 0.477, respectively, which are not as good as the performance of TH_0.5. As shown in Table 3, the dynamic threshold method obtained a high macro mean F1 value of 0.477, showing certain potential, but the Hamming loss value was 0.0206, 31.2% different from the method in this article. To find out the reason, the label cardinality of the training set and the test set is checked, which are 1.46 and 1.25 respectively, and there are certain differences. Reducing the difference between the training set and the label cardinality and selecting the threshold value cannot give full play to the advantages of this dynamic threshold method in the current data set. The proposed label calibration method, TH_0.5, achieves a lower Hamming loss of 0.0157 and the highest macro average F1 value of 0.488.