Wheat freshness recognition leveraging Gramian angular field and attention-augmented resnet

2.1 Biophotonics technology

Biophoton emission (BPE) pertains to the spontaneous emission of photons by biomolecules within living organisms during their transition from a higher to a lower energy state without the need for external stimuli. Although this emission is inherently feeble, modern photoelectric detection technologies are adept at capturing its subtle intensity. The spectral spectrum of BPE predominantly spans the 200–800 nm range, exhibiting a continuous distribution. Based on the presence or absence of external optical stimulation, BPE can be broadly classified into two categories: spontaneous bioluminescence and delayed bioluminescence. Amidst the relentless advancement of ultra-weak light detection technologies and profound investigations into biological information transmission, remarkable strides have been achieved in the realm of BPE research, as evidenced by studies [9,10].

In recent years, the burgeoning biophotonics technology has progressively garnered attention for its application in grain quality analysis. Wu et al. [11] conducted pioneering analytical research on the delayed luminescence characteristics of wheat samples, exploring variations across varieties and harvest years. Their findings underscored discernible differences in ultra-weak luminescence intensity, attributed to both production year and vitality within the same variety. Furthermore, Liang et al. [12] achieved remarkable precision in distinguishing four wheat varieties through the innovative integration of ultra-weak spontaneous light signals with power spectrum feature analysis. Previous investigations have consistently highlighted significant disparities in the spontaneous photon emission levels of wheat samples from diverse storage periods. Gong et al. [13,14], leveraging biophotonic instrumentation, conducted a meticulous study on the biophotonic signals of five wheat samples with varying storage histories. By incorporating an enhanced multi-scale permutation entropy algorithm for feature extraction from the photon signals, they revealed a compelling trend: the permutation entropy values of photon counts escalated with the prolongation of storage duration. This groundbreaking discovery furnishes empirical evidence for the efficacy of biophotonics in detecting wheat freshness, opening new avenues for quality assurance in the grain industry.

2.2 Feature extraction

Currently, the cornerstone of biophotonics research revolves around utilizing biophotonic testing equipment to accumulate photon emissions from experimental samples over time, generating one-dimensional time series data. This data primarily undergoes feature extraction in the statistical or frequency domains, subsequently leveraging diverse machine learning classification algorithms, such as support vector machines and BP neural networks, for discrimination. These methodologies fall under the umbrella of Time Series Classification (TSC), where time series data are arranged sequentially in time, capturing the dynamic changes within a system or process. TSC poses a pivotal yet intricate challenge in data mining, aimed at grouping time series data into distinct categories based on their inherent characteristics. To tackle this, researchers have devised a range of sophisticated techniques, including dynamic time warping [15], which aligns time series to measure their similarity; Bag-of-SFA Symbols [16], which captures local patterns; shapelet-based methods [17], which identify discriminatory subsequences; and Collective of Transformation Ensembles [18], an ensemble approach. Additionally, a multitude of machine learning algorithms [19] have been adapted to this domain, further enriching the TSC toolkit.

However, whether these extracted features are the most suitable for determining wheat quality and whether other more suitable features can be extracted from biophoton data remain to be seen. Most of these studies are still at the laboratory stage and have not yet formed a complete theoretical system or widely applied technical standards. In recent years, deep learning, as an important branch of artificial intelligence, is widely used in multiple fields such as image recognition, natural language processing, and speech recognition. Convolutional neural network (CNN) [20] is one of the most popular deep learning models, which, unlike traditional "feature-based" classification frameworks, do not require manual feature extraction. During the deep learning process, feature extraction involves automatically learning and identifying useful features from a large amount of data through multi-layer network structures, significantly enhancing the performance of models in various AI tasks. Their powerful feature extraction and pattern recognition capabilities enable them to perform excellently in processing one-dimensional time series data, such as electroencephalogram analysis [21], anomaly detection [22], etc.

Motivated by this inquiry, could the accumulated one-dimensional biophoton data potentially be transformed into two-dimensional images, facilitating further analysis via deep learning techniques? In the realm of mathematics, there exist proven methodologies for converting one-dimensional data into two-dimensional images, including but not limited to the relative position matrix [23], recurrence plot [24], Gramian angular field (GAF) [25,26], and Markov transition field [27]. These methods offer promising avenues for transforming our biophoton data into a format more amenable to deep learning-based processing and analysis.

In this study, we endeavor to transform the amassed wheat biophoton data into two-dimensional visual representations and subsequently leverage CNNs for feature extraction and classification, thereby validating the feasibility of differentiating wheat freshness utilizing biophotonic techniques. Initially, we collect biophoton data from wheat samples spanning multiple years and employ the GAF method to transform these temporal data into two-dimensional image data. Subsequently, we select the ResNet architecture as our primary network for feature extraction and classification tasks. To further refine classification performance, we integrate the Gaussian context transformer (GCT) attention mechanism, enhancing the network's ability to prioritize and learn from salient features within its layers. This approach aims to improve the accuracy and robustness of wheat freshness discrimination based on biophotonic analysis.

The main contributions of this article are summarized as follows:

1. Propose the adoption of biophotonic technology to address the method of distinguishing wheat freshness, thereby catalyzing the transition of this advanced technology from the confines of the laboratory into practical, real-world applications.

2. Innovatively propose the use of the GAF method to encode biophotonic data, transforming the original one-dimensional time-series data into two-dimensional data, with the pixels of the GAF image maintaining the correlation between the original time-series data.

3. Leveraging the residual network architecture and the attention mechanism, the GAF-ResNet-GCT network is proposed, a novel approach designed to amplify both the feature extraction capabilities and recognition performance of deep learning networks, ushering in a new era of enhanced accuracy and efficiency.

3.1 The experimental equipment and configuration

This experiment employs the BPCL-ZL-TGC ultra-weak photon acquisition instrument as the primary measuring device, as depicted in Figure 1. The instrument is comprehensively designed, comprising four integral components: a measurement darkroom, collection equipment, signal analysis instruments, and a user-friendly front-end software interface. The measurement darkroom serves as the repository for test samples, ensuring optimal conditions for photon detection. The collection equipment, guided by software settings, diligently gathers the biophoton counts emanating from the samples within the darkroom. Subsequently, the signal analysis instruments meticulously process the collected data, performing essential analyses before seamlessly integrating the results into the intuitive front-end software for display, facilitating straightforward interpretation and analysis.

Figure 1

The BPCL-ZL-TGC ultra-weak photon acquisition instrument (taken by the author using a mobile phone in the school's food processing laboratory, permission and authorization have been obtained from the laboratory).

Prior to the measurement process, the samples underwent a rigorous pre-treatment phase within a darkroom for a duration of 30 min. This step was crucial in mitigating the disruptive effects of ambient light, ensuring the accuracy and reliability of the subsequent measurements. Environmental conditions were meticulously controlled, with an indoor temperature maintained at (28 ± 2)°C, a precise measurement temperature of (27.5 ± 0.5)°C, and an indoor humidity level kept within (45 ± 5)%. Furthermore, the test spectral wavelength range was set to encompass 380 to 620 nm, and the equipment operated at a standard voltage of 220 V, 50 Hz, ensuring optimal performance throughout the experimental procedure.

The wheat samples with storage years of 2020, 2021, 2022, and 2023 were collectively purchased from the local wheat seed market around October 2023. All relevant experimental work was initiated shortly after sample collection and completed between late 2023 and early 2024. Each precisely weighed sample (10.00 ± 0.02 g) underwent a 1,000-s measurement period, with biophoton counts meticulously recorded every second. Upon completion, we segmented the data from each experimental sample into ten equal parts, yielding a comprehensive dataset comprising 4,000 experimental samples, with 1,000 samples dedicated to each year.

Figure 2 distinctly showcases the temporal variation of photon counts for randomly selected wheat samples, spanning the years 2020–2023. The x-axis denotes time, while the y-axis represents the photon count, providing a clear visual representation of the dynamics. While these graphs exhibit characteristics akin to time series data, each year's sample data also possesses distinctive features, underscoring their uniqueness. Despite a discernible pattern in the data distribution, it is not immediately apparent which year's data corresponds to which graph. Therefore, to effectively distinguish between these samples, the implementation of feature extraction techniques and the application of classification algorithms becomes imperative.

Figure 2

Photon counts for wheat samples from each year from 2020 to 2023.

The determination of specific features to extract from the data often lacks a definitive criterion. Traditional approaches encompass the extraction of statistical features (e.g., mean, variance) and the transformation of time-series data into the frequency domain through techniques like Fourier transform and wavelet transform. However, this article innovatively harnesses the power of deep learning, which circumvents the manual selection of features by enabling the autonomous discovery of discriminative characteristics. To this end, we employ the GAF methodology, transforming one-dimensional time-series data into two-dimensional image data. This transformation facilitates the utilization of deep learning algorithms, which can then learn and extract the distinguishing features in an automated fashion.

3.2 Related concepts of GAF representation

The GAF is a method that converts one-dimensional time-series data into two-dimensional images. The core of the GAF method lies in pairing the data points in the time series, calculating the cosine of the angle between them, and representing these calculations as pixel values in an image. This method can effectively capture the dynamic and periodic characteristics of time-series data, preserving the complete information and temporal dependencies of the time series. Specifically, the GAF encompasses two forms: the Gramian angular summation field (GASF) and the Gramian angular difference field (GADF). GASF generates an image by calculating the cosine of the sum of angles between each pair of time series values in a polar coordinate system, while GADF generates an image by calculating the sine of the difference in angles.

The implementation of the GAF typically involves the following steps:

1. Preprocessing and normalization: Preprocess the raw time series data by normalizing it to a specific interval (e.g., [0, 1] or (−1, 1)) to eliminate potential impacts from different scales on the results. This is typically achieved using the min–max normalization method. Assuming there is a time series X = ( x 1 , x 2 , ⋯ , x n ), the raw time series data can be normalized using formula (1):

   (1) x ˆ i = ( x i − max ⁡ ( X ) ) + ( x i − min ⁡ ( X ) ) max ( X ) − min ⁡ ( x ) .

2. Constructing the Gram matrix: Treat each data point in the time series as a vector in a high-dimensional space and compute the inner product between these vectors to form the so-called Gram matrix. Each element of this matrix represents the similarity between two data points. The Gram matrix is expressed as shown in formula (2):

   (2) G = x ˆ 1 · x ˆ 1 x ˆ 1 · x ˆ 2 ⋯ x ˆ 1 · x ˆ n x ˆ 2 · x ˆ 1 x ˆ 2 · x ˆ 2 ⋯ x ˆ 2 · x ˆ n ⋮ x ˆ n · x ˆ 1 ⋮ x ˆ n · x ˆ 2 ⋱ ⋯ ⋮ x ˆ n · x ˆ n ,

   where each element is the inner product of two time series data points, as shown in formula (3):

   (3) 〈 x ˆ i , x ˆ j 〉 = x ˆ i · x ˆ j = ∑ t = 1 n x ˆ it × x ˆ jt .

   As the elements in the Gram matrix follow a Gaussian distribution centered at 0, it is difficult to distinguish its information from Gaussian noise. Moreover, since univariate time series are one-dimensional, the dot product cannot differentiate between valuable information and Gaussian noise, thus necessitating a change in space. Inspired by polar coordinate transformation, we will construct a bijective mapping between the one-dimensional time series and a two-dimensional space, mapping the preprocessed time series values into a polar coordinate system, where the timestamps serve as the radii, and the scaled values are converted into angles through the inverse cosine function. The transformation formula is as follows:

   (4) θ i = arc cos ( x ˆ i ) , x ˆ 1 ∈ X ˆ ,

   (5) r i = t i N , i ∈ N ,

   where t i is the timestamp.

   As any operation analogous to an inner product inevitably combines the information from two different observations into a single value, the following definitions are typically used for performing inner product calculations. This enables better preservation of the complete information and temporal dependencies within the time series, ensuring no information is lost

   (6) GASF = ( cos ⁡ ( θ i + θ j ) ) n × n , GADF = ( sin ⁡ ( θ i − θ j ) ) n × n .

3. Generate GAF image: Based on the definitions of GASF or GADF mentioned above, the cosine or sine values of the angles for each pair of values in the polar coordinate system can be calculated to generate a two-dimensional image

Figure 3 presents the GASF and GADF images corresponding to the wheat biophoton data for different years shown in Figure 2.

Figure 3

The GASF and GADF images corresponding to biophoton data shown in Figure 2.

The advantage of the GAF approach lies in its unique ability to seamlessly integrate temporal information from time-series data with the prowess of deep learning algorithms, which have garnered immense popularity in image processing. This fusion not only elevates the efficiency of feature extraction but also streamlines the entire data analysis process. By converting one-dimensional time-series data into two-dimensional image representations, GAF paves the way for the application of advanced deep learning techniques, thereby simplifying and enhancing the analysis.

3.3 ResNet

CNNs are widely utilized models in deep learning, characterized by their core features of local connectivity, parameter sharing, and automatic feature extraction. Despite CNNs' powerful feature extraction capabilities, they still face challenges when dealing with extremely deep networks. As the number of network layers increases, the training error tends to gradually decrease and reach saturation, but then as the number of layers further increases, the training error rises again, manifesting as the so-called degradation problem. The emergence of ResNet [28] effectively addresses this issue by introducing shortcut connections, which not only ensures the depth of the network but also optimizes gradient propagation, making the network easier to train and converge. This is of great significance for complex tasks in practical applications, such as high-precision image classification and large-scale image processing.

Specifically, the residual block in ResNet comprises two paths: one is the path for normal convolution operations, which typically begins with a convolution layer, followed by a batch normalization layer and a ReLU activation function. This combination helps reduce internal covariate shifts and accelerates the training process. The ReLU activation function adds nonlinearity, enabling the network to learn more complex patterns, followed by another convolution operation and batch normalization. The other path is an identity mapping path. As shown in Figure 4, finally, the outputs of these two paths are added together to form the final output of the residual block. This output is then activated by a ReLU activation function, generating the final output feature of the residual block.

Figure 4

Residual block in ResNet.

As illustrated in the Figure 4, x represents the input, and F(x) denotes the output of the residual block before the second activation function. The core formula of the residual block can be expressed as H ( x ) = F ( x , ω i ) + x , where x and H(x) are the input and output of the block, respectively, F ( x , ω i ) represents the operations performed by the stack of convolutional layers, and ω i are the weights of these layers.

This design has two key advantages: First, it simplifies the learning process because instead of learning a complete output, it learns a residual mapping. Second, it provides a shortcut that allows skipping some layers, alleviating the problem of gradient vanishing, enabling deeper networks to be effectively trained. The residual blocks in ResNet are mainly classified into two types: basic residual blocks and bottleneck residual blocks. A basic residual block consists of two 3 × 3 convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The input passes through these convolutional layers and is then added to the original input, before being output through a ReLU activation function. The bottleneck residual block contains three convolutional layers: The first 1 × 1 convolutional layer is used to reduce the number of channels, followed by a 3 × 3 convolutional layer, and finally, another 1 × 1 convolutional layer to restore the original number of channels. This design reduces computational complexity while retaining sufficient information.

3.4 GCT attention module

The primary role of introducing attention mechanisms into CNN is to enhance the model's ability to focus on critical information within the input data, thereby improving the model's performance and efficiency, such as the channel attention mechanism SE [29]. However, literature [30] points out that the SE module tends to learn a negative correlation between features, meaning that when the difference between the global context and the average value increases, the obtained attention excitation value will correspondingly decrease. Based on this correlation, literature [31] proposes GCT for modeling global context, where GCT can directly replace the two fully connected layers in SE with a Gaussian function that incorporates negative correlation. Compared to SE, GCT is able to better learn the negative correlation between global context and attention activation values with fewer introduced parameters, thereby enhancing the model's expressive power.

The basic structure of GCT is illustrated in Figure 5. GCT comprises three parts: GCA (global context aggregation), normalization, and Gaussian context excitation. Specifically, a feature map of dimension X ∈ R C × H × R undergoes global information aggregation in the spatial domain of the feature map through GCA to obtain z, as shown in the following equation:

Figure 5

Schematic diagram of the GCT network structure.

In the equation, C denotes the number of channels, while H and W represent the width and height of the feature map, respectively. Afterwards, z undergoes feature normalization using the function norm(), as shown in the following equation:

In the equation, z − u represents the mean shift, while u = 1 C ∑ k = 1 C z k denotes the mean of z, and σ is the global context standard deviation used to ensure a stable distribution of the function's output norm(). It is calculated through

The normalized result z ˆ is then used as input to a Gaussian function G ( ∙ ) to obtain the attention activation value g, as specifically shown in the following equation:

Combining all the above operations into a single equation, we construct the GCT module, as shown in the following equation:

3.5 Proposed GAF-Resnet-GCT network

This article proposes the integration of the GCT attention mechanism within residual blocks, constituting a novel architecture dubbed the GAF-ResNet-GCT network. This attention mechanism augments ResNet's capabilities by enabling it to autonomously discern and extract more discriminative features across various levels, dynamically learn optimal feature weights, and subsequently adjust the significance of features across channels. As a result, the representational prowess of the network is significantly bolstered. The diagram of the network structure is shown in Figure 6.

Figure 6

The diagram of the proposed GAF-Resnet-GCT network structure.

The entire network architecture consists of six residual blocks. The GAF image is initially passed through a standard convolutional layer for extracting initial features, followed by six residual blocks. In the third and fifth residual blocks, the first convolution halves the input image size and doubles the number of channels; hence, a 1 × 1 convolution is used to implement the shortcut connection. All other connections use identity mappings directly. Within the residual blocks, batch normalization and the ReLU function are applied after the convolutional layers. In the output stage, global average pooling is used to adjust the multi-channel data into a one-dimensional format, which is then connected to three fully connected layers. The numbers of neurons in these fully connected layers are 1,024, 512, and the number of classification categories (four categories in this experiment), respectively. The middle fully connected layer uses the ReLU activation function, while the final layer employs the SoftMax function to calculate the maximum probability value. In the residual module, the GCTmodule receives the feature map from the previous layer as input and generates a channel attention weight matrix through an attention layer. This weight matrix is then used to perform a weighted summation on the input feature map, resulting in a feature map that contains global context information. This process not only enhances the spatial representation capabilities of the features but also enables the network to focus on the information regions that are more critical to the task.

3.6 Diagnostic procedure of proposed method

Figure 7 illustrates the diagnostic procedure of the proposed GAF-Resnet-GCT.

Figure 7

The diagnostic process of the proposed GAF-Resnet-GCT. Source: Created by the authors.

The detailed procedures for freshness diagnosis are as follows:

1. To collect photon data from wheat samples spanning multiple years using a BPCL-ZL-TGC ultra-weak photon system

2. To transform biophoton data into two-dimensional images used GAF.

3. Construct the GAF-Resnet-GCT model, configure the network hyperparameters such as learning rate, the number of training iterations, and initialize the network weights.

4. The GAF-Resnet-GCT is trained utilizing the samples from the training set, and the deviation between the network's output and the sample labels is computed utilizing the cross-entropy loss function. This error is then propagated back through the network according to the Adam gradient descent algorithm, leading to an update of the network's weight parameters. The network's performance is assessed using the validation set after each training iteration; however, error backpropagation and updates to the network weights are not executed during this evaluation phase.

5. The fully trained GAF-Resnet-GCT is employed to evaluate the test set and provide the ultimate diagnostic outcome.

3.7 Evaluation metrics

Accurately assessing the performance of a model is crucial. To ensure a comprehensive evaluation of the model, this article adopts Accuracy, Precision, Recall, and F1 Score as key indicators to evaluate the model's performance. Each of these metrics reflects the model's performance from different angles, complementing each other and collectively providing a comprehensive measurement standard for model evaluation. Their calculation formulas are as follows:

where TP, FP, FN, and TN, respectively, represent the number of true positives, false positives, false negatives, and true negatives.

4.1 Comparative experiments with other methods

To rigorously evaluate the efficacy of our proposed algorithm, we selected a diverse array of deep learning algorithms (CNN, ResNet) and traditional machine learning algorithms (KNN, SVM, BP) as benchmarks for classification performance comparison. Notably, traditional machine learning approaches necessitate manual feature selection, which encompassed six statistical features: median, mean, quartile deviation, mean deviation, variance, and coefficient of variance in this study. This comprehensive evaluation framework allowed us to assess the strengths and limitations of our algorithm in comparison to both modern deep learning and established machine learning techniques. Table 1 presents the experimental results.

The table reveals that among the three traditional methods – KNN, SVM, and BP – the accuracy rates stand at 81.6, 87.2, and 89.2%, respectively, with their precision, recall, and F1 scores demonstrating relative parity. Notably, the BP algorithm and SVM algorithm outshine in classification performance. These methods rely on six manually selected statistical features as inputs into various machine learning algorithms. Nevertheless, the manual selection of features from the biophotonic time-series data, lacking optimality, limits the optimality of classification results. Consequently, there remains a pressing need for feature optimization through diverse methodologies to further enhance performance.

Both CNN and ResNet, rooted in deep learning algorithms, eliminate the dependency on manual feature selection by enabling the network to learn autonomously. The accuracy rates of these two methods stand at 91.2 and 91.8%, respectively, with ResNet marginally surpassing CNN due to its innovative use of residual structures. These structures facilitate more efficient updates and iterations of network weights, contributing to slightly higher accuracy. Notably, even with a moderately deep network architecture in this experiment, the effectiveness of both methods is expected to escalate with an increase in the number of layers. Furthermore, this study validates the feasibility of transforming one-dimensional biophotonic data into two-dimensional GAF graphs and subsequently leveraging deep learning for feature extraction and classification, underscoring the potential of this proposed approach.

Upon integrating the attention mechanism GCT, the GAF-ResNet-GCT network achieved a notable improvement in accuracy, reaching 93.1%, marking a 1.3% increase over the baseline ResNet model. This enhancement underscores the power of the attention mechanism in enhancing the network's ability to prioritize key features and elevate classification precision, ultimately translating into higher classification accuracy. Moreover, this achievement underscores the significant potential of incorporating attention mechanisms into deep learning frameworks, as proposed in this article, to bolster overall performance. To make the results clearer, Figure 8 presents bar charts corresponding to the results of different algorithms in Table 1.

Figure 8

Experimental results using different methods.

4.2 Comparative experiment between GASF and GADF

To assess the impact of the two GAF methods – GASF and GADF – on experimental outcomes, additional experiments have been conducted. While previous tests employed GASF, the hyperparameters utilized in this section mirror those for training GASF images, ensuring a fair comparison. Table 2 summarizes the results. Initially, employing GADF to generate the two-dimensional matrix and inputting it into the proposed network revealed a notable improvement in accuracy, reaching 94.3%, a 1.3% increase over GASF. A closer examination of the two-dimensional graphs from Figure 3 reveals that GADF outperforms GASF in color intensity, detail rendering, and line proportion, contributing to its superior detection accuracy. Furthermore, an intriguing discovery was made when both GASF and GADF-generated graphs were shuffled and fed into the network: the accuracy climbed even higher to 94.5%, suggesting a slight yet significant enhancement over exclusive GADF use, demonstrating the potential of combining both methods.

4.3 Confusion matrix for classification results

Figure 9 illustrates the confusion matrices for the final classifications achieved using GDSF-ResNet-GCT and the BP network for comparative purposes. Analyzing the GDSF-ResNet-GCT matrix, we observe a marked improvement in accuracy, with just three misclassifications in 2020, two in 2021, and two in 2022, whereas all samples from 2023 were correctly identified. This heightened recognition accuracy hints at enhanced internal biological activity and more robust BPE in the newer wheat seeds. Conversely, the BP method's matrix reveals a higher degree of misclassification, with four errors in 2020, four in 2021, three in 2022, and two in 2023, underscoring the reduced freshness and weaker biometric signals emanating from older wheat samples compared to their fresher counterparts.

Figure 9

Confusion matrix.