A hybrid detection algorithm for 5G OTFS waveform for 64 and 256 QAM with Rayleigh and Rician channels

Arun Kumar; Haya Mesfer Alshahrani; Faiz Alotaibi; Aziz Nanthaamornphong

doi:10.1515/eng-2024-0008

Article Open Access

A hybrid detection algorithm for 5G OTFS waveform for 64 and 256 QAM with Rayleigh and Rician channels

Arun Kumar , Haya Mesfer Alshahrani , Faiz Alotaibi and Aziz Nanthaamornphong

Published/Copyright: April 3, 2024

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information

From the journal Open Engineering Volume 14 Issue 1

Abstract

Signal detection in orthogonal time frequency space modulation is one of the critical aspects for enhancing the throughput of the beyond fifth-generation framework, especially in challenging environments characterized by high mobility and multipath propagation. In this work, we proposed a hybrid detection algorithm by combining zero-forcing equalization (ZFE) and minimum square error equalization (MMSE), also known as ZFE–MMSE, with Rician and Rayleigh channels for 64 and 256 quadrature amplitude modulation. The combination of ZFE and MMSE allows for a more comprehensive approach to equalization. ZFE addresses channel-induced distortions, while MMSE tackles the impact of noise. Together, they enhance the overall performance of the equalization process, resulting in improved signal detection accuracy. The analysis and comparison of the simulation parameters with traditional detection systems include bit error rate, power spectral density (PSD), and complexity. The projected algorithms achieve a signal-to-noise ratio and a PSD gain of 2 dB and −800, respectively, outperforming the traditional detection techniques.

Keywords: OTFS; detection method; BER; PSD

1 Introduction

In wireless communication, managing the complex time-frequency dispersion resulting from channel flaws is the main challenge in signal recognition, particularly in complex modulation schemes like orthogonal time-frequency space (OTFS) [1]. More advanced equalization and detection algorithms are required in order to efficiently extract the transmitted data and decipher the received signals. Furthermore, because of these algorithms’ high processing demands and computational complexity, implementing them in real time can be challenging. The handling of Doppler shifts and delay fluctuations, which are frequent in high-mobility situations, complicates things [2]. The effective resolution of these issues is essential for the successful integration of OTFS into practical communication systems. A state-of-the-art method for wireless communication that enhances data transfer in challenging settings is called OTFS modulation. In contrast to conventional modulation methods, which mainly concentrate on either frequency or time, OTFS simultaneously and creatively uses both dimensions. The “delay-Doppler spreading” method in OTFS disperses data symbols over the time–frequency plane. This disperses data in a way that makes it extremely resistant to fading and echoes, two types of channel impairments. Because of this, OTFS performs better in difficult situations like high-mobility communication or interior spaces with plenty of reflections. Utilizing the spatial variability of the time–frequency domain is the basic notion behind OTFS [3]. Information is encoded in this space to strengthen the signal and make it more immune to interference, allowing for dependable communication even in situations where other approaches might not work. This is especially helpful for applications that depend on reliable, high-quality communication, such as 5G networks, Internet of Things (IoT) devices, and driverless cars. OTFS has the potential to completely transform wireless communication, which could result in increased data speeds and dependability in a variety of real-world scenarios [4]. In order to detect signals in OTFS modulation, one must extract the broadcast data from the received signal while taking into account the intricate time–frequency structure that OTFS introduces. One needs sophisticated processing methods that take advantage of delay–Doppler spreading to do this [5]. In order to restore the original data symbols and undo the effects of channel distortions, these algorithms use two-dimensional (2D) equalization techniques. OTFS detection works well in places with a lot of movement and multiple paths because it reduces the issues caused by changing delays and Doppler shifts. This makes wireless communication systems more reliable and efficient [6]. The OTFS equalization technique reverses the dispersion of signals over the time–frequency plane by using a 2D equalization process. To compensate for the distortions and enable the recovery of the sent data, it makes use of information about the Doppler characteristics and latency of the channel [7]. The OTFS detection algorithm estimates the sent symbols by using the equalized signal. To reliably decode the received data, it requires complex processing methods that frequently make use of matrix operations and computational optimizations [8]. The article is structured as follows: Section 1 introduces the topic and the proposed algorithms. Section 2 provides the system model of an OTFS waveform and the hybrid algorithm zero-forcing equalization–minimum square error equalization (ZFE–MMSE), which includes the mathematical formulation of the hybrid method for OTFS waveform. Section 3 reviews the literature and analyses published articles in the field. Section 4 includes the simulation results, including complexity, bit error rate (BER), and PSD analyses. Section 5 concludes and provides future work.

2 Literature review

The published articles on the topic of 5G OTFS signal detection are listed in this section, as indicated in Table 1. As the published and proposed articles are examined, it can be shown that the proposed article is the only one that makes use of the signal detection analysis on the OTFS signal with Rayleigh and Rician channels.

Table 1

Literature review

References	Remarks	Relationship with proposed work
[9]	This study’s authors investigate the problem of low-complexity OTFS modulation signal detection using deep neural networks (DNN). Additionally, DNN architecture is examined, in which each symbol multiplexed in the delay–Doppler grid is associated with a unique DNN. As a result, learning fewer parameters and requiring less effort than a full DNN that considers every symbol in an OTFS frame simultaneously is the considered symbol-level DNN.	In order to improve the throughput of the OTFS framework, a detection method is implemented in the proposed work and Naikoti and Chockalingam [9]. The PSD of the suggested approach, which is obtained in the work for Rayleigh and Rician that is presented, has not, however, been covered in the latter publication.
[10]	The design of efficient signal detectors for index modulation (IM) systems in OTFS IM is presented in this article. First, OTFS-IM is assessed for the linear equalizer based on the MMSE and the associated soft-aided decision. To further improve the performance, a vector-by-vector-aided message passing (VV-MP) detector and associated soft decision are created. Every instant message symbol in this detector is viewed as a full vector that is utilized for message forwarding and computation. It is shown that the OTFS-IM system, which employs the recommended detectors, can perform better in terms of BER than the OTFS and orthogonal frequency division multiplex (OFDM) with IM systems based on simulation results.	A linear equalization method was suggested in the study of Wu et al. [10], and the framework’s BER is estimated. The hybrid algorithms are developed, and parameters like complexity, PSD, and BER are examined in the suggested article.
[11]	Reliable communications are provided by the innovative 2D IM technology known as OTFS. Underwater acoustic (UWA) channels include multipath delays and Doppler shifts that are orders of magnitude larger than those in wireless radio transmission. Serious time- and frequency-selective fading will result from this. The receiver has to recover the corrupted OTFS signal when there is ISI or inter-carrier interference (ICI). The proposed technique uses feature information from received OTFS signal sequences to train the neural network to recognize signals. The numerical results demonstrate that the SC-CNN-BiLSTM-based OTFS detection strategy performs with a reduced BER when compared to 2D-CNN, FC-DNN, and conventional signal identification approaches.	While the development of OTFS detection methods for UWA applications was covered in the study of Zhang et al. [11], the focus of our article is on signal detection in 5G systems. This article improves the framework’s throughput. However, additional parameters like PSD and system complexity are also examined in the suggested article.
[12]	The BPICNet OTFS detector, which integrates the concepts of parallel interference cancellation, Bayesian inference, and NN, was introduced by the authors of this study. Simulation findings show that the proposed detector outperforms the state-of-the-art.	Kosasih et al. [12] analysed the BER after using machine learning (ML) algorithms to find the signal in OTFS. On the other hand, we have created a hybrid method for the OTFS 5G waveform in the suggested article. PSD analysis is done using both the Rician and Rayleigh channels, aside from BER.
[13]	This article investigates a high-Doppler aerial communication network whose mobile nodes can reach relative speeds of more than 1,200 m/s. The system in question is a mobile ad hoc network that enables nodes to join and leave the network while they are in the air. An antenna array is also put on each node to enable directed communication between mobile nodes. Physical layer properties, the number of delay-Doppler bins in the DD domain used for OTFS IM, the combination of systems with OTFS IM, and the impacts of directed versus two-ray channels on the BER are assessed. It is shown that the combination of MIMO-OTFS IM and OTFS IM over a two-ray channel provides a stable, low-BER aerial communication network.	Chu et al. [13] look at how well the OTFS waveform works for aerial communication networks. But for this article, we have implemented hybrid detection systems for the OTFS 5G waveform that use both Rayleigh and Rician channels.
[14]	In the Monte Carlo Markov chain using the Gibbs sampler, the variational mean-field approximation and variational Bayesian expectation-maximization techniques are applied. Finally, an efficacy comparison is made between the performance and complexity of the proposed algorithms with those of the current methods. Experimental testing in high-mobility scenarios and low-latency applications shows that the suggested methods perform significantly better in terms of bit error rate and normalized mean square error, although they are slightly more expensive in terms of complexity load.	The norm minimization approach was used in the study of Ouchikh et al. [14] to improve the framework’s throughput and complexity performance. Nonetheless, we have developed a hybrid approach in our work to improve the framework’s PSD, BER, and complexity. The two publications were cantered on BER performance enhancement.
[15]	The work that is being presented implements novel hybrid algorithms for 16 × 16, 64 × 64, and 256 × 256 MIMO topologies. QR-maximum likelihood detection (QR-MLD), QR-MMSE, QR-ZFE, and QR-beam forming (QR-BF) are some of these methods. The hybrid algorithms achieved an efficient BER of 10⁻³ at an SNR of 2.9 dB with little complexity. Additionally, the proposed algorithms are compared with conventional methods. It should be noted that the QR-MLD performs 3 dB better than the MMSE. It is found that the QR-MLD significantly improved the framework’s throughput gain and provided good performance.	For various MIMO structures, Kumar et al. [15] used QR-based linear minimization approaches. Using the ZFE–MMSE method with the Rician and Rayleigh channels, the suggested method did improve the BER, PSD, and complexity of the framework.
[16]	MIMO is the most important technology for reaching the high data rate needed for mobile communication in the next generation. It has many antennas on both the sending and receiving ends. It is challenging to identify signals in these systems. Here, we provide a novel QRM-MLD technique with maximum likelihood detection to reduce the latency and complexity of the large MIMO system. The simulation results demonstrate that the proposed system achieved reduced latency and complexity with a minor influence on bit error rate performance when compared to existing approaches.	Kumar [16] used the Rayleigh channel technique in conjunction with QRM-MLD for MIMO systems. We have used the ZFE–MMSE approach with the Rician and Rayleigh channels in the suggested article. The decrease in latency and framework complexity is the main focus of both papers. Nonetheless, the suggested approach produced a better result.
[17]	This research uses an MIMO system to lower the signal fading rate and increase the transmitted signal performance. To reduce the impact of channel estimate mistakes, the ZF and MMSE detection techniques are employed. MATLAB software was used to run the simulation. According to the simulation results, the suggested MIMO VBLAST scheme outperforms the conventional system in terms of successful message delivery and network throughput.	Linear detection methods are used in the MIMO-OFDM framework by Farzamnia et al. [17]. It can be observed that as the number of antennas in a framework increases, so does its complexity and system throughput. We have integrated the ZF and MMSE algorithms for the OTFS waveform in the Rician and Rayleigh channels in our suggested paper. When compared to the earlier article, the suggested article performed better.
[18]	This research uses an MIMO system to lower the signal fading rate and increase modified Walsh-Hadamard code division multiplexing (MWHCDM). An ICI reduction plan is put out in this study. The suggested method lowers the ICI by using ZF and MMSE detection on the MWHCDM signal structure. The computer simulation results showed that in the periodic blockage channel, the suggested one achieves both improved spectrum efficiency and outstanding bit error rate performance in the transmitted signal.	For the MWHCDM framework, Kojima and Yamada [18] created a ZFE and MMSE approach. The ZFE and MMSE for OTFS waveforms were integrated into our article, and they were then compared to the traditional ZFE and MMSE.

The following lists the suggested article’s contributions:

As far as we know from the currently available literature, the suggested study is the first to apply hybrid detection methods (ZFE–MMSE) for the OTFS framework with Rician and Rayleigh channels.
Most of the articles solely focused on improving the BER performance using QPSK and 64 quadrature amplitude modulation (QAM). For 64 and 256 QAM, the proposed algorithm predicts the BER and PSD performance in Rayleigh and Rician channels. It was found that in the Rician channel, the proposed algorithm works better.

3 System model

In OTFS IM, OTFS equalization plays a crucial role in the signal-detecting process. OTFS is a novel wireless communication method that improves data transmission in difficult settings by utilizing both the time and frequency dimensions. It is necessary to comprehend the complexities of OTFS IM in order to comprehend OTFS equalization. In OTFS, a method called delay–Doppler spreading is used to disperse data symbols around the time–frequency plane. The purpose of this spreading is to overcome the difficulties caused by variable channel conditions, multipath propagation, and high mobility. However, it also introduces time–frequency dispersion, necessitating the use of equalization methods in the signal identification process. The following are the main steps in the OTFS equalization process:

Channel estimation: The initial stage involves estimating the communication channel’s properties, particularly its latency and Doppler profiles. Understanding how the signal has been distributed in the time–frequency plane requires knowledge of this information.
2D equalization: To reverse the time–frequency dispersion, OTFS equalization uses a 2D equalization procedure. It applies equalization filters to mitigate the distortions caused by the channel while simultaneously taking into account the time and frequency domains.

The mathematical equations for the OTFS equalization system model can be quite complex and are typically defined by specific OTFS equalization algorithms. Below is a simplified representation of the OTFS equalization system model, providing an overview of the key components involved. The received signal is represented by the time (t) and frequency (f) dimensions in the time–frequency domain ( Y ( t , f ) ) . The time–frequency domain of transmitted signal is represented as ( X ( t , f ) ) . This is the original transmission of the signal. Channel impulse response in Doppler frequency and time-delay domains ( h ( τ , υ ) ) : This describes the channel’s dimensions, such as latency ( τ ) and Doppler frequency ( υ ) . In the time-delay and Doppler frequency domain ( W ( τ , υ ) ) , The channel estimation determine the design of the equalization filter. It seeks to undo the channel’s impact on the signal that is received. The ⊗ symbolizes the convolution procedure, which blends the equalization filter with the received signal. The following equation can be used to illustrate the system model:

(1) Y ( t , f ) = X ( t , f ) ⊗ ( h ( τ , υ ) × W ( τ , υ ) ) .

In this equation, Y ( t , f ) is the received signal after equalization, X ( t , f ) is the original transmitted signal, h ( τ , υ ) is the channel impulse response, W ( τ , υ ) is the equalization filter, and the convolution operation ( ⊗ ) is used to perform the operation. This equation showcases how the received signal Y ( t , f ) is obtained by convolving the transmitted signal X ( t , f ) with the product of the channel impulse response and the equalization filter. In order to precisely identify the transmitted data symbols in OTFS IM, the objective is to efficiently reverse the effects of the channel. Time–frequency dispersion effects are offset by the equalization process. By matching the received signal with the original transmission signal, it essentially “unspreads” the received signal. In OTFS IM, OTFS equalization is essential for precise signal identification. It improves wireless communication in difficult situations by reducing the impact of channel impairments and facilitating accurate data recovery, which makes it appropriate for applications such as IoT, autonomous vehicles, and 5G networks. It should be highlighted, nonetheless, that OTFS equalization might have a significant computational complexity, necessitating the use of effective methods for real-time implementation.

The Rayleigh fading channel is characterized by a random amplitude, typically modelled as a complex Gaussian random variable with zero mean and unit variance. The received signal h in the Rayleigh channel can be expressed as follows:

(2) h = h i + j h q ,

where h i and j h q are independent and identically distributed Gaussian random variables with zero mean and variance 1 / 2 . The magnitude of the channel gain ∣h∣ follows a Rayleigh distribution. The probability density function (PDF) of the Rayleigh-distributed random variable is given by

(3) f ∣ h ∣ = 2 x σ 2 e − x 2 σ 2 ,

where σ 2 is the variance of the Gaussian random variables.

The Rician fading channel incorporates a dominant line-of-sight (LOS) path in addition to the scattered paths. The channel gain is modelled as a sum of the direct path and the scattered paths, where both components follow Gaussian distributions. The received signal h r is given by

(4) h r = K h LOS + 1 − K h NLOS ,

where K is the Rician factor, representing the ratio of the power of the LOS component to the power of the non-line-of-sight (NLOS) components, h LOS is a complex Gaussian random variable representing the LOS component, and h NLOS is a complex Gaussian random variable representing the NLOS components. The magnitude of the channel gain h r follows a Rician distribution. The PDF of h r is given by

(5) f | h r | ( x ; K ) = 2 K Ω exp − K − K + x 2 Ω I 0 2 k x 2 Ω ,

where Ω is the non-centrality parameter, and I 0 is the modified Bessel function of the first kind with order 0.

3.1 Hybrid detection methods

ZFE is a linear filter with a channel-response inversion architecture. This means that the transmitted signal should be exactly replicated in the ZFE filter’s output, free from any channel distortion. ZFE, however, has the potential to magnify noise, which could lower the signal-to-noise ratio (SNR) and cause inaccuracies in the data that are detected. A statistical method called minimum square error equalization (MMSE) is used to calculate a signal’s value from noisy observations. The MMSE estimator minimizes the mean squared error between the estimated and true signals. The MMSE estimator can be used to estimate the broadcast symbols from the received noisy signal in the context of OTFS detection. An approach that shows promise for enhancing OTFS signal detection performance is the hybrid ZFE–MMSE detection method. It works especially effectively in applications like optical fibre and wireless communications, where large data rates and low error rates are necessary. Using both the ZFE and MMSE algorithms together in the ZFE–MMSE hybrid detection method makes it better at finding OTFS signals. Prior to removing the ISI brought on by the channel, the received signal is pre-processed using the ZFE filter. The MMSE estimator receives the ZFE filter’s output, after which it estimates the transmitted symbols. In wireless communication, signal processing techniques like ZFE and MMSE are combined. ZFE effectively removes interference but might be noise-sensitive, whereas MMSE offers a strong noise reduction but might not be able to remove interference entirely. The advantages of both approaches are combined in the hybrid approach. ZFE lowers interference first, while MMSE minimizes noise effects to further hone the signal. In the end, this combination improves the quality and dependability of wireless communication by optimizing signal recognition in difficult situations and providing a good compromise between noise mitigation and interference reduction. In order to lessen the issues caused by multipath propagation and interference, wireless communication uses the signal processing technique known as zero-force equalization. Enhancing signal detection and eliminating inter-symbol interference (ISI) are the two main objectives of this linear equalization technique. ZFE operates on the fundamental principle of “pushing” the equalization coefficients to zero by appropriately adjusting them in order to minimize interference. Recovering signals with zero-force equalization is fairly simple and efficient; it functions best when the channel is well understood. However, it may not perform as well in channels with high dispersion or rapidly changing conditions, where more advanced equalization methods may be needed to accurately recognize the signal. There are three primary steps in the process:

Channel estimation: Initially, the channel’s impulse response is calculated. This impulse response represents the distortion of the broadcast signal as it passes through the wireless medium.
Coefficient calculation: The equalization coefficients are then computed based on the channel estimation. These equations are designed to minimize channel-induced distortions and interference so as to align the transmitted symbols with the received signal.
Equalization: To lessen interference and ISI, the equalizer applies the estimated coefficients after processing the input signal. Because of the signal’s remarkable similarity to the initial transmitted symbols, precise detection is made possible.

ZFE is a useful instrument that provides a simple and affordable way to enhance signal detection in some situations where channel conditions are reasonably steady and well-characterized. It might not work well in more difficult settings, though, when sophisticated equalization methods are required to get beyond drastically altered channel conditions or fast-changing conditions. The zero-force equalizer output, Ŷ ( t , f ) , in the time–frequency plane is computed as follows:

(6) Ŷ ( t , f ) = W ( t , f ) × Y ( t , f ) .

The ZFE coefficient, W ( t , f ) , in the time–frequency plane is calculated as follows:

(7) W ( t , f ) = H ( t , f ) − 1 ,

where H ( t , f ) − 1 represents the inverse of the channel response in the time–frequency plane. This is used to cancel the effects of the channel and interference. The MMSE equalizer for OTFS signal detection can be expressed as follows, focusing on the time-frequency plane aspects. The MMSE equalizer output, Ŷ ( t , f ) , in the time-frequency plane is computed as follows:

(8) Ŷ ( t , f ) = W ( t , f ) × Y ( t , f ) .

The MMSE equalizer coefficient, W ( t , f ) , in the time-frequency plane is calculated as follows:

(9) W ( t , f ) = E [ X ( t , f ) Y ( t , f ) * ] / E [ | X ( t , f ) | 2 ] .

In equation (9) , E [ ] denotes the expected value (mean), Y ( t , f ) * represents the complex conjugate of the received symbol, the numerator , E [ X ( t , f ) Y ( t , f ) * ] , represents the cross-correlation between the transmitted and received symbols in the time-frequency plane, and the denominator, E [ | X ( t , f ) | 2 ] , is the expected power of the transmitted symbol in the time-frequency plane. The ZFE–MMSE hybrid detection method has several advantages over traditional detection methods, such as

Enhanced SNR: By attenuating noise and enhancing the intended signal, the ZFE filter lowers mistake rates and enhances SNR.
Lower error rate: By providing more precise estimations of the transmitted symbols, the MMSE estimator lowers the error rate even more.
Robustness to channel variations: Compared to conventional detection techniques, the ZFE–MMSE hybrid detection method is more resilient to changes in the channel.

4 Simulation results

We extensively analysed the recommended detection techniques in this study using Matlab 2016. We examined the PSD and BER parameters. To perform a simulation, we need to account for 10,000 symbols, 64 and 256 QAM, 64-FFT, Rayleigh, and Rician channels. We examined the framework’s throughput when various detection techniques were used on it in Figure 1. Under the Rayleigh channel for 256 QAM, the BER of 10⁻³ is obtained for the SNR of 5.8 dB, 7 dB, 8.1 dB, and 9.6 dB. With gains of 1.2, 2.1, and 3.8 dB compared to the original OTFS signal, the suggested hybrid method (ZFE–MMSE) showed the best results. This can be seen in the graphs.

Figure 1

256-QAM BER with the Rayleigh channel.

The BER performance of the OTFS system’s 256-QAM Rician channel is shown in Figure 2. It is evident from this instance that the OTFS outperformed the Rayleigh channel in terms of BER performance. The suggested hybrid technique yields SNR increases of 1.3, 2.8, and 4.2 dB. Therefore, it has been verified that the suggested algorithm performs better than the traditional ZFE and MMSE approaches. In a Rayleigh channel, the scattered multipath components are the only ones. However, the received signal of a Rician channel consists of scattered multipath signals in addition to a strong LOS component. In comparison to Rayleigh channels, the BER performance for 256-QAM IM is superior in Rician channels. This is because the effects of fading are mitigated by a dominant LOS component. The LOS element lessens the chance of inaccurate symbol decoding due to fading by acting as a trustworthy reference. The Rician channel’s inherent variety reduces the impact of fading and enhances BER performance because the Rayleigh channel lacks this fixed LOS component.

Figure 2

256-QAM BER with the Rician channel.

Figure 3 displays the 64-QAM OTFS BER curves under the Rayleigh channel. Given that low-order IM schemes have bigger symbol spacing and are less sensitive to noise, it is evident from the graph that they performed better than high-order IM. They reduce BER by offering more space between symbols, which makes it simpler to discern between them. High-order IM, on the other hand, has a greater bit rate because it packs more bits per symbol, which increases its susceptibility to interference and noise. At SNRs of 3.3, 4.5, 5.7, and 7.8 dB, the BER of 10⁻³ is obtained. Therefore, the experimental findings verified that the suggested hybrid strategy outperforms the ZFE and MMS approaches.

Figure 3

64-QAM BER with the Rayleigh channel.

The BER performance of a 64-QAM OTFS system operating in a Rician channel is examined in Figure 4. It is observed that the 64-QAM Rician channel outperforms the 256-QAM Rayleigh channel in terms of throughput. In addition, the SNR gain is maximized in comparison to the OTFS system. To get the BER of 10⁻³, SNR increases of 1.4, 2.8, and 6 dB are obtained. Hence, when compared to the conventional system, the suggested method produced an effective and optimal BDER performance.

Figure 4

64-QAM BER with Rician channel.

As illustrated in Figure 5, the power spectral density (PSD) in a Rayleigh channel has a flat PSD and is comparatively uniform throughout the frequency spectrum. The PSD values for the OTFS, ZFE, MMSE, and ZFE–MMSE algorithms are −1,400, −1,580, −1,780, and −2,100. However, because of the high data rate and power concentration in the signal, the PSD for 256-QAM IM exhibits a notable peak near the carrier frequency. The PSD has a bigger peak because the IM technique includes more signal states 256 in this case, which causes the energy per symbol to be distributed more broadly throughout the spectrum. This higher peak shows the power that is concentrated at the carrier frequency. This makes the signal more vulnerable to frequency-selective fading and needs equalization steps to fix the spectral distortions.

Figure 5

256-QAM Rayleigh channel PSD estimation.

Figure 6 illustrates the concentrated power spectrum of the power spectral performance of 256-QAM IM with ZFE and MMSE channel equalization in a Rician channel. For the algorithms used by OTFS, ZFE, MMSE, and ZFE–MMSE, the PSD values are −1,500, −1,700, −210, and −2,500. A dominating spectral peak appears at the carrier frequency in the Rician channel due to the presence of a strong LOS component. Through the reduction of interference and mitigation of multipath fading effects, ZFE and MMSE improve the power distribution and increase spectral efficiency. As a result, the PSD exhibits a more concentrated and sharper peak around the carrier frequency, which enhances transmission efficiency, permits greater data rates, and minimizes distortion, making it an excellent choice for reliable communication in Rician channels.

Figure 6

256-QAM Rician channel PSD estimation.

Signal detection in OTFS waveform is complex due to its unique time–frequency processing, combining dimensions simultaneously. The high-dimensional nature, coupled with the intricacies of the delay-Doppler channel model, increases computational demands. Overcoming noise, interference, and channel impairments adds to the complexity, making OTFS signal detection challenging but promising for mitigating multipath fading in wireless communications. The complexity analysis of the suggested and traditional methods is shown in Table 2.

Table 2

Complexity analysis

Method	Complexity	Remarks
ZFE	O ( N 3 )	Simple, low memory, and high complexity for large N amplifies noise
MMSE	O ( N 3 ) to O ( N 4 )	Lower error rate, good at low SNRs, higher complexity, and increased memory needs
Proposed ZFE–MMSE	O ( N 3 ) to O ( N 4 )	Best performance balance, adaptable, most complex, demanding processing, and memory

Thus, ZFE can be chosen for its simplicity and cheap memory requirements; nevertheless, for big N, it has significant complexity and noise amplification. While MMSE has a lower mistake rate, it comes with more complexity and memory requirements. Although ZFE–MMSE requires the most processing power, it offers the finest performance balance. The best option will depend on the particular application.

5 Conclusion

By distributing data symbols throughout time and frequency, OTFS uses a special 2D technique that provides notable benefits in terms of resistance to channel distortions. However, the OTFS detection procedure is difficult and complex. The main difficulty is dealing with the intricate time–frequency dispersion that different delays and Doppler shifts introduce. Because dependable wireless communication is dependent on effective OTFS detection algorithms, OTFS is a potential technology for applications including IoT devices, driverless cars, and 5G networks. The successful implementation of OTFS in real-world circumstances depends on the creation and optimization of these algorithms. One significant advancement in the field of communication systems is the recognition of OTFS signals using the combined methods of hybrid ZFE and MMSE. This innovative approach combines the benefits of both the ZFE and MMSE approaches to address the unique issues brought forth by OTFS IM. As part of the hybrid technique, the ZFE part makes good use of the sparsity of the OTFS channel to get rid of interference between symbols and help the signal recover at first. It makes use of the topology of the OTFS channel to ensure a high degree of detection reliability. On the other hand, the MMSE component enhances the estimations and further increases signal recovery by taking into consideration the statistical properties of the noise and interference. When ZFE and MMSE work together, they make it possible to detect OTFS signals very well, even when communication is difficult because of multiple paths and channels that change over time. This increases the overall performance of communication systems and broadens the range of scenarios in which OTFS IM can be applied, and traditional methods are not feasible. As such, the hybrid ZFE+MMSE technique holds great promise for application in various emerging technologies, such as next-generation wireless communication systems like 5G and beyond, autonomous cars, and massive IoT deployments. Its ability to recognize OTFS signals quickly and precisely makes it possible to transfer data and communicate more effectively in difficult real-world scenarios. It is crucial to remember, though, that additional research and development may be required for the implementation and optimization of the hybrid ZFE+MMSE method in order to maximize its efficacy in certain use situations. Still, this technique represents a significant step toward OTFS IM’s full potential, and it will probably have a significant influence on future developments in wireless communications.

Funding information: The authors state no funding involved.
Author contributions: All authors contributed equally in this paper.
Conflict of interest: All authors do not have any conflict of interest.
Data availability statement: The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.

References

[1] Li M, Liu W, Lei J. A review on orthogonal time–frequency space modulation: state-of-art, hotspots and challenges. Comput Netw. 2023;224(109597):1–28. 10.1016/j.comnet.2023.109597.Search in Google Scholar

[2] Albreem MA, Kumar A, Alsharif MH, Khan I, Choi BJ. Comparative analysis of data detection techniques for 5G massive MIMO systems. Sustainability. 2020;12(21):1–12. 10.1109/ACCESS.2020.3017625.Search in Google Scholar

[3] Tiwari S, Das SS. Circularly pulse‐shaped orthogonal time frequency space modulation. Electron Lett. 2020;56(3):157–60. 10.1049/el.2019.2503.Search in Google Scholar

[4] Kumar A, Albreem MA, Gupta M, Alsharif MH, Kim S. Future 5G network based smart hospitals: hybrid detection technique for latency improvement. IEEE Access. 2020;8:153240–9. 10.1109/ACCESS.2020.3017625.Search in Google Scholar

[5] Ramachandran MK, Surabhi GD, Chockalingam A. OTFS: a new modulation scheme for high-mobility use cases. J Indian Inst Sci. 2020;100:315–36. 10.1007/s41745-020-00167-4.Search in Google Scholar

[6] Raviteja P, Phan KT, Hong Y. Embedded pilot-aided channel estimation for OTFS in delay–Doppler channels. IEEE Trans Veh Technol. 2019;68(5):4906–17. 10.1109/TVT.2019.2906357.Search in Google Scholar

[7] Surabhi GD, Augustine RM, Chockalingam A. Multiple access in the delay-Doppler domain using OTFS modulation. Inf Theory. 2019;1902.03415:1–9. 10.48550/arXiv.1902.03415.Search in Google Scholar

[8] Nimr A, Chafii M, Matthe M, Fettweis G. Extended GFDM framework: OTFS and GFDM comparison. 2018 IEEE Global Communications Conference (GLOBECOM). Abu Dhabi, United Arab Emirates: IEEE; 2018. p. 1–6. 10.1109/GLOCOM.2018.8647704.Search in Google Scholar

[9] Naikoti A, Chockalingam A. Low-complexity delay-Doppler symbol DNN for OTFS signal detection. 2021 IEEE 93rd Vehicular Technology Conference (VTC2021-Spring). Helsinki, Finland: IEEE; 2021. p. 1–6. 10.1109/VTC2021-Spring51267.2021.9448630.Search in Google Scholar

[10] Wu Y, Xiao L, Xie Y, Liu G, Jiang T. Efficient signal detector design for OTFS with index modulation. Digit Commun Netw. 2022. 10.1016/j.dcan.2022.10.026.Search in Google Scholar

[11] Zhang Y, Zhang S, Wang B, Liu Y, Bai W, Shen X. Deep learning-based signal detection for underwater acoustic otfs communication. J Mar Sci Eng. 2022;10(12):1–20. 10.3390/jmse10121920.Search in Google Scholar

[12] Kosasih A, Qu X, Hardjawana W, Yue C, Vucetic B. Bayesian neural network detector for an orthogonal time frequency space modulation. IEEE Wirel Commun Lett. 2022;11(12):2570–4. 10.1109/LWC.2022.3209076.Search in Google Scholar

[13] Chu TMC, Zepernick HJ, Westerhagen A, Höök A, Granbom B. Performance assessment of OTFS modulation in high doppler airborne communication networks. Mobile Netw Appl. 2022;27:1746–56. 10.1007/s11036-022-01928-4.Search in Google Scholar

[14] Ouchikh R, Aïssa‐El‐Bey A, Chonavel T, Djeddou M. Sparse channel estimation algorithms for OTFS system. IET Commun. 2022;16(18):2158–70. 10.1049/cmu2.12469.Search in Google Scholar

[15] Kumar A, Gour N, Sharma H, Shorfuzzaman M, Masud M. Hybrid detection techniques for 5G and B5G M-MIMO system. Alex Eng J. 2023;75:429–37. 10.1016/j.aej.2023.06.005.Search in Google Scholar

[16] Kumar A. Detection in 5G mobile communication system using hybrid technique. Natl Acad Sci Lett. 2021;44(1):39–42. 10.1007/s40009-020-00962-8.Search in Google Scholar

[17] Farzamnia A, Moung E, Hlaing NW, Kong LE, Haldar MK, Fan LC. Analysis of MIMO system through zero forcing and minimum mean square error detection scheme. In 2018 9th IEEE Control and System Graduate Research Colloquium (ICSGRC). Shah Alam, Malaysia: IEEE; 2018. p. 172–6. 10.1109/ICSGRC.2018.8657515.Search in Google Scholar

[18] Kojima T, Yamada T. A novel ZF and MMSE receiver for modified Walsh-Hadamard code division multiplexing in helicopter satellite communications. 2021 International Conference on Advanced Technologies for Communications (ATC); Ho Chi Minh City, Vietnam: IEEE; 2021. p. 18–21. 10.1109/ATC52653.2021.9598285.Search in Google Scholar

Received: 2023-11-09

Accepted: 2024-01-30

Published Online: 2024-04-03

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/eng-2024-0008

Keywords for this article

OTFS; detection method; BER; PSD

Creative Commons

BY 4.0