Best low-cost methods for real-time detection of the eye and gaze tracking

2.1 Eye landmark detection

The quality of the basic face area and facial landmark detection influences how accurate facial recognition algorithms are. The face recognition method was used by Cheng et al. ( 17 ) to extract face areas, extract single-eye regions, and carry out very accurate position estimates. Retina-Face which predicts face bounding boxes using extra-supervised as well as self-supervised multitask learning was suggested by Dent et al. ( 18 ). Occlusion is a problem in face detection, the Occlusion-aware Face Detector (AOFD) was developed by Chen et al. ( 19 ), which uses an adversarial training technique to identify faces with few visible facial landmarks. In other approaches. Putro et al. ( 20 ) employed a face area scaled to 128 × 128 pixels preceding a bounding-box estimate of eyesight, while some, such as Ahmed ( 21 ) and Leo et al. ( 22 ) utilized statistically facial-landmark data to crop off individual eyes preceding a real-time eye segmentation technique.

Regarding face-landmark-detection algorithms, Wang et al. ( 23 ) extracted essential points from face-bounding boxes that reflect facial landmarks. Early landmark detection algorithms relied heavily on statistical approaches to fit a deformable face mesh.

Eye appearance-based models rely on the photometric appearance of the eyes. Using a Semi-Circular Edge shape and Semi-Ellipse Edge Shape characteristics, a completely automated eye localization approach based on the geometric form of the iris and eyelid was suggested in ( 24 ). Manir and Rabul used also these characteristics to combine the (histogram of oriented gradients, Hit-or-miss transform, and Local binary pattern) features allowing for the detection of eye candidates in face pictures, which improves the process overall and improves performance during the verification step by extracting slope edges ( 25 ). Xia et al. ( 26 ) presented hybrid regression and isophote curvature methods for eye center localization, but their model struggles with specular reflection or closed eyes. Abbasi and Khosravi ( 27 ) proposed a particle filter-based pupil detection technique, but their model performs worse in real-world settings. Choi et al. ( 28 ) introduced a convolutional neural network-based eye pupil localization technique, while Liu et al. ( 29 ) suggested a weight binarization cascade convolution neural network for eye localization.

Naseem et al. ( 30 ) suggested Faster RCNN, and a rectangular-intensity-gradient technique for face, eye detection, openness identification, and eye center localization, with suggested AlexNet assisting in detecting eye state (opened or closed). On the other side, Ahmad and Laskar ( 2 ) used support vector machines (SVMs), and convolutional neural networks (CNNs)for iris-shape feature extraction to propose an accurate eye center identification and localization model. It used iris shape characteristics from the upper face area to extract rough eye candidates by the CNN to extract high-level features, and SVM for classification.

In-the-nature facial-landmark identification has improved significantly with CNN-based landmark detectors. The approaches ( 31 – 35 ) are commonly assessed about 68 points employing datasets of labeled visible-light pictures.

Many iris-landmark identification techniques ( 36, 37, 38 ) leverage cropped single-eye areas from facial-landmark recognition findings. Choi et al. ( 35 ) presented a segmentation-based eye position estimate by cutting off a rectangle area employing landmarks of the eye socket and eye corner using 68 points while Ablabatski et al. ( 38 ) discovered five iris features in a 64 × 64 single-eye area. A quick eye-identification method that utilizes a Siamese network was created for NIR partial face images ( 39 ). Moreover, Bazarevsky et al. ( 40 ) recognized the left- and right-eye centers using the facial landmark recognition technique with reasonable accuracy.

2.2 Eye-tracking methods

Eye movements are critical in eye tracking and also influence human-computer interaction. The widespread technology to track eyes has broadened its use. Eye-tracking technologies, for example, are increasingly utilized to detect abnormalities linked to a variety of psychological conditions ( 41 ). Maurage et al. ( 42 ) described in detail how they used a gaze-tracking device to identify alcoholism. Data on gaze-tracking technologies in neurological disorders and their potential therapeutic effect on brain function were provided by Bueno et al. ( 43 ). Eye tracking has also been used by Robertson et al. ( 44 ) to suggest a method for mild speech comprehension difficulties in dyslexic children.

Eye tracking is frequently used to improve learning. In order to show how well a C programming course is taught, Sun et al. ( 45 ) used eye-tracking equipment to identify learners’ difficulties, offer them recommendations, and help them improve their self-efficacy for learning the C language. While Molina et al. ( 46 ) presented an empirical analysis combining eye-tracking and student-subjective evaluation.

Using real-time eye tracking, Kerr et al. ( 47 ) presented a corrective method that uses closest neighbor calibration locations to calculate anticipated drift at the user’s attention after analyzing calibration information that has been gathered from the user. Pavlas et al. ( 48 ) provided helpful guidance on how to establish low-cost tracking. They used ITU Gaze Tracker and Eye Writer software in their system. Lee et al. ( 49 ) developed a method for assessing 3D gaze position using light reflections (Purkinje images) on the surface of the cornea and lens while accounting for the 3D optical framework of a human eye system. Borsato et al. ( 50 ) created a stroboscopic catadioptric eye tracking (SCET) software using roller shutter cameras and stroboscopic structured infrared light. Krafka et al. ( 51 ) introduced the Gaze Capture software, which used a similar technique to track eyeballs in restricted spaces, using an infrared LED for labeling and the eye’s inner space for training. In ( 52 ), the authors suggested a computationally efficient approach for extracting iris regions from pictures, with an emphasis on segmentation for iris localization. To determine the iris error-free location, the authors ( 53 ) employed rough entropy, a soft-computing technique, and Circle Sector Analysis (CSA). The approach fared well when contrasted with the most advanced technologies, but it is extremely costly in real-time applications. Park et al. ( 54 ) presented a network-based stacked-hourglass approach of learning for localizing eye area features localization, which they tested on the Columbia Gaze, EYEDIAP, MPIIGaze, and UT Multiview datasets. These models can help human-computer interactions in real-time gaze detection applications. Deep learning is a sophisticated technology that has rapidly evolved and is frequently employed in various gaze estimation applications ( 55 ).

Several methods for estimating gaze based on appearance have been presented. For example, the authors of ( 56 ) presented a multimodal convolutional neural network; however, they first utilized the SURF cascade technique ( 57 ) and the limited local mode architecture ( 58 ) to distinguish faces and landmarks of the face before feeding this network data. The researchers ( 51 , 59 ) utilized the Gaze Capture dataset to train an iTracker based on a CNN network; the network gathers eye pictures and face images with their positions associated in a face grid-like structure. In ( 60 ) face-based two-dimensional and three-dimensional gaze detection was proposed. The researchers used a weight mechanism that takes information from multiple parts of the face and trains the pattern utilizing a standard CNN network. The researchers ( 61 ) employed a recurrent convolution network with distant cameras to consider the person and head posture as a specific issue for 3D gaze prediction. They used a many-to-one recurring model to predict gaze coordinates and fed the eyes region, face, as well as face landmark points into the CNN network as distinct inputs. Manir and Rabul ( 15 ) developed the gaze estimation method, which makes use of the Iris Center and Eye Corner to estimate gaze in low-resolution face pictures. A circular gradient-intensity-based operator is used for a rough estimate, a CNN model is used for the correct Iris Center, and an Explicit Shape Regression approach (ESR) is used for Eye Corner localization. The experimental findings indicate that the suggested technique may be utilized to estimate gaze with more accuracy in both still pictures and movies. In ( 62 ), the authors introduced a Gazemap for gazing calculation, which is a graphical illustration of the eyeball, iris, and pupil at its exact center. The study’s authors generated an intermediary graphic representation rather than returning both eyeball angles, which improves the gaze estimate procedure. A capsule network for analyzing gaze information from an image’s ocular region was provided by ( 63 ). The authors created a pose-aware Gazing-Net architecture for gaze estimation utilizing the concept of capsules, which transforms the intensity of pixels to feature initialization characteristics that aggregate into a greater number of features as the network grows in depth.

Some researchers have used intrusive devices for gaze tracking such as sophisticated electrodes, contact lenses, and head-mounted devices ( 64, 65, 66, 67, 68 ). The main downside of such methods is that they necessitate physical contact with the users and cannot be utilized without their permission. Furthermore, such gadgets are difficult and costly to use.

Although there is a wealth of research on gaze estimation, only a few have examined gaze prediction via unaltered cameras. The study by Xia et al. ( 69 ), for example, proposed a two-phase gaze prediction strategy based on neural network algorithms and logistic regression analysis. Their proposed technique may be employed on a range of mobile devices without the requirement for additional hardware or extensive prior expertise. On the other hand, the suggested gaze tracking system in ( 70 ) used a single on-camera allows for unrestricted head movement, and operates in real-time with high accuracy. It was used to recognize faces, extract the eye area, and calculate the user-screen distance.

3.1 Overview of the different techniques

In this paper, two different techniques are used as explained in the following:

3.1.1 Haar cascade classifier

Several researchers ( 71, 72, 73 ) suggest employing the Haar cascade to determine feature coordinates and detect faces. Viola and Jones ( 74 ) proposed the Haar cascade as a machine-learning method for detecting objects in pictures. Through a drawn rectangle, a trained Haar cascade checks a picture to determine if it contains the required item. The Haar method is speedier because it performs high-speed calculations that depend on the number of pixels within the rectangle feature instead of the value of each pixel in the picture. The item is detected in four stages: Haar-like feature, integral picture, AdaBoost learning, and Cascade Classifier ( 75 , 76 ).

The Haar Cascade Classifier for face detection is built around Haar-like characteristics. Haar features are employed to detect whether or not a feature exists in a given picture. Every characteristic produces just one value, which is computed by summing the pixels beneath the black rectangle.

The Haar-like component is a rectangular property that adds a distinctive hint to a picture to help it swiftly distinguish faces ( 77 ). Figure 1 depicts instances of common Haar-like characteristics ( 75 ).

Figure 1:

Type of feature (a) edge, (b) line, and (c) four-triangle.

The Haar Cascade Classifier has the disadvantage of discovering an additional feature in the picture that is falsely comparable to the eye characteristic ( 6 ).

3.1.2 Facial landmark extraction

There are various techniques for the extraction of facial landmarks; the essential techniques in this study are as follows:

3.1.2.1 Using the dlib facial landmark detector

Dlib library indicated landmark detection technique is an implementation of Kazemi and Sullivan’s Regression Tree Ensemble (ERT) ( 78 ). According to MultiPIE ( 79 ), this approach extracts 68 face landmarks using a simple and rapid algorithm as shown in Figure 2 ( 77 ).

Figure 2:

The map for dlib facial landmarks.

These predicted locations are then revised through a continuous procedure including a cascade of regression coefficients. Regressors make another estimate based on the one before it at each iteration, seeking to decrease the variance of misalignment of the predicted points ( 1 ). To determine the shape of the eye, we chose to use the dlib package in our work. In the initial phase, the facial contour is calculated via the dlib.get frontal face detector () method. Next, we use the dlib program to input facial features for estimating eye form (shape predictor 68 face landmarks.dat) ( 78 ).

3.1.2.2 Using MediaPipe face mesh

The media pipe ( 80 ) is a powerful library for recognizing faces and facial landmarks. Pictures of the eyes are obtained from the library. To extract 478 face landmarks as shown in Figure 3 ( 81 ), MediaPipe Face Mesh employs a residual neural network architecture ( 82 ).

Figure 3:

Media pipe face mesh solution map.

The landmarks used in this paper to detect the face and eyes are as follows in Table 1;

Table 1:

Indices of the landmark.

Parts of facial	Indices of landmarks
Face border	[10, 338, 297, 332, 284, 251, 389, 356, 454, 323, 361, 288, 397, 365, 379, 378, 400, 377, 152, 148, 176, 149, 150, 136, 172, 58, 132, 93, 234,
	127, 162, 21, 54, 103,67, 109]
Left eye	[362, 382, 381, 380, 374, 373, 390, 249, 263, 466, 388, 387, 386, 385,384, 398]
Right eye	[33, 7, 163, 144, 145, 153, 154, 155, 133, 173, 157, 158, 159, 160, 161, 246]

The MediaPipe face nets ( 83 ) are utilized to produce a three-dimensional representation of the face and retrieve three-dimensional nose values to serve as a reference for calculating head position. The X and Y output coordinates of the face mesh solution are normalized according to the frame length. The z vector represents the depth of the face wire mesh, which reflects the head’s distance from the camera.

We use the media pipe command map_face_mesh = mp.solutions.face_mesh to specify face attributes for calculating eye shape ( 78 ).

The procedure begins by reading through each frame of the video and passing every frame to the library to earn facial landmark recognition. Images of both the left and right eyes are saved separately ( 1 ).

In terms of test reliability, we find that results extracted using MediaPipe metrics are better than using dlib metrics; therefore, this paper used MediaPipe metrics for eye detection.

3.1.2.3 EAR feature

The EAR function works by measuring the distance from the eyes’ landmarks. In general, the EAR measure calculates a ratio based on the vertical as well as horizontal distances of six ocular landmark locations, as demonstrated in Figure 4 ( 79 ).

Figure 4:

EAR features.

Beginning with p1 and concluding with p6, the coordinates in question are identified by numbers clockwise starting at the left-eye corner. Rosebrock ( 84 ) claims that each of the six coordinates from p1 to p6 is two-dimensional. When the eyes are open, the EAR value remains essentially constant, according to ( 85 ). When the eyes remain closed, however, the gap between coordinates p3 and p5 and p2 and p6 disappears, and the EAR ratio falls to 0 ( 84 ). In this paper, the EAR function is used to predict whether the eye is closed or not.

To determine the EAR value, the numerator computes the distance that exists across horizontal landmarks, while the denominator computes the distance that exists between both vertical landmarks and multiplies it by two to equalize, as shown in Equation (1) ( 86 ).

(1) EAR = p 2 − p 6 + p 3 − p 5 2 p 1 − p 4

3.2 Datasets

The dataset of the suggested system has eye images of people from different races and geographic areas. The global eye dataset included images of people’s eyes from European countries, whereas the local eye dataset included images of people’s eyes from Asian countries.

Four different types of datasets were employed in this work, including two for training models and another two for results testing.

There are two different datasets used for training as follows:

3.2.1 First type: global eye dataset

The SBVPI (Sclera Blood Vessels, Periocular, and Iris) comprises 1800 images of 55 participants, categorized by gender, eye class, and view/gaze-direction labels. The dataset includes 450 images in each gaze direction, with a maximum resolution of 1700–3000 px. During a single recording session, a camera (DSLR @ Canon 60D) was used to capture RGB-colored images, as shown in the samples in Figure 5 ( 87, 88, 89, 90 ).

Figure 5:

Global dataset.

3.2.2 Second type: local eye dataset

In datasets created by the proposed method, the sole equipment needed was a camera, ideally integrated into the top portion of the screen of the laptop, as demonstrated in Figure 6 ( 6 ).

Figure 6:

The experimental setup.

The assumptions are that a person is seated directly facing the screen and the embedded camera at the highest point of the horizontal axis of their gaze with excellent lighting. This assumption allows for picture capture in real-world settings rather than traditional laboratory settings.

The proposed system can correctly predict the direction of eye gaze by steps of the algorithm following:

1. The computer screen should be placed at the level of the nose of the person sitting in front of it.

2. The head should be straight while sitting and facing toward the webcam and it not move or with slight head movements.

3. The computer should be close to the person at a distance of roughly 35 cm–50  cm away, the proxy distance between the user and the screen can be calculate by Equation (2)

(2) D = ( P U x − P S x ) 2 + ( P U y − P S y ) 2

where D is denoted by proxy distance, (P _Ux , P _Uy ) is denoted by the position of the user, and (P _Sx , P _Sy ) is denoted by the position of the screen computer.

There are used two different techniques to create a Local eye dataset as follows:

3.2.2.1 Haar cascade technique

The study utilized a local collection of human eye images to analyze eye movement in four directions. The images were collected from 15 individuals, with each having a different skin color. The Haar cascade technique was employed to determine the eye region in the images. The method focused on the upward movement of the eye in normal cases, as the iris does not rise significantly, and the downward movement is similar to the closing movement. The study involved 5000 images, captured under a different illumination (Figure 7).

Figure 7:

Haar – local dataset.

Figure 8:

MediaPipe – local dataset.

3.2.2.2 Landmarks technique utilizing the MediaPipe library

The study utilized a local collection of human eye images captured from a webcam to analyze eye movement in four directions captured under indoor illumination and different skin colors. The MediaPipe Face Mesh technique was employed to determine the eye region in the images. The whole dataset is from 15 individuals: 1000 images for each of the five directions of eyes – 1000 images for the left, 1000 images for the right, 1000 images for the center, 1000 images for the top, and 1000 images for the down (Figure 8).

The global dataset has been used in the first experiments of the research. However, in the final research experiments, it was discarded and the focus was only on local datasets, as they are more diverse and realistic.

For the test in real-time, two approaches were used to get test data.

1. The first approach uses the webcam to capture real-time images

2. The second approach uses YouTube to download a video of eye movements. As it is necessary to extract the image of the eye, the videos used are face-focused, and Table 2 shows examples of the specifics of the videos.

Table 2:

Details of the video dataset.

Source	YouTube
Type	Mp4
Gender	Female
Size	8.87 MBs
Resolution	1920 × 1072
Frames per second	30
Duration (s)	20
Total images	500
Included images	453

3.3 Proposed method

The detection of the eye and gaze tracking are critical in the age of interactive technologies. The proposed system model consists of two parts: a built system and real-time run, the proposed system algorithm is shown in Figure 9.

Figure 9:

Proposed system algorithm.

The proposed model consists of two stages built as follows:

3.3.2 Processing

The most frequent method for tracking the direction of the eye’s gaze is to track the eye’s iris. To find the position of the eye’s iris, first, determine the region of the eye.

There are two methods for detecting the eye region:

1. Utilizing landmarks, either by using a library (dlib) or a more developed library (MediaPipe) to more correctly describe the landmarks of the face, this paper used the MediaPipe library.

2. Determining the eye region using the Haar Cascade Classifier technique.

There are two basic approaches to identifying the direction of eye gaze using simply the webcam, as follows:

3.3.2.1 The direct method is the convolution neural networks (CNN) method

CNN is a deep neural network that uses a grid-like arrangement to analyze input. It is made up of three layers: convolutional, pooling, and fully connected (FC). The convolutional layer filters data, whereas the pooling layer samples feature maps. The final layer, the FC, generates the final output using an activation function such as sigmoid or SoftMax ( 93 ).

The architecture of the proposed model (Di-eyeNET) is separated into two blocks, as illustrated in Figure 10, Table 3:

Figure 10:

Architecture of Di-eyeNET.

Table 3:

Summary of the parameters for training of Di-eyeNET.

Parameters	Local
Total images	5000
No. images-train	4000
No. images-test	1000
No. class	5
Total parameters	46,822
Trainable parameters	46,822
Non-trainable parameters	0
No. epoch	100
Batch-size	32
val_loss	0.0183
val_accuracy	0.9959
Early stopping	After 98 epochs val-loss not improved
Optimizer	Adam
Loss function	Categorical cross-entropy

The initial block is made up of two convolutional layers that use 128 large 5 × 5 filters that have a stride of 2 to collect adequate spatial information while reducing the dimension of the output feature maps. The layer is followed by a max-pooling layer that reduces the feature map size by 2 × 2 along each dimension using ReLU activations.

The second block is then used, which consists of one convolutional layer plus a max-pooling layer with two filters. The spatial dimensions associated with the feature maps reduce by half after block 1 and after each max-pooling function, we increase the total number of filters in the convolutional layers.

To ensure that the feature maps have enough representational capacity, the last two blocks’ output is routed to a flattened layer, which is subsequently routed to a 128-D Fully Connected (FC) layer. To limit the number of trainable parameters while retaining great performance, the proposed design avoids a significant number of FC layers. Finally, the FC layer is connected to a single SoftMax layer that defines the identification of four eye directions.

To extract just the necessary characteristics while discarding the others, a two-step technique was used: Block 1 of the first stage comprises two convolutional layers. Since, after the first stage, we made the dropout of nearly half of the variables (characteristics), dropout was used after block 2 to reduce by about a quarter.

The model was run for more than 100 epochs with an Adam optimization strategy, a learning rate of 0.001, a batch size of 16, and an MAE loss function, with an input shape of 64 × 64. Each model’s input image comprised three channels.

The models of CNN employed in the present research were built with Keras, an open-source neural network library written in Python. Additionally, we applied the most effective ResNet50 and VGG16 pre-trained model architectures.

VGG16 is a convolutional neural network model for image recognition. It is unique because it uses only 16 layers with weights rather than a large number of hyper-parameters ( 94 , 95 ).

We used this pseudo code to import and load the VGG16 pre-trained model:

from keras. applications. vgg16 import VGG16

model= VGG16 ( )

where the size image must be (224 × 224).

As part of the skip connection concept, the Resnet50 classic neural network served as the foundation for several computer activities. Using 150 layers, this model enables us to train CNN ( 94 ).

We loaded the ResNet50 pre-trained model with this pseudo code:

keras.models.load_model(’resnet_50.h5’)

where the size image must be (64 × 64).

Several training experiences for Di-eyeNET, ResNet50, and the VGG16 were undertaken. Analyses of the results demonstrate that the suggested model (Di-eyeNET) beats other techniques on certain parameters. The accuracy of Di-eyeNET, ResNet50, and VGG16 were 0.9959, 0.7933, and 1.7965, respectively; the MAEs of Di-eyeNET, ResNet50, and VGG16 were 0.0183, 0.5532, and 0.1416, respectively; during the testing phase, all models have almost comparable loss values; however, VGG16 has greater loss values during the training phase.

The size of the suggested model is decreased to 75 %, and the response time to 80 % when contrasted with the other models.

3.3.2.2 Engineering method

The engineering method is based on calculating distances, where after finding the eye region using Python libraries to determine the landmarks of the face and the eye.

There are many ways to know the position of the iris of the eye, which reflects the direction of the gaze, as the following:

Draw two perpendicular lines (horizontal and vertical) using specific equations (horizontal line, vertical line) on the iris region (blackest region), which is detected using the Hough transform and contour detection. After that, we identify the point of junction of the two straight lines, which represents the location of the iris and the direction of the gaze. The pseudo-code Python to draw two perpendicular lines are shown in Figure 11:

Figure 11:

Draw two perpendicular lines procedure.

Used the Euclidean Distance function to find the point of intersection (ratio) of two perpendicular lines in Python as shown in Figure 12:

Figure 12:

Point of intersection of two perpendicular lines procedure.

Divide the eye region into five regions: three horizontally, in which the blackest region represents the direction of gaze (left, center, right), and two vertically, in which the blackest region represents the direction of gaze (up, down), as shown in Figure 13:

Figure 13:

Divide the eye region procedure.

Used the EAR feature to calculate the proportion of the distance between both landmarks of the eyes to know the closed-eye state, which is shown in Figure 14.

Figure 14:

EAR procedure.

4.1 Experimental setup

The suggested system was created using the Num Py, Open CV, Tensor Flow, Keras, dlib, and MediaPipe libraries in Python 3.9 for training the models.

To create this system, we utilized a laptop with an i7 processor, 16 GB of RAM, and a GeForce GTX graphics card. The development environment utilized was Anaconda. Training and testing were conducted as two separate steps in the system’s adoption.

4.2 System evaluation

Evaluation is an important phase in which the experimental data are utilized to forecast the results and assess the algorithms. Accuracy, loss (MAE), response time, and model size are some of the metrics used to assess the performance of the various models ( 1 ).

1. Accuracy: is the key statistical effectiveness parameter used to evaluate a model. Accuracy is explained in Equation (3).

   ACC = TP + TN TP + FP + TN + FN × 100

   Here, true positives are denoted by TP, true negatives by TN, false positives by FP, and false negatives by FN.

2. Mean Absolute Error (MAE): is a statistical indicator of the differences (loss) between measured (y′) and real (y). Equation (4) shows the formula for calculating MAE.

   MSE = 1 N ∑ i = 1 N Y i − Y i ′

   Here, n is the total number of classes, y′ denotes output measured, and y denotes output actually.

3. Response time: For real-time applications, response time is critical.

4. Models Size (Load Time): When implemented in real-time, size models must load rapidly and easily. It is difficult to load the model instantly when it is too huge. Except for the proposed model, the majority of other models are too huge to be employed in real-time.

4.3 Results and discussion

In this paper, four methods were proposed to track the direction of eye gaze, which are: CNN of Haar, CNN of Mesh, Mesh of Inter. Point, and Mesh of Split.

There are two methods of the first direct type: CNN of Haar and CNN of Mesh which depend on training convolutional neural networks, while Mesh of Inter. Point and Mesh of Split are two types of geometric methods, which depend on calculating the distances between facial landmarks using the Python library (MediaPipe).

Mesh of Inter. Point method uses three functions (a function to determine the iris of the eye, a function to draw the two perpendicular lines, and a function to know and examine the intersection point of the straight lines), whereas the Mesh of Split method uses two functions (a function to divide the eye into five sections and a function to determine the iris of the eye in any part after the division; the darker region represents the direction of gaze).

The results in the following tables were acquired utilizing the proposed approach to enhance the lighting since the results of the Haar cascade method without an enhancement approach were extremely few and did not exceed 50 % in the event of a poorly lit examination room.

The difficulties of doing studies in real-time appear apparent since the results obtained are high, but their accuracy may be reduced by half if standard criteria are not followed.

It is important to note that while the results of the CNN methods were nearly 99 % accurate in the training phase, they were significantly lower in the real-time examination phase due to the challenge of controlling the proper lighting and the subject’s sitting position, as the angles of taking pictures affect the quality of the image entering the network and consequently affect the anticipated result. Therefore, all of these factors were taken into consideration when the results for the following tables were obtained, allowing for a fair comparison of the proposed techniques.

In the CNN of Haar method, the dataset images for training were constructed using the Haar Cascade Classifier approach by drawing a rectangle around the eye and truncating it, with the cut-out eye area defined as integrating the brow and the area around the eye with the eye. Thus, the large number of features trained by the Di-eyeNET method influenced the decision-making stage of the real-time tests, where only three classes were identified (it did not identify the top direction in real-time); although training accuracy was high (99 %) for all classes and test accuracy was 93 % (see Table 4).

The experiments were repeated by increasing the number of images and persons and modifying the training settings, but no satisfactory results were produced owing to the existence of like features impacting the resolution, such as the brow (which constitutes a major portion of the image).

As a result, we used the mesh function to create images for our dataset, which was distinguished by the creation of images that took only the eye because it draws a frame around the eye and then crops the eye only, and there are no additional features, as it only focuses on the features of the iris in the CNN method’s training phase. As a result, the findings of the training and testing phases were highly accurate (99.7 %) for all classes, as shown in Table 5.

The results of the proposed methods vary according to the various assessment measures including accuracy, MAE, model size, and response time. According to the results of the experiments in Table 6, the CNN method outperforms the engineering method. The CNN method is the direct method because it uses a single function, whereas the engineering method uses multiple functions, as well as its accuracy is higher because neural networks play a significant role in determining whether the human eye looks up, forward, or down by measuring the parameters of the top eyelid and the amount of iris movement.

Moreover, the CNN of Mesh method is the most rapid and accurate. As for the time to load the software when running, which depends on the size of the software, note that the time to load optimal method (CNN of Mesh) is relatively longer, but the response time when working with it in real-time is the shortest, which is important because the faster the response, the more practical and acceptable the software at the user.

As a result, the techniques based on CNN produce superior results in the limited mobility range, whereas the geometry methods produce the best results in the wide mobility range.

An analysis of all experiments led to a conclusion, the software performed better when utilizing ready-made videos rather than trying the system in real-time since real-time results are affected by head movement and light, and the person’s distance from the screen, which has a direct effect on the testing results. The individual must be no more than 50 cm from the screen and must not turn their head since the camera is not sufficiently accurate to capture images clearly if they are farther away. Furthermore, to improve the system’s real-time performance, we proposed a strategy to improve the lighting of the webcam. The results in the table show that the accuracy was greatly improved while not affecting the response time, which is one of the most important measures of the system’s success when implemented in real-time.

Despite the use of eye trackers, appearance-based approaches have low accuracy in gaze evaluation. Variable lighting, changes in ocular images, and head posture fluctuations all contribute to this variability, making high accuracy in camera-based gaze detection algorithms difficult ( 70 ).

Table 7 shows that all researchers employed strong constant light and maintained a consistent head position. Table 7 compares the recent works of researchers who use CNN based on specific tunable factors (equipment, accuracy, method, dataset, and function).

It demonstrates that there is now a trend toward employing a standard camera and away from eye-tracking equipment since the emphasis has been on methods based on appearance. The accuracy results in Table 7 also indicate that the method presented for this study is the best.