Optimization of the road bump and pothole detection technology using convolutional neural network

3.1 BGV-based convolutional neural network-highway anomaly detection (BCNN-HAD)

The proposed BCNN-HAD method takes as input images captured nearby highway bridges. The system uses these images to recognize and extract features associated with road anomalies like bumps and potholes. The BCNN-HAD method, based on BGV-YOLOv5, enhances automated image processing and improves the identification of road abnormalities. The system’s CNN is designed to extract significant characteristics from these images, helping to detect and categorize road surface irregularities accurately. Although input handling is more involved than other architectures, the goal is to preserve the rapid processing rate of the CNN network by reducing the overall number of network layers. Unlike many studies, the suggested approach effectively identifies speed bumps and potholes on road surfaces in highway bridges, as shown in the flowchart of Figure 2.

Figure 2

Overall flow of proposed BCNN-HAD work.

3.2 BGV-YOLOv5-based system

3.2.3 Features extraction with CNN

The essential aspect of BGV-YOLOv5 focuses on utilizing CNN. CNN captures significant attributes from the processed images, highlighting traits important for identifying road irregularities like bumps and potholes. Figure 3 shows the proposed workflow of the system.

Figure 3

BridgeGuardVision system.

3.3 Conventional neural network architecture

Initially, the camera captures a color image with dimensions of a × b × c (where a = 672 columns and b = 376 rows, along with c = 3 indicating the RGB color index). Every pixel in an image contains an 8-bit resolution, and the color illustration adheres to the RGB paradigm.

In equation (1) mentioned before, I denote a matrix with three dimensions. The initial layer, the “ImageInputLayer,” uses an information n normalization process. This requires subtracting the average image (mean) of the simulated set, in which the highest value of n is 714, from every image that is input, as specified in equations (2) and (3). The entered images are colored, with an elevation of 376 pixels, an expanse of 672 pixels, and three color channels saved in the RGB color space. These photos are sorted and stored within a tensor with a dimension of (376, 672, 3).

The following layer, known as the “Convolution2Dlayer,” has convolutional processing, including 37 filters with a specific width and height of 3 × 3 ( f 1 , f 2 , … , f x = 37 ) . The layer has a stride of 1 and does not include pooling. However, it incorporates padding of the “same” kind to capture image features and avoid sub-sampling surrounding information. The layer’s data are stored in a tensor with dimensions (376, 672, 3, 37).

From equations (4)–(6), f x is the filtering window with dimensions ( 2 g + 1 ) × ( 2 h + 1 ) and d x and d y are the indicators indicating move across the window on both the u and v axes. Figure 4 shows the all-functional layers of the CNN architecture and their functions.

Figure 4

Various layers of CNN architecture and their functions.

The third layer, “batchNormalizationLayer,” normalizes a group of data inside a small batch, treating each feature individually. This helps speed up the development of the CNN and makes it less sensitive to how the network is set up. Multiple normalization layers should be incorporated between convolutional regions and anomalies for best results. Here, we employ a ReLU layer after a convolutional layer stored in a (376, 672, 3, 37) matrix.

In equations (7) and (8), where μ x represents the average, α x represents the amplification factor and β x represents the offset for each feature.

The fourth layer, known as the “ReluLayer,” consists of a linear unit that has been rectified (ReLU) and applies the thresholding work to the input. Numbers below zero are set to zero, while numbers above zero stay the same. When evaluating ReLU to the tanh and the sigmoid functions, ReLU does not have saturation zones. These saturation areas indicate that the outcomes of the neurons tend to remain relatively stable, resulting in a limited range of deviations from the initial conditioning gradient and little changes to network parameters. This results in a need for more development in the instruction process. Equation (9) demonstrates that the ReLU function is significant in contemporary neural networks, which are stored in a tensor of dimensions (376, 672, 3, 37).

The fifth layer, referred to as “fullyConnectedLayer,” defined by equation (10), creates a neural network that is fully connected with a result size of 3. This is because it is separated into three categories ( i = 1 , 2 , 3 ) . The output value generated by the ReLU layer is converted into a series of integers using the vec() method. This implies that a matrix is transformed into a vector by merging its components. This procedure is often referred to as leveling. This level resizes the inputs by a set of weights K i and then supplies a bias to the vector h i . The outcome is subsequently sent through a feedback loop of transit work. This indicates that our completely linked neural network comprises 28,046,592 components and yields three outcomes (28, 046, 592).

The sixth level (three inputs and three results), “Soft Max,” likewise employs a SoftMax function, which is commonly utilized as the last level of classifiers utilizing artificial neural networks, as shown in equation (11). It changes a vector of L exact numbers into an additional vector of L real values that add up to 1, allowing them to be understood as probabilities. For this situation, the value of L is 3.

The last layer, called “classOutput” with three inputs and thre outcomes, functions as a categorization layer. The formula (12) computes cross-entropy loss for segmenting the outputs into separate categories (3,3).

where X represents the sample count, Y represents the class numbers, q i represents the weight of class I, and represents the signal that the nth study is part of the ith class and one represents the output for sample n for class I, which is the value from the softmax function.

The suggested BCNN-HAD technique using BGV-YOLOv5, including the “BGV” approach, is its capability to offer specialized computer vision technology designed explicitly for highway bridges. BGV improves image processing by extracting reliable features from images taken near bridges, raising road anomaly identification accuracy. This integration guarantees that the model can easily adjust and accurately perform under different environmental conditions typically seen in highway bridge situations. Based on BGV-YOLOv5, the suggested solution utilizes a complete process for identifying highway anomalies. The procedure starts by gathering and preparing photos taken near highway bridges. Then, a CNN extracts features. Sophisticated picture processing and training with different datasets guarantee flexibility under other environmental circumstances. The system includes a sensor fusion module for a comprehensive understanding. The use of BGV-YOLOv5 in anomaly detection helps improve bridge status monitoring. The suggested technique standardizes image input by deducting the mean image from the dataset during preprocessing. This guarantees consistent characteristics and enhanced quality. The architecture’s Rectified Linear Unit (ReLU) layer is essential, as it introduces non-linearity by converting negative values to zero. This aids in learning features, avoiding saturation areas, and improving the efficiency of the CNN by enabling a more comprehensive range of training gradients.

3.4 Network hyperparameter proposal

In CNNs, hyperparameters are important in the design and include aspects like the input picture dimensions, the number of convolutional methods used, the size of convolutional filter systems, and the count of outputs of the neural network. These conditions are set up before the training process. Conversely, network variables in the suggested system have values dynamically chosen by the optimization technique, as indicated by (BGV-YOLOv5). Efficient training of CNNs depends greatly on computer technology, especially when using GPUs for iterative procedures. Increasing the network’s size alone might not result in improved performance or feature learning, even though optimizing the network architecture, adjusting hyperparameters, and starting with the right database are crucial. When categories are easily identifiable, unnecessarily complicated designs may not be needed. The learning of filters for various categories can be enhanced using a larger input size during training despite the potential for higher computing costs. Given that prediction is the network’s primary objective, it is important to note that utilizing high-quality hardware is a prerequisite for achieving optimal performance.

The BCNN-HAD approach uses a neural network structure with important layers, including Convolution2D, batchNormalization, and fully connected. The Convolution2D layer uses 37 filters with a 3 × 3 window, without pooling but with padding. The batchNormalization layer standardizes data within tiny batches, improving CNN development. The fully connected layer constructs a neural network that is fully linked and produces three outcomes using a rectified linear unit (ReLU) activation function. Combining these layers helps enhance the anomaly detection capabilities of the BCNN-HAD approach. Network parameters are essential for CNN performance in the suggested approach. These settings, such as the input image’s dimensions, filters’ sizes, and the number of outcomes, are established before training. Appropriately selected hyperparameters are crucial for optimal feature learning and computational efficiency during training. The thoughtful use of hyperparameters contributes to the overall success and effectiveness of the suggested CNN-based anomaly detection method.

4.1 Dataset

A total of 174 images of yellow roadway cones and 35-speed bumps that were not seen were gathered and stored in folder 0. Folder 1 had 231 pictures of potholes, while folder 2 had 304 pictures of smooth streets. The pictures were taken with a resolution of 376 × 672 pixels. The sun’s position was a significant consideration in the selection of the photos. For instance, it was sometimes hard to get excellent pictures in the bright afternoon light at about six o’clock. As a result, several photos and images that didn’t have any bumps or imperfections were not included to ensure a fair representation in all categories. The collection includes three categories of images: ones that show speed bumps, potholes, and ones that don’t have either. In addition, the photos were captured from the outside of a vehicle. The BCNN-HAD uses a simplified seven-layer structure, less resource-intensive on the computer’s hardware than Inception V2, which has 12 layers. Results using the test dataset of speed bump images and the trained CNN model [19].

4.2 Performance metrics

4.2.4 F1-score

The F1-Score, a mathematical mean of recall and accuracy, offers a fair evaluation of BCNN-HAD’s performance. This measurement considers both incorrect positive and negative results, thoroughly assessing the model’s precision in detecting road irregularities. A high F1-Score implies a balanced trade-off between precision and recall, displaying the model’s effectiveness in precise anomaly identification while minimizing errors. To summarise, accuracy ensures overall correctness, precision highlights the accuracy of optimistic predictions, recall analyses the ability to identify anomalies thoroughly, and the F1-Score evaluates BCNN-HAD’s performance. Together, these measurements help achieve the system’s objective of establishing a more secure and long-lasting transportation network in highway bridge engineering.

(15) Recall = TP TP + FN ,

(16) F 1 = 2 · precision · Recall Precision + Recall .

Equations (13)–(16) offer a numerical representation of the primary assessment criteria. Figures 5 and 6 represent the analysis of various performance metrics of the proposed BCNN-HAD. Regarding highway bridge engineering and BCNN-HAD, these metrics assist in evaluating the model’s capacity to effectively detect and handle irregularities on the pavement, which helps to maintain the durability and safety of traffic infrastructure.

Figure 5

Evaluation of accuracy and precision based on the BCNN-HAD method.

Figure 6

Evaluation of accuracy and precision based on the BCNN-HAD method.

4.3 Training and validation time

“Training and validation time” is a measurement that quantifies the duration required to train and validate the CNN model using the suggested BGV-YOLOv5 algorithm. This measure is essential for comprehending the computational effectiveness and pace of the training procedure, which is vital for the actual execution and utilization of the model.

Training Time is the time needed to train a CNN model using the training dataset. It is an important measurement, and shorter training durations are often favored for quicker model growth. Nevertheless, finding a middle ground between minimizing training time and guaranteeing that the model attains both high accuracy and robustness is essential. Validation Time is the duration required to assess the model’s effectiveness, which was trained upon a distinct validation dataset. Just like the time spent on training, it is preferable to have shorter validation durations since they enable faster iterations in improving the model. Verification is crucial for assessing the model’s ability to generalize unfamiliar data.

Training and Validation Combined Time refers to the whole duration of both the training and validation processes. This measurement offers a thorough perspective on the time spent designing and assessing the model. It helps calculate the total computational resources used for both the training and validation stages, providing information about the efficiency of the model creation process. Equation (17) shows the total training time taken by the epoch ( T i ) as,

Here, T i is the total time at epoch I, a is the slope of the line, indicating the rate at which time rises per epoch, i is the epoch number, and b is the y-intercept, reflecting the initial time at epoch 1. Figure 7 shows the training and validation evaluation curve compared with other object detection methods.

Figure 7

Evaluation of training and validation time for BCNN-HAD.

Extended training and validation durations may suggest a model requiring significant computational resources or more advanced hardware. Reduced durations indicate effectiveness in the model structure, optimization methods, or parallel processing skills. To summarise, “training and validation time” is a practical measure that affects the practicality and scalability of implementing the suggested BGV-YOLOv5 algorithm in real-life scenarios. Reducing this time while ensuring high-quality outcomes is essential for successful deployment.

The proposed BCNN-HAD method and BGV-YOLOv5 are essential in transportation infrastructure and road safety. By effectively detecting road anomalies, they enhance infrastructure maintenance and road safety in highway bridge engineering. This comprehensive approach aligns to establish a safer and longer-lasting transportation network. The study assesses the efficiency of the BCNN-HAD technique utilizing parameters such as mean average precision (mAP) and processing speed. mAP evaluates the precision of object detection by taking into account precision-recall curves. The processing speed measures how fast the system detects abnormalities in real-time situations. These measurements offer a thorough assessment of the accuracy and effectiveness of the suggested approach in identifying irregularities on highway bridges.

4.4 Effectiveness of sensor fusion techniques

Sensor fusion combines information from several sensors to improve the overall comprehension and precision of the surroundings. When identifying road bumps and potholes in bridge engineering, combining information from several sensors is important to improve the system’s capabilities. Within highway bridge engineering, sensor fusion techniques in the suggested system are crucial for improving the identification of road anomalies. Combining data from several sources, such as bridge structural health monitoring systems, cameras, accelerometers, and GPS, is essential in understanding the bridge environment. Evaluation criteria include the correlation and synchronization of sensor data, managing redundancy, real-time processing capabilities, improving accuracy through fusion, adaptability to environmental changes, influence on maintenance decision-making, and scalability to handle larger data volumes. Effective sensor fusion guarantees that the system adjusts to various situations, offers practical insights for maintenance choices, and stays efficient as the monitoring infrastructure grows. A thorough assessment of these factors enables engineers to improve the sensor fusion algorithms, guaranteeing the dependability and efficiency of road anomaly detection in practical highway bridge scenarios.

Sensor integration is essential in BGV-YOLOv5 for detecting highway anomalies by merging data from systems that monitor structural health, cameras, accelerometers, and GPS. This inclusive method improves the comprehension of the area around the bridge, which helps detect road irregularities more effectively. Combining many sources ensures a comprehensive viewpoint for designing intelligent transportation systems in highway bridge engineering.

4.5 Evaluation of different suggestions of current technology

The suggested BGV-YOLOv5-based system for identifying road irregularities is compared to various object identification models designed to detect irregularities on roadways. This evaluation entails analyzing the advantages and disadvantages of the BGV-YOLOv5 system in terms of accuracy, completeness, and overall effectiveness. Table 1 shows the comparative analysis of the proposed BCAA-HAD system with other proposals. Our advanced neural network utilizes a modest seven layers, which is less demanding on the processing hardware than Inception V2, which has 12 levels. Our approach attains an accuracy of 99.13%, as indicated in Table 2.

Table 1

Results using the test dataset of speed bump images and the trained CNN model

S. no.	Input image	Preprocessing	Segmented image	Output image	Output description
1					Image of a speed bump taken during the day. With great precision, our model can identify the speed bump
2					With the help of brightness and contrast adjustments, the collected and enhanced speed bump picture is accurately forecasted
3					Orientation is changed by augmenting the captured picture of the speed bump. Using our algorithm, about 70% of the pixels may be accurately classified as speed bumps
4					Showing an image of an unlabeled speed bump. Even though there are no unlabeled speed bumps in our training picture dataset, our model still manages to segment around 25% of the pixels accurately
5					Accurate segmentation is not achieved when the speed bump picture is shot from a vertical perspective. Twenty percent or so of the pixels are incorrectly labeled as speed bumps even though they belong to a different class

Unlike other works with the same approach, this study chooses not to use strides or combined techniques to gather all the accessible information. In addition, BCNN-HAD includes a “BatchNormalization” layer by layer to speed up the training process and reduce the impact of network initialization, a feature not present in other systems. Furthermore, the proposed method uses a “SoftMax” layer that enables the interpretation of the fully linked network’s output as probabilities.

The research evaluates machine learning models for identifying objects, specifically focused on YOLO series detectors such as YOLOX, for jobs requiring real-time processing. BGV-YOLOv5, using CNNs, demonstrates practicality and efficiency in detecting road irregularities, improving road safety in highway bridge engineering, and contributing to intelligent transportation systems.