Evaluation and prediction of SVM parameters in engineering cost based on random forest hybrid optimization

2.1 Project cost forecasting index system

In line with the literature referred to in the literature review, through reading, analysis, and summary, the prediction indicators of construction project cost are identified, and 14 characteristic indicators are extracted from the literature through repeated sorting and checking. It includes construction-related influencing factors (whole flooring area, above-ground floor area, variety of higher floors, underground floor area, variety of underground floors, shape type, quantity of floors, and flooring peak) and macroeconomic factors (consumer price index, gross domestic product, loan market quoted interest rate, newly started housing area, average wage of employees, total population at the end of the year). The frequency statistics of the construction project cost prediction index are listed in Table 1.

To sum up, the final construction project cost prediction index system is divided into two levels, and the indicators at each level are shown in Table 2.

The prediction index of construction project cost is divided into quantitative index and qualitative index. Among them, the total construction area above ground, the number of upper floors, the underground construction area, the number of underground floors, the number of floors, the floors, the consumer price index, the gross domestic product, the quoted interest rate of the loan market, the newly started housing area, the average wage of employees on the job, and the total population at the end of the year are quantitative indicators, and the actual project data value can be input [8]. The structural type is a qualitative index, which is a character variable described by literal language and should be quantified before input into the foresee model. Based on the literature review, the isometric division method is used to determine the representative category composition of building structure types, and the structure types are represented by a scale of 1–4, to discretization and quantification of qualitative indicators. The quantization method is shown in Table 3.

2.2 Data acquisition based on the Roberta model

In 2018, Google proposed a pre-training model BERT, which enables the model to process language tasks quickly by pre-training a large quantity of unlabeled expected data [9]. The BERT model structure is mainly composed of stacked multi-layer bidirectional transformer encoders. The model construction is shown in Figure 1. The input layer of the model contains information such as the syntax and location of speech classes. The construction of the BERT input word vector is given in the lower part of Figure 1 through three layers of embeddings: word vector token, sentence segmentation vector, and position vector. Therefore, when the model represents the meaning of words, it will combine the context information of words, which makes the model represent the words more accurately.

Figure 1

BERT model structure.

The pre-training stage of the BERT mannequin consists of two semi-supervised tasks, specifically the protecting language mannequin and the subsequent sentence prediction task. The mechanism of masking language model is similar to close. The model uses “MASK” to mask some words in the input sentence at will, and then predicts the masked words by the model, thus completing the probabilistic modeling of the model [10]. However, most natural language processing tasks need to understand the relationship between the upper and lower sentences, and the semantic relationship between sentences is learned through the next sentence foresee task in the BERT model. In this task, the model randomly selects two adjacent sentences from the file to construct a positive example, while two non-adjacent sentences form a negative example, and the relationship between sentences is obtained by training the model [11]. The combination of the masking language model and the next sentence foresee task makes the word vector obtained by the BERT model have higher quality, which greatly improves the effect of the subsequent task.

Roberta’s model is an improvement on the pre-training process of the BERT model and the scale of the training corpus set. The essence of this semantic model is an improved BERT model. First of all, in the Roberta model, the masking language model will also randomly cover the tokens of some words in the input statements, unlike the static MASK quality of the BERT model, which will fix the token value of the words during pre-training. Roberta’s model adopts dynamic MASK machine quality during pre-training [12]. The dynamic MASK of the Roberta model replicates multiple corpus data during the pre-training stage of the model and masks some words in each corpus. After that, the corpus set statements and their copy statements are input into the model for training, to obtain corpus Tokens represented by different masks. Second, in the Roberta model, the full sentence method is used to replace the next sentence foresee task of the BERT model. This method changes the input limit of a single sentence of the model so that the language model can read multiple sentences with the length set at one time, which enhances the model’s ability to represent the semantic features of long sentences. This makes the model better able to handle long sequences of language tasks [13]. Finally, the number of corpora per batch in the Roberta model is larger than that in the BERT model pre-training, and the corpus set size of the Roberta model for model training is more than ten times larger than that in BERT model training. Therefore, the Roberta mannequin will take a longer time in pre-training; however, the Roberta pre-training mannequin will enlarge the variety of the corpus and enhance the overall performance and generalization capacity of the language mannequin through gaining knowledge of a wide variety of corpus.

This test used to be performed on the AI Studio platform, and the strolling surroundings used to be GPU: Tesla V100. Video Mem:16GB, CPU: 4core-s.RAM: 32GB.Disk:100GB. The deep knowledge of the mannequin framework adopts the paddle-paddle framework of the home flying pulp team, which is the first typical deep-study framework in China. In addition to the convolution unit layer, pooling unit layer, and different unit layers for constructing neural networks, as nicely as the simple suite of activation features and loss functions, the framework additionally has pre-trained fashions such as Bert and Ernie that can be invoked to meet the wishes of customers [14]. Since the content material of this scan is in the path of herbal language processing, for the motive of finishing the experiment, the libraries are referred to as encompass paddlenlp, paddle, paddle hub, etc. The particular state of affairs is proven in Table 4.

To compare the capability of different pre-trained models on the list item indicator path classification task, the pre-trained models BERT, Roberta, and ERNIE were selected in this study to build the classification model of the list item indicator path. The specific parameter settings of the pre-trained models BERT, Roberta, and ERNIE are shown in Table 5.

Since the weights of the pre-trained model can already extract features well, AdamW and Cross Entropy Loss are chosen as the optimizer and loss capability, respectively, during model fine-tuning, which can enable the training model to converge quickly while preventing parameter overfitting. Considering that the GPU in the experimental environment has only 16G memory and the number of text input characters of the list items exceeds 100, the batch size and max seq length are set to 64 and 128, respectively [15]. Research on modeling reveals that the learning rate and training cycle of the dataset have a major impact on the model’s correctness. The loss function of the model rapidly converges when the learning rate is set to 1E-4 or 1E-5, as the link between the learning rate and the loss function is demonstrated in Figure 2. The model’s loss function either does not converge or converges to a greater value for learning rates of 1E-3, 1E-6, and 1E-7. The model’s learning rate is found to be 1E-4, and its training cycle is 10, taking into account that the greater the learning rate, the shorter the model’s training period.

Figure 2

Relationship between learning rate and loss function.

3.1 Sparse RF

Aiming at high-dimensional or ultra-high-dimensional feature vector spaces, Alipour et al. proposed an RF algorithm in 2001, which used the Bootstrap resampling method to extract multiple sample sets from the original data for decision tree modeling, and then obtained the final result through averaging or voting among multiple decision trees. The essence of this algorithm is to combine several weak decision tree models to form a strong decision tree model [16]. Aiming at the problem of processing the measurement data of large quantity quantitative evaluation index that may appear in the intelligent evaluation of future building engineering design, a sparse RF model based on a random weight network was proposed by taking the random weight network model as the decision tree and combining the RF theory with the sparse representation theory.

From the sample set, Bootstrap resampling (i.e., sampling with put back) selected M* ≤ M samples to form a sample subset for parameter training of the k RWN, and a sample subset cluster was obtained. Taking the KTH RWN decision tree as an example, the SRF algorithm repeatedly randomly selects an attribute from all n-dimensional sample data attributes and selects the optimal attribute branch from it to establish a diagonal element matrix for attribute selection, and the diagonal element randomly selected to the position is 1. Otherwise, the diagonal element is 0 [17].

The RWN decision tree model is constructed for the sample subset so that it meets y k ( j r ) = f k ( x k ( j , ) ) , r = 1 , M ∗ ¯ , k = 1 , K ¯ , then the KTH optimal out-of-network weight of the RWN decision tree model is obtained by resolving the least squares optimization issue described in Eq. (1)

Then, a comprehensive evaluation model of residual can be obtained, which is called the RF model according to the random weight network.

Furthermore, Eq. (2) can be rewritten into the following matrix form:

Since the redundant comprehensive evaluation model shown in Eq. (2) is an integrated model composed of several weak decision trees, overfitting and waste of running time may be caused if the quantity of result trees is improperly selected. According to the research results of Guo et al., not all decision trees play a role, particularly when the quantity of decision trees is large, only a few decision trees play a role, and the rest have almost no impact on the result [18]. Therefore, the concept of sparse coding is introduced into the RF model, and the comprehensive evaluation model of redundancy is sparse. Based on the above assumptions and concerning the work of previous researchers in reference, the LASSO problem shown in the following formula can be solved to calculate sparse voting weights:

3.2 RF and SVM machine fusion prediction

The RF algorithm was chosen because of its excellent performance in processing high-dimensional data, especially in the face of complex nonlinear relationships, which can effectively reduce the risk of overfitting. RF integrates multiple decision trees and uses Bootstrap resampling technology to enhance the stability and prediction ability of the model. This property enables random forests to provide more reliable results in project cost forecasting. SVM is chosen as another important component in fusion models because of its excellent performance in small sample learning and its ability to efficiently handle high-dimensional feature spaces. SVM’s choice of kernel functions, such as Gaussian kernel, enables it to capture complex patterns in the data, which are critical for nonlinear features in engineering cost prediction.

Suppose there is a training sample set [19]

After mapping the sample points into a high-dimensional space, the SVM prediction model creates the following linear regression estimation function:

The following function minimization is the same as the function approximation problem:

By introducing two relaxation variables, the above function could be transformed into the following form:

Using the Lagrange form, the above formula can be shifted to

The goal function must satisfy the following requirements to obtain the minimal value:

Using Karush–Kuhn–Tucker, the dual problem can be obtained

The SVM regression function can be obtained by solving the above problems.

To meet Mercer conditions, the most used Gaussian kernel features are commonly chosen as follows:

According to the relevant principles of RF and SVM mentioned above, the steps for cost estimation by combining the two are as follows:

1. According to the collected data of transmission and transformation project cost and the relevant data of general project cost management published by the State Grid, the data should be standardized and identified according to the attribute characteristics of the project cost, and the engineering data should be standardized and labeled [20].

2. Segment the data, pick out about 80% of the engineering facts as coaching samples, and the last 20% of the engineering records as check samples, and practice the random woodland algorithm for classification learning. Then, take a look at samples that are used to check the getting-to-know effect. After passing the test, the attribute significance of the sum is given by the classifier. The essential elements affecting the price are chosen as the enter variables of the aid vector computer model, and the guide vector laptop mannequin is skilled with education samples and examined with take a look at sets.

3. If the estimated accuracy of the take-a-look-at mannequin is much less than 10%, it is regarded that the estimated mannequin is wonderful and can be used to estimate the value of new projects.

In the fusion model, the expression F(X) representing the fusion model was derived [21].

Construct the objective function as

where Y represents the project’s true cost. The absolute value of the error between the fusion model’s actual value and its anticipated value is represented by the objective function. The fusion model’s prediction impact is better the closer it gets to 0. Consequently, it is anticipated that the objective function – that is, min 1 N ∑ k = 1 N L ( Y k , F ( X k ) ) , where N is the number of samples and corresponds to the fitness value in the particle swarm optimization algorithm – will be reduced.

To confirm the impact of the interpolation method, 108 tasks with whole statistics were screened. In addition to the unit fee variable, 10% of records are randomly lacking in the final variables. After finishing the interpolation, the anticipated facts are in contrast with the true data. The non-stop variable information had been evaluated via the root suggesting rectangular error root mean square error (RMSE) and implying absolute proportion error mean absolute percentage error (MAPE) index. The nearer the cost was to 0, the higher the mannequin impact was. The contrast index of kind variable facts takes accuracy and F1 value, and the accuracy cost degrees from [0,1]. The nearer the price is to 1, the higher the mannequin impact is. F1 price is the harmonic common of accuracy charge and recall charge [22]. Thus, the impact of unbalanced samples is eliminated, and the greater the price of F1, the higher its performance. The techniques of studying curve and grid search had been used to optimize the parameters of RF, and the consequences of vital parameter adjustment have been proven in Table 6.

The model can be widely used to estimate the cost of new residential, commercial, and infrastructure projects, and help project managers to make reasonable budget and resource allocation. By improving forecast accuracy, the risk of project overruns is reduced. Governments and industry organizations can use the model to analyze market trends and formulate corresponding policies to promote sustainable development of the construction industry. For example, simulating the impact of different policies on costs, helps decision-makers optimize resource allocation.

3.3 Prediction model construction

The construction of the project cost foresee model according to the CNN and support vector ensemble model (CNN-SVM) is shown in Figure 3.

Figure 3

General idea of model construction.

A CNN is an improvement of the backpropagation (BP) neural network. Its network structure generally consists of five parts. Besides the input layer for data input and the output layer for model output, it also has its own convolutional layer, pooling layer, and fully connected layer. In the process of network learning, the error BP algorithm is used to constantly update the weights and bias values of node connections in the network [23]. However, compared with the BP neural network, CNN no longer adopts a full connection structure, and only local connections are made between layers. Moreover, combined with weight sharing, a CNN greatly reduces the number of parameters. CNN extracts data features based on convolution operation and pooling operation, which reduces the complexity of data reconstruction and enhances the data features.

In CNNs, the convolution layer and pooling layer appear alternately. In the process of model construction, the appropriate network structure can be selected according to the actual demand. From input to output, each layer of the CNN establishes connections through the neural nodes, transmits information layer by layer, and performs successive convolutional pooling operations to map the original data into a high-dimensional spatial, and then the extracted features are processed by the fully-connected layer and the result is input through the output layer, which ultimately yields the network output [24]. The structure of the contextual neighboring network is illustrated in Figure 4.

Figure 4

CNN structure diagram.

The CNN mainly consists of two processes: feature extraction and classification (prediction). It correctly extracts the aspects of entering facts via the convolutional pooling operation, and then inputs the extracted elements into the single-layer perceptron to obtain the classification (prediction) function. The convolutional neural community has the right function extraction ability; however, it is convenient to fall into nearby most fulfilling or overfitting troubles while using the single-layer perceptron for classification or prediction. Moreover, the venture funding prediction mannequin studied in this work belongs to regression prediction, and the output layer of the convolutional neural community has to pick out regression characteristics instead of standard classification functions. Considering the good performance of support vector machines in nonlinear data prediction, this study uses support vector machines as the output layer function of a CNN to build project investment prediction models [25].

Based on fully analyzing the advantages of contextual neighboring networks and the SVM model, this study proposes a project investment prediction pattern according to the CNN and support vector ensemble model (CNN-SVM). By using the features of local connection and weight sharing of the CNN, the data features of influencing factors of project investment prediction can be extracted effectively based on greatly reducing the calculation and complexity of the model, and input it into the SVM at the top of the pattern to realize the project investment prediction.

The specific steps for constructing a CNN-SVM combined model, that is, feature extraction based on CNN and input SVM for project investment prediction are as follows:

1. Obtain data. Collect and clean data to ensure data quality and integrity. The dataset was divided into a training set (80%) and a test set (20%).

2. Feature selection. Through literature review and expert interview, 14 key characteristic indexes related to project cost were determined. The features are standardized to ensure that the dimensions of different features are consistent.

3. Build a RF model. The optimized parameters were used to construct an RF model. Train the model and use the training set to fit.

4. Construct an SVM model. The SVM model was constructed using the optimized parameters. Train the model and use the training set to fit.

5. Feature extraction and fusion. The trained RF model is used to predict the test set and extract important features [26]. The extracted features are input into the SVM model for further prediction.

6. Fusion model training. A fusion model (CNN-SVM) was constructed by combining the output results of RF and SVM. Using a CNN as a feature extractor, deep features are extracted and then these features are input into the SVM for final prediction.

7. Model evaluation. The fusion model was evaluated using test sets to calculate prediction accuracy (e.g., MAPE, RMSE, etc.). Compared with the RF and SVM models alone, the performance improvement of the fusion model is verified.

8. Model optimization and iteration. According to the evaluation results, the model parameters and structure were further adjusted, and several iterations were carried out. The datasets and models are updated regularly to ensure the validity and accuracy of the models in practical applications.

4.1 Goodness of fit

This time, the data are collected from 775 residential project samples from 2015 to 2022 from the Guanglianda Index network. Guanglianda Index network is a relatively open and complete database in the current domestic project cost database. However, because the project dataset is too large, the efficiency of the manual collection is too low, so data collection is carried out through the program of automatically obtaining web content, and then the engineering features are extracted.

Since the conventional parameters have little influence on the model constructed at this time, the index to measure the branching quality and the splitting strategy are processed with the default value of the algorithm, and the random state is fixed as 0 to ensure the repeatability of the experiment. For pruning parameters, such as the maximum depth of the parameter tree, the values from 1 to 50 are calculated in sequence. The maximum value of 50 is an initialized random value. When the value of a parameter becomes stable after a certain value, you can determine whether the maximum value is reasonable. The results are shown in Figure 5 (Maximum depth). It can be observed that the R ² value reaches a maximum value of 0.3778 when the maximum depth parameter is 15 and stabilizes after 35. Therefore, the best possible value for the maximum depth argument is identified as 15, and the range of values from 1 to 50 is also reasonable.

Figure 5

Plot of decision tree vs goodness-of-fit.

Similarly, through generation selection calculation in the range of 1 to 100, the optimal parameter value of the minimum number of lobular nodes required for splitting can be obtained as 7. The line graph between the minimum number of lobular nodes required for splitting and R ² is shown in Figure 5 (Minimum number of lobular nodes). It can be observed that the value of R ² is prone to be stable when the minimum number of lobular nodes required for splitting is greater than 70, so the iterative range from 1 to 100 is reasonable.

4.2 Project cost forecast analysis

According to the above most efficient parameters, the selection tree prediction mannequin is established, and then the facts of the check set are substituted into the mannequin to reap the expected cost of the model. The overall performance of the three single algorithm fashions and the fusion mannequin proposed in this study in the expected price and the actual cost of the take-a-look-at set is proven in Figure 6. RMSE is sensitive to large errors and can effectively reflect the prediction ability of the model in extreme cases. This is particularly important for project cost forecasting because large errors can lead to significant economic losses. Mean absolute error (MAE) gives equal weight to each prediction error, making it easy to understand how the model performs as a whole. It provides a clear margin of error to help decision-makers assess the reliability of the model. R² is a dimensionless index that facilitates comparison between different models. It intuitively shows how different models perform on the same dataset, helping researchers choose the best model.

Figure 6

Comparison of prediction effects of various algorithms. (a) Decision tree algorithm. (b) Neural network algorithm. (c) SVM support vector machine algorithm. (d) Fusion algorithm.

It can be shown from the figure that the error range between the project cost predicted by the decision tree model and the real project cost is about 300 yuan/m, the error rate is about 12.17%, and the goodness of fit of the model is 0.4516. The decision tree model, BP neurological network modeling, SVM modeling, and fusion model evaluation indexes and prediction results were compiled. As can be shown, the RF-CNN-SVM fusion model has the greatest goodness of fit (R ²) value out of the four models, while the models’ values for MSE, RMSE, MAE, and MAPE are the lowest. It is proved that under the premise set in this study (including the collected data, the method of data preprocessing, the parameters set by the model, etc.), each index of the fusion model is superior to the single decision tree algorithm, BP neurological network modeling, and SVM modeling. Therefore, as one of the algorithms of machine learning, fusion algorithms, such as BP neurological network and SVM support vector machine, can be used to resolve the issue of engineering cost. The R ² of the fusion algorithm is 0.6174, which is the algorithm with the highest goodness of fit among the four models. The MAPE value of the fusion algorithm is 0.0955. In other words, the prediction accuracy of the fusion algorithm reaches 90.45%, which meets the accuracy of the estimation requirements. The RMSE value of the fusion algorithm is 251.23, which can be understood as the difference between the forested value and the real value of the RF modeling, which is ±251.23.

Through an examination of the underlying concepts of the aforementioned algorithms, it becomes evident that both the RF arithmetic and the decision tree arithmetic are part of machine learning tree models, with the RF method including Bootstrap resampling into the decision tree algorithm. The fusion technique suggested in this research is superior to the decision tree arithmetic for creating the engineering cost prediction model because this method significantly increases the algorithm’s anti-noise and fitting capabilities. Engineering cost prediction models have been established with remarkable success using BP neural network and SVM arithmetic, according to literature inquiry and research. This is because these two algorithms can forecast the situation of small samples. The RF model is a probabilistic model, and only on the premise of sufficient sample size can a prediction model with high goodness of fit be generated. This study collected 775 residential building engineering samples, which met the requirements of the fusion model in terms of sample quantity, so the fusion algorithm model also achieved good prediction results.

In practical engineering, data quality is ensured and datasets are regularly updated to reflect market changes and the latest trends in the engineering industry. Periodically iterate and verify the model, evaluate the model performance, adjust the model parameters, and optimize strategies according to the actual prediction results. Establish an interdisciplinary team of data scientists, engineering cost experts, and project managers to ensure that the model is scientific and practical. It emphasizes the integration of technology and management to ensure that technical achievements can be smoothly translated into actual project management and decision support. In the future, with access to more real-time data and the continuous optimization of algorithms, this model will provide more powerful support for the digital transformation of the engineering cost field.

Although the performance of the model on the test set is good, the complexity and uncertainty of the engineering project may lead to the deviation of the predicted results of the model in practical application. Different regions and different types of projects may have different cost structures and influencing factors, and the applicability of the model may be limited.