Ionic surfactants critical micelle concentration modelling in water/organic solvent mixtures using random forest and support vector machine algorithms

2.1 Experimental dataset

The database used in this research was extracted from our previous study. ¹⁸ It consist of total of 258 experimental cases, with 12 input variables and one additional output variable to predict. The input variables are: molar fraction of organic solvent in water (dimensionless), molecular weight of organic solvent (in g mol⁻¹), octanol-water partition coefficient of solvent defined as log P or log K _WO (dimensionless), number of C, H, Br, Cl, N, Na, O and S atoms, and solvent temperature in water when measuring CMC defined as T (in K). The output variable is the logarithm value of CMC (in mol L⁻¹ or M).

According to Soria-López et al. ¹⁸ the organic solvents dissolved in water used to measure the CMC were four alcohols (ethanol, ethylene glycol, isopropanol and methanol) and acetone. The temperature range of the mixture of organic solvent in water oscillated between 298.15 K and 323.15 K. The database includes experimental data of all the input variables mentioned above, as well as the logarithmic value of CMC for a total of 10 ionic surfactants, 4 of which are anionic and 6 cationic. The anionic ones are sodium deoxycholate (SDC), sodium dodecylbenzenesulphonate (SDBS), sodium dodecylsulfate (SDS) and sodium N-lauroylsarcosinate (SDDS), while the cationic ones are benzyldodecyldimethyl ammonium bromide (BDAB), cetylpyridinium chrolide (CPyCl), cetyltrimethylammonium bromide (CTAB), docecylpyridinium chloride (DPC), dodecyltrimethyl ammonium bromide (DTAB) and tetradecyltrimethyl ammonium bromide (TTAB).

2.2 Data division

The previous step to the implementation of the algorithms to create different predictive models of the CMC values is the data division. According to Ishola et al. ²³ the predictive models developed using machine learning have to be validated. This validation is used for the selection of an optimal internal validation model and for the evaluation of its generalized predictive performance or external validation. ^{23 , 24} In this research, the internal validation consists of a training group (T) and a validation group (V), while the test group (Z), in this case, is the external validation. Taking this into account, the database was divided into three large groups randomly. The first group, training (T), constitutes 70 % of the database to develop different models. The second group, validation (V), corresponds to 20 % of the database, and is used to find the best model from all the models developed in the training group. Finally, the third group, testing (Z), represents 10 % of the database and has the function of evaluating the model’s performance with data that has not been used in the training group.

2.3 Machine learning models

As has been previously mentioned, two machine learning based algorithms were used to predict the CMC values of ionic surfactants: Random Forest (RF) and Support Vector Machine (SVM).

2.3.1 Random forest

The Random Forest (RF) is a class of ensemble learning algorithm first announced by Breiman ¹⁹ in 2001. ²⁵ This algorithm has been widely used to deal with classification and regression problems. ²⁶ This algorithm uses bootstrap aggregation, commonly known as bagging, to produce decision trees. ²⁷

According to Iranzad and Liu, ²⁸ two important techniques are integrated in RF: bagging and random node splitting. Bagging consists of repeatedly choosing a random sample with replacement from the training data and fitting trees to these samples. This leads to trees that grow from different samples and are quite distinct from one another. Regarding to random node splitting, this technique allows for the consideration of only a random subset of features in each node split of the tree. This leads to uncorrelated trees by preventing features that are very strong predictors of response from being selected by many trees. ²⁸ Accordingly, this algorithm overcome certain limitations of decision trees such as overfitting problems and their inability to store non-linear and non-balanced data. ^{29 , 30} The final prediction of the RF observation is executed by majority voting in classification problems or by averaging the results of all trees in regression problems ³¹ (Figure 1).

Figure 1:

Set of individual trees that are part of the random forest for regression tasks. Inspired in Andrade Cruz et al. ³⁰

In this research, all RF models generated employed the following three hyper-parameters combinations: number of trees (from 1 to 200 in 199 steps, using a linear scale), maximum depth (from 1 to 200, using a linear scale) and pre-pruning (true and false). It is also worth mentioning that two RF models were normalized to a certain range to avoid any of the variables having a greater influence on other variables causing a disproportionate effect on model training. In this study, two normalization methods were applied in linear scale to both input and output variables: range transformation (denoted by the subscript R) which normalizes the data on a scale between −1 and 1, and Z-transformation (denoted by the subscript Z). The normalization process was first applied to the training and validation data and then to the test data. All results were then de-normalized for comparison with other models. Therefore, in this research, three approaches were developed for the RF models (RF, RF_R and RF_Z).

2.3.2 Support vector machine

Support Vector Machine (SVM) is supervised machine learning algorithm first introduced by Cortes and Vapnik ²⁰ in 1995. ²⁹ This algorithm is based on the principle of structural risk minimization which can lead to an improvement of the generalization capability and a decrease in the upper limit of the generalization error. ³² In addition, according to Gaye, Zhang, and Wulamu, ³³ SVM work well with high-dimensional data. This algorithm is a binary classifier in which the main aim is to find an optimal hyperplane that shows a maximum margin between the feature vectors of all the data of the different classes (Figure 2). ^{34 , 35}

Figure 2:

Support vector machine as a binary classifier. Inspired in Andrade Cruz et al. ³⁰

This algorithm is used for both linear and nonlinear regression and classification problems. ³⁶ SVM uses the kernel technique to solve in the case of nonlinear problems. ³⁷ According to Boualem et al., ³⁸ kernel functions are used to translate the input data into a higher dimensional feature space. The most studied kernel functions are the linear, the polynomial of degree d and the radial basis function (RBF). ³⁸

In this study, all SVM models were generated using the following four hyper-parameters combinations: SVM type, kernel type, gamma and C. The guide of Hsu et al., ³⁹ was used as a reference study to select the values of the gamma and C hyper-parameters. In addition, LibSVM, an SVM library proposed by Chang and Lin ⁴⁰ was used. ⁴¹ The SVM models were generated using SVM type (epsilon-SVR and nu-SVR, both work on regression problems), kernel type (radial basis function, RBF), gamma (values between 9.5·10⁻⁷ and 256 in 28 steps, in linear or logarithm scale) and C (values between 9.8·10⁻⁴ and 1,048,576 in 30 steps, in linear or logarithm scale). The two normalization methods mentioned above were also applied in some SVM models on both in linear and logarithmic scales, described with the subscript L in this case, to both input and output variables: range transformation (between −1 and 1), and Z-transformation. The normalized results were then de-normalized in the same way as already mentioned in the RF models (RF_R and RF_Z). Therefore, six approaches were developed for the SVM models (SVM, SVM_L, SVM_R, SVM_R-L, SVM_Z and SVM_Z-L).

2.4 Metrics

The evaluation of the modelling fit and the prediction performance of the different algorithms were measured using the following three statistical parameters: root mean squared error (RMSE) (Eq. (1)), mean absolute percentage error (MAPE) (Eq. (2)) and the linear squared correlation coefficient (R²) (Eq. (3)). All these parameters were calculated for all phases.

RMSE measures the error between two data. Its values range from 0 to ∞ and closer the value is to 0, the less error there is between two data. MAPE measures the percentage of absolute error. Its values range is also from 0 to ∞, being 0 % error when MAPE is equal to 0. Finally, the range of R² values is from −∞ to 1, with R² = 1 representing a perfect fit. ⁴² Therefore, at lower values of RMSE and MAPE and higher values of R², the predictive model shows a better fit. ⁴³

where y and x represent the actual and predicted values, respectively. Furthermore, y ‾ is the mean of data y and n is the total number of the data.

2.5 Computational resource and software

The different RF and SVM models were developed using the RapidMiner Studio Educational 10.2.000 version (RapidMiner GmbH.). The computational equipment used were an Intel^® Core™ i9-10900 K at 3.70 GHz with 64 GB RAM with Windows 11 Pro. The obtained data (258 experimental cases) were collected using Microsoft Excel 2013 (Microsoft). Figures 1 and 2 were made with Microsoft PowerPoint 2016 (Microsoft). The graphical representations (Figures 3, 5 and 7) and the scatter plots (Figure 4 and 6) were made with SigmaPlot 14.0 (Systat Software Inc.).

3.1 Random forest models

In the case of RF models, three different predictive models have been developed using the RF algorithm. Figure 3 presents the values of statistical parameters of RMSE, while Table 1 shows the values of the statistical parameters for MAPE and R² for each of the predictive models for the training, validation and testing phases.

Figure 3:

Graphical representation of the RMSE values for training, validation and testing for the RF models. RMSE is root mean square error (M).

According to the results, minimal differences were observed in the values of the three statistical parameters across all phases for the three RF models developed. For the RMSE values, the range oscillates between 0.069 M and 0.095 M (Figure 3). In contrast, the MAPE values ranged from 4.6 % to 6.5 %, while the R² values varied between 0.954 and 0.980 (Table 1).

Regarding the statistical performance of the models for the training phase, the RF_Z model demonstrated the lowest RMSE value (0.073 M) corresponding to a MAPE value of 5.4 %. On the other hand, the RF_R model showed the highest RMSE and MAPE value (0.091 M and 6.1 %). In the validation phase, the RF_R model achieved the lowest RMSE value (0.085 M), followed by the RF model (0.089 M) and the RF_Z model (0.095 M), corresponding to MAPE values of 6.2 %, 5.9 % and 6.5 %, respectively (Figure 3 and Table 1, respectively). The R² values were similar in the three models during both training and validation phases (Table 1). Finally, during testing phases, the RF_Z model showed the best performance, with the lowest RMSE and MAPE values (0.069 M and 4.6 %, respectively) and highest R² value (0.970) for the testing phase. The other two models were worse in terms of the three statistical parameters (Figure 3 and Table 1).

According to the previously described, it can be concluded that the three models performed adequately with internal data (training and validation) and showed excellent generalized predictive performances with external data (testing), indicating that these predictive models did not present overfitting problems.

The best RF model was selected to further study its performance in CMC modelling of ionic surfactants. The selection criterion used for selection was the lowest RMSE value during the validation phase. Based on this criterion, the normalized model in the range −1 to 1 (RF_R) was selected, since it showed the lowest RMSE (0.085 M), although it did not show the lowest RMSE value for the testing phase. Therefore, the RF_R model was selected for further performance analysis.

Figure 4 illustrates the dispersion of the real CMC values versus those predicted by the RF_R model for the three phases. The dispersion of the CMC values for the internal data (training and validation cases) is shown in Figure 4A. According to this figure, it can be observed that, in general, the points are close to the red dashed line with the slope of 1. This indicates that the most cases exhibited low fitting errors, i.e., minimal dispersion between the real and predicted values. However, there are some cases that are noticeably far from the straight line, showing significant dispersion errors between real and predicted values. In relation to these special cases, it is worth mentioning three training cases with the highest fitting errors: (−0.50, −1.07), (−0.77, −1.28) and (−0.23, −0.61).

Figure 4:

Scatter plots presenting actual and predicted values of log CMC real (x-axis) and log CMC predicted (y-axis) for the RF_R model in the training and validation phases (A) and testing phase (B). The red dashed line corresponds to the line with slope one.

Regarding the first case (−0.50, −1.07), the model predicted a value of −1.07 M, while the actual value was −0.50 M, (model overestimation). This case showed the highest absolute error (0.572 M), since it is the point that is farthest from the slope line 1, and a significant absolute percentage error value (115.2 %). The second case (−0.77, −1.28) also presented an overestimation of the model (−1.28 M vs −0.77 M). In addition, it showed the second highest value of absolute error (0.511 M) and the third highest value of percentage error value (66.2 %). Finally, the third case (−0.23, −0.61), the model predicted a log CMC value of −0.61 M versus actual value (−0.23 M) leading a model overestimation of the model. Furthermore, this case showed an absolute error of 0.377 M and the highest value of absolute percentage error (160.7 %). In fact, if these three training cases are removed, the values of the three statistical parameters are significantly improved (RMSE = 0.070 M, MAPE = 4.6 %, and R² = 0.980).

Figure 4B illustrates the dispersion of CMC values for the external data in testing phase. According to this figure, it is observed that most of the cases are close to the slope line 1 (red dashed). However, two test cases with the highest error values are highlighted. In this sense, the first case (−1.36, −1.52), in which the model predicted a value of −1.52 M versus −1.36 M resulting in a model overestimation of 11.7 %, had the highest absolute error value (0.159 M). On the other hand, the second case (−0.84, −0.73) presented the highest absolute percentage error value (12.9 %) and the second highest absolute error value (0.108 M).

3.2 Support vector machine models

Regarding SVM models, six different predictive models have been developed using SVM algorithm. Figure 5 presents the values of statistical parameters of RMSE. On the other hand, statistical parameters for MAPE and R² are represented (Table 2).

Figure 5:

Graphical representation of the RMSE values for training, validation and testing for the SVM models. RMSE is root mean square error (M).

Considerable differences were observed in the values of the statistical parameters RMSE (Figure 5) and MAPE (Table 2) for all phases between the SVM models developed using a linear scale and those using a logarithmic scale. In this sense, the SVM_R and SVM_Z models showed the highest values of these two statistical parameters compared to the other SVM models.

Regarding the RMSE values, their range oscillated between 0.011 M and 0.147 M, and between 0.2 % and 9.5 % in the case of MAPE (Figure 5 and Table 2 respectively). In addition, significant variations were observed for R² values between different SVM models (between 0.913 and 0.999).

In terms of the statistical performance of the models for the training phase, the SVM_R model presented the highest RMSE and MAPE values (0.147 M and 9.5 %, respectively), and the lowest R² value (0.913), while the SVM model presented the lowest RMSE and MAPE values (0.011 M and 0.2 %). The fits of the other SVM models (SVM_L, SVM_Z-L and SVM_R-L) were very close to those of the SVM model (Figure 5 and Table 2, respectively). During the validation phase, the SVM_Z-L model demonstrated the lowest RMSE value (0.044 M) followed closely by the SVM_R-L model (0.049 M). Moreover, the R² values for these twoo models were very similar (Table 2). In contrast, the SVM_Z model exhibited the highest value (0.147 M) and a MAPE of 8.8 %, followed by the SVM_R model (with RMSE = 0.144 M and MAPE = 8.7 %). Finally, the SVM_R-L model showed the lowest RMSE and MAPE values (0.040 M and 2.1 %, respectively) for the testing phase. The SVM_Z-L model was not far behind the SVM_R-L model in terms of the three statistical parameters. In contrast, the SVM_R and SVM_Z models showed the poorest performance. These results are illustrated in Figure 5 and Table 2.

According to these results, SVM models using a logarithmic scale achieved better results with the internal data compared to SVM models using a linear scale. Moreover, these predictive models showed the best generalized predictive performances with the external data, indicating that they are more suitable for modelling the logarithmic CMC values.

As mentioned in the previous section, the best SVM model was chosen to further evaluate its performance. Considering the criterion previously described, the model (SVM_Z-L), with a RMSE value of 0.044 M, was selected for the validation phase, although it was not the one with the lowest value for the test phase (Table 2 and Figure 5).

Figure 6 shows the deviation of the actual CMC values from those predicted by the SVM_Z-L model for the three phases. Figure 6A shows for the training and validation cases. According to this figure, the case (−1.31, −1.49) is noteworthy, as it contains high fitting errors and is in a position away from the red dashed line with respect to the other cases. In fact, this case has the highest absolute error (0.172 M). Furthermore, in this point, the model predicted a value of −1.49 M versus −1.31 M (real value), leading to an overestimation of the model (absolute percentage error of 13.1 %).

Figure 6:

Scatter plots presenting actual and predicted values of log CMC real (x-axis) and log CMC predicted (y-axis) for the SVM_Z-L model in the training and validation phases (A) and testing phase (B). The red dashed line corresponds to the line with slope one.

Figure 6B shows that most of the test cases are close to the slope line 1. However, one case (−1.36, −1.55) is worth mentioning, which is very attractive to the human eye and is located in the middle zone of the graph. For this test case, the model predicted a value of −1.55 M versus −1.36 M (overestimation of model). The absolute error and the absolute percentage error were the highest of the all the test cases (0.194 M and 14.3 %, respectively). In fact, when this point is removed, the RMSE and MAPE values drop to 0.034 M and 0.020 M, and R² increases to 0.993.

3.3 Comparison with the ANN model obtained from the previous study

According to the previous sections, the best models in this research are: RF_R for the RF algorithm and SVM_Z-L for the SVM algorithm. In this section, a comparison of these models is carried out with the best ANN model shown in the previous study ¹⁸ (Figure 7).

Figure 7:

Graphical representation of the RMSE values (in M) for training, validation and testing for the three better models. RMSE is root mean square error (M).

According to the results, it is stated that the best models for the three algorithms (RF, SVM and ANN) are the normalized ones. This corroborates that normalization usually provides excellent fits because this method prevents any variable from having a greater effect on others during model training. In fact, the previous study by Soria-Lopez et al. ¹⁸ demonstrated that all the normalized ANN models were the ones that showed the best fits with respect to the non-normalized ANN models, with the lowest RMSE values for the validation phase. However, the models developed in this research did not follow that pattern. Regarding the RF and SVM models, no clear differences, no defined pattern, were observed between the non-normalized and normalized ones. The three RF models developed were similar in terms of fits both for the validation phase and for the rest of the phases, with the RF_Z model showing the lowest RMSE value in the validation phase (Figure 3). As for the SVM models, the following pattern was analysed: linear scale models showed the worst fits for all statistical parameters compared to those developed from a logarithmic scale for all phases. Among the SVM models developed using a logarithmic scale, the SVMZ-L model showed the lowest RMSE value in the validation (Figure 5).

On the other hand, it can be observed that the RF_R model showed the worst fits for all phases in the RSME (Figure 7), MAPE and R² values (Table 3). On the other hand, ANN_Z model presented a lower RMSE value for the validation phase (0.040 M), followed very closely by the SVM_Z-L model (0.044 M). Moreover, the MAPE and R² values were practically similar. During the training phase, the SVM_Z-L model presented slightly better fits than the ANN_Z model. Furthermore, this model showed better fits for the external data (testing phase) with RMSE and MAPE values of 0.051 M and 2.6 % versus 0.069 M and 3.5 % for ANN_Z model (Table 3).

According to these results, it can be concluded that machine learning-based models are suitable for modelling the logarithmic CMC values of ionic surfactants. This research demonstrates that other machine learning approaches, such as the Random Forest and the Support Vector Machine, are promising tools to replace traditional laboratory measurements. These algorithms were successfully applied in this research. In this sense, RF and SVM together with ANN are three approaches that work adequately for this type of problems.

Other studies can be found in the literature that have used machine learning-based models to model the logarithmic CMC values. A complete comparison of the best models obtained in our study with other previous models in the literature is important to know the scope of our models. However, this comparison is difficult due to differences in data sets, hyper-parameter usage, and surfactant types used to predict in the models found in the literature. The study by Boukelkal et al. ⁴⁴ showed that nonlinear machine learning-based methods (ANN, RFR, and SVR) provided better fits for the prediction of CMC of different surfactant classes. Among those models, the SVR model for the global phase was the best with an RMSE value of 0.205 M and an R² of 0.974 for a total of 593 experimental cases. ⁴⁴ Our best SVM model (RMSE = 0.031 M and R² = 0.996) for a total of 258 experimental cases presented better statistical values (Figure 7 and Table 3). Another study by Rahal, Hadidi, and Hamadache ⁴⁵ used regression and ANN methods to predict the CMCs of 50 anionic surfactants. The results showed that the ANN model with a 4-3-1 architecture presented the best results with an R² of around 0.940 for the training phase. ⁴⁵ Our best ANN for this phase showed a higher R² value (0.997) (Table 3). Finally, Chen and colleagues ¹² used tree-based ensemble algorithms including RF to predict the CMCs of 779 experimental cases of different surfactant classes. The results showed that the RF model exhibited an RMSE, MAPE, and R² of 0.200 M, 23.93 %, and 0.972, respectively, for the training phase, while 0.325 M, 10.32 %, and 0.927, respectively, for the testing phase. ¹² In our study, the RF model showed better fits (Figure 7 and Table 3).

Based on these results, it can be stated that our models outperform all previously published models in all statistical metrics available for comparison, as they exhibit a lower root mean square error and a higher correlation coefficient. However, further research is still needed on these machine learning approaches in this area. New parameter combinations, data splitting, new normalization methods, among others, need to be studied to assess whether the models can be further improved.