Predicting Time to Graduation of Open University Students: An Educational Data Mining Study

1 Methods

2 Results

2.2 Model Evaluation

2.2.1 Confusion Matrix

The confusion matrix reflects the current situation in the data set and the number of correct/false predictions from the model. Model performance is calculated based on the number of observations classified correctly and those classified incorrectly. The rows show the actual number of samples in the test set, while the columns represent estimates from the model. In Table 2, TP and TN show the number of observations that are classified correctly. False positive (FP) indicates the number of observations predicted as 1 (positive) when it should be in class 0 (negative). FN indicates the number of observations predicted as 0 (negative) when it should be in class 1 (positive).

Table 4 shows the confusion matrix for the five DM algorithms. In a confusion matrix for each algorithm with dimensions of 2 × 2, the main diagonal denotes the percentage of instances predicted correctly, and elements other than the main diagonal denote the percentage of prediction errors. Table 4 shows that based on the RF algorithm, 65.6% of those whose actual study period was less than or equal to 10 semesters (successful) and 10.7% of those with more than 10 semesters (not successful) were predicted correctly. Based on the NN algorithm, 67.3% of those whose actual study period was less than or equal to 10 semesters (successful) and 8.8% of those with more than ten semesters (unsuccessful) were predicted correctly. In addition, based on the LR algorithm, 68.2% of those whose actual study period was less than or equal to 10 semesters (successful) and 7.8% of those with more than ten semesters (unsuccessful) were predicted correctly. Furthermore, based on the NB algorithm, 41.6% of those whose actual study period was less than or equal to 10 semesters (successful) and 22.2% of those with more than ten semesters (unsuccessful) were predicted correctly. Lastly, based on the kNN algorithm, 64.9% of those whose actual study period was less than or equal to 10 semesters (successful) and 10.8% of those with more than 10 semesters (unsuccessful) were predicted correctly.

Table 4

Confusion matrix for five DM algorithms

Algorithm			Predicted
			Not success	Success	Sum
Random forest (RF)	Actual	Not success	10.7%	15.2%	10,334
		Success	8.5%	65.6%	29,485
		Sum	7,642	32,177	39,819
Neural networks (NN)	Actual	Not success	8.8%	17.1%	10,334
		Success	6.8%	67.3%	29,485
		Sum	6,217	33,602	39,819
Logistic regression (LR)	Actual	Not success	7.8%	18.2%	10,334
		Success	5.9%	68.2%	29,485
		Sum	5,444	34,375	39,819
Naïve Bayes (NB)	Actual	Not success	22.2%	3.7%	10,334
		Success	32.4%	41.6%	29,485
		Sum	2,1758	18,061	39,819
k-Nearest neighbor (kNN)	Actual	Not Success	10.8%	15.2%	10,334
		Success	9.2%	64.9%	29,485
		Sum	7,950	31,869	39,819

2.2.2 ROC

Performance comparison between the proposed models based on the AUC, CA, F1, precision, and recall metrics is presented in Table 5. All proposed models have more than 75% accuracy or are in the high category, except for the NB model. It means a high correlation exists between the predicted and actual data in all proposed models (except NB). Specifically, the RF model achieves the best results compared to other models regarding precision metrics, with percentages of 76.3 and 81.1 on invisible data (test). It can be interpreted that the RF model can predict students who complete their studies effectively with low type I (false positive) prediction errors. On the other hand, the NN model achieves the best results compared to other models in terms of recall metrics, with respective percentages of 84.9 and 90.9 on invisible data (test). It can be interpreted that the NN model is comparatively effective in predicting the possibility of students completing their studies with low type II errors (false negatives). Overall, the RF and NN models have almost the same values (not significantly different) when viewed from the CA, F1, precision, and recall metrics, so it can be said that these two models have the best performance in predicting student study status based on time to graduation compared to other models.

Table 5

AUC, CA, F1, precision, and recall value

Model	AUC	CA	F1	Precision	Recall
RF	0.795	0.763	0.847	0.811	0.886
NN	0.793	0.761	0.849	0.797	0.909
LR	0.779	0.760	0.850	0.790	0.921
NB	0.780	0.638	0.697	0.918	0.562
kNN	0.778	0.757	0.843	0.809	0.880

Note. RF, random forest; NN, neural networks; LR, logistic regression; NB, Naïve Bayes; kNN, k-nearest neighbor.

The AUC–ROC metric was used for further evaluation of the performance of the ML algorithm, especially in the case of imbalanced data. Table 5 shows that the RF and NN models have almost the same AUC values (not significantly different) and are the highest compared to the other models, with respective percentages of 79.5 and 79.3. The same results are also shown in Figure 1. Thus, the RF and NN models have the best performance based on the AUC metric in predicting student study status based on time to graduation compared to other models.

Figure 1

AUC–ROC.

2.3 Model Explanation

2.3.1 Variable Importance

Variable importance is a measure of the extent to which certain variables contribute to the predictions of the dependent variable. Figure 2 shows the variable importance of the best models, namely RF and NN. In the RF model, demographic variables such as gender, marital status, and region seem to have a less significant contribution in explaining student time to graduation. Demographic variables such as age, employment status, and minimum wage as well as academic data such as GPA seem to be the most important variables in explaining student study status based on time to graduation. These four variables have a greater contribution than the other three variables. In the NN model, demographic variables such as gender and region are also variables with a small contribution in explaining the success of student studies based on their time to graduation, coupled with the minimum wage variable which in the RF model is actually the most important variable. Demographic variables such as age and employment or occupational status and academic data variables such as GPA seem to consistently be the most important variables in the NN model in explaining student study status based on time to graduation. In addition to these variables, the marital status variable, which previously had a not-so-large contribution to the RF model, became the most important variable in explaining student time to graduation. Overall, all predictor variables (i.e., demographics and academic data) used in the RF and NN models have a contribution (we found no variables with negative RMSE values) with different magnitudes in explaining student study status in terms of time to graduation.

Figure 2

Variable importance.

2.3.2 Partial Dependence

Partial dependence is used to calculate the dependence of the final prediction on a certain variable. Two sets of variables were examined, i.e., demographic information and academic data. This demographic information is related to the unique attributes of students such as gender, age, marital status, employment, region, and minimum wage. Figure 3 depicts that the RF and NN models predict a high probability of success for students who are female, single, and do not have a permanent job to complete their studies in no more than 10 semesters.

Figure 3

Partial dependence profile for demographics information.

Students from regions with a minimum wage of more than IDR 3,000,000 also demonstrated a high probability of success in completing their studies. However, the RF and NN models have different predictions in terms of age. The RF predicts a high probability of success for students aged 30–50 to complete their studies, while those over 50 have a lower probability of success in completing their studies. It is different from the NN predicting students over 50 years of age have a higher probability of completing their studies in no more than 10 semesters.

The next focus is on academic data relating to prior students’ academic grades as represented by GPA. The dependence plot of the GPA variable is presented in Figure 4. The figure demonstrates that the RF and NN models predict a high probability of success for students who have a GPA of more than 3.0 to complete their studies in no more than 10 semesters.

Figure 4

Partial dependence profile for academic data.

3 Discussion

The current study proposed a new model to predict student study status in terms of time to graduation in developing countries such as Indonesia and in an open higher education institution based on the DM algorithm, taking data on demographics, GPA, and student study duration as data sources. The performance of the RF, NN, LR, NB, and kNN algorithms was examined and compared to predict student time to graduation. In comparing the performance of the algorithms, two parameters are used. The first parameter is the prediction of student study status in terms of time to graduation based on demographic information and prior cumulative GPA. Yağcı (2022) also used the same parameter in predicting student final examination grades. The second parameter is a comparison of ML algorithm performance indicators based on AUC, CA, F1, recall, and precision metrics. This parameter is carried out especially on imbalanced data sets (Biecek & Burzykowski, 2021). Apart from comparing the performance of the ML algorithms, this research also demonstrated important variables that contribute to predicting student time to graduation.

The results of this study indicate that the proposed model achieves a classification accuracy of 60–76%. RF, NN, LR, and kNN are algorithms with a very high level of classification accuracy that can be used to predict student study status based on time to graduation, while NB has a low classification accuracy. The results of this study are in line with the results of previous studies (Musso et al., 2020; Yağcı, 2022) which showed that the RF, NN, LR, NB, and kNN algorithms have very high classification accuracy in predicting student academic performance and study status. Similar results were also found by Alturki et al. (2022) that LR, RF, kNN, and NB have very high accuracy in predicting the academic performance of doctoral students. In addition, Waheed et al. (2020) also found that the performance of the LR, NN, and RF algorithms have the highest accuracy in identifying students at high risk of academic failure based on their demographic characteristics.

One of the important findings from this study is that the RF and NN algorithms have the highest classification accuracy and AUC compared to the other three proposed algorithms (such as LR, kNN, and NB). Based on these findings, it can be said that the RF and NN algorithms have more accurate results in predicting student study status using ML algorithms. The results of this research are in line with the results of previous research which also showed the performance of the RF and NN algorithms as the best compared to the LR, kNN, and NB algorithms in predicting student final exam grades using only three academic data attributes (i.e., midterm exam grades, department data, and faculty data), even the results of the two are very similar (Yağcı, 2022). Sassirekha and Vijayalakshmi (2022) also demonstrated that the performance of the RF algorithm was the best with an accuracy of up to 90% in predicting student academic progress compared to the LR, kNN, and NB algorithms. Xu et al. (2019) in their study found that the highest accuracy obtained through the NN algorithm compared to RF in predicting the academic performance of undergraduate students, with accuracy that is not much different.

In addition to finding a model that can accurately predict the time to graduation of students, the results of this study also identified the variables that provide the greatest increase in model performance. These variables were identified using the RF and NN algorithms which demonstrated the best performance. Based on these results, it can be said that demographic variables (i.e., gender, age, marital status, employment, region, and minimum wage) and academic variables (i.e., GPA) are important predictors of student study status. The results of this research are in line with the results of previous studies which show that demographic information such as wage, age, occupation, place of residence (Costa-Mendes, Oliveira, Castelli, & Cruz-Jesus, 2021; Cruz-Jesus et al., 2020), and prior academic achievements (Hoffait & Schyns, 2017) are important variables that can be used to predict academic achievement scores and student study status. In addition, the results of this study also succeeded in identifying the most important variable in predicting student study status, namely prior academic achievement (i.e., GPA). This result is in line with the findings of Yağcı (2022) and Hannaford et al. (2021), showing that prior student academic achievement has the most important contribution in predicting student study status based on their time to graduation. Furthermore, Suhaimi et al. (2019) have also noted that many studies have demonstrated GPA as an important predictor of students’ time to graduation. This is possible because when students get a low GPA, especially in the early days of their studies, they have to repeat the course they got a low grade in the following semester especially when the course is a prerequisite, and this will certainly prolong their graduation time.

Our study has reported that the most important predictor of time to graduation for students at the Open University is GPA. In addition, taking into account various factors that may affect the time to graduation, one of which is age, our study revealed that age is also considered an important predictor. Age is one of the factors that should be considered as a predictor of students’ graduation time given that the context of our current study is an Open University that embraces the educational paradigm of “lifelong learning.” By adhering to this educational paradigm, on the one hand, anyone can become an Open University student regardless of their age as long as they have a high school diploma or its equivalent, and this certainly provides a wide opportunity for everyone to be able to pursue higher education. On the other hand, the wide age range of students may make it challenging to provide relevant learning facilities and support to each student to encourage them to complete their studies in a reasonable time. The wide age range of students at the Open University also has consequences for the high diversity of students’ employment status or study orientation. At the Open University, relatively young students generally expect to complete their studies as quickly as possible so that they can apply for jobs with their diplomas. Meanwhile, older students tend to have relatively stable jobs, so their studies are more focused on gaining knowledge and motivating their children to emulate them in terms of their enthusiasm for learning, and thus, their studies are less focused on an orientation to graduate as quickly as possible. The findings of a study by Sánchez-Gelabert, Valente, and Duart (2020) demonstrate similarly that when it comes to the online university context, older students tend to focus their studies on opportunities to improve theoretical knowledge or the acquisition of new knowledge and concepts. Meanwhile, younger students tend to focus their studies more on improving their chances of getting a job, developing their careers, and increasing practical knowledge that is highly beneficial to their work (Sánchez-Gelabert et al., 2020).

In addition to the age factor, our study has reported that based on RF and NN algorithms, minimum wage and employment status, respectively, become important factors after GPA in predicting students’ graduation time in Open University. It has been pointed out that those with a higher minimum wage and no job, who can focus more on their studies, tend to have a higher chance of completing their studies within a reasonable time. Meanwhile, those with a lower minimum wage or who are working (so-called working students), so studies may not be their main focus, will tend to take a longer time to graduate because they will be more likely to take fewer courses or credits each semester. At the Open University, students are facilitated to take a few courses or credits each semester, even taking just one course is allowed and facilitated. The study conducted by Ecton, Heinrich, and Carruthers (2023) showed that working students tend to take fewer courses or credits which results in a lower percentage chance of graduating in a reasonable time compared to non-working students. Even though working students take a longer time to graduate, their performance in terms of GPA still shows similar results or is not much different from non-working students who can complete their studies faster or in a reasonable time (Ecton et al., 2023).

Overall, the RF and NN models, as the best ones proposed in this study, are able to predict student study status based on time to graduation with an accuracy of 76%. Accordingly, it can be said that the student’s time to graduation can be predicted with these models in the future. Predictions on a student’s future study status in terms of time to graduation provide an opportunity for students to evaluate the work methods they use in order to improve their academic performance (Yağcı, 2022). A similar finding was also demonstrated by Bernacki, Chavez, and Uesbeck (2020), where their early warning prediction model based on malleable behavior managed to correctly identify 75% of students who were not successful in a course or could not fulfill the prerequisites for more advanced courses. In addition, Burgos et al. (2018), through a prediction model that they built via a tutoring action plan, managed to reduce the student academic failure rate by 14% from the previous year. Furthermore, the most important variables in predicting academic achievement that have been identified can be useful for policymakers and other stakeholders to identify appropriate and most cost-effective interventions (Wang et al., 2023). Since age and cumulative GPA are the most important variables related to the teaching and learning process in predicting student graduation time, improving the quality of student learning assistance through weekly tutorials, learning resources or modules, and open educational resources/massive open online courses tailored to the diverse needs and characteristics of students such as age and occupational is an important thing to do by open higher education institutions. Quality improvements in these areas are expected to increase students’ GPA and cumulative GPA by increasing their engagement in available learning services and their motivation to learn and subsequently provide equal opportunities for every student to complete their studies or degree in a reasonable time regardless of their age, occupation, or region of residence.

4 Conclusion

This study contributes to current developments that utilize demographic information and GPA recorded in university databases to predict student study status in terms of student time to graduation in developing countries such as Indonesia. In terms of methodology, this study contributes to providing additional insight into predicting students’ time to graduation by utilizing ML approaches or algorithms such as RF, NN, logistic regression (LR), NB, and kNN. In terms of obtained results, this study contributes to complementing previous literature, which focused more on comparing the accuracy of ML models and did not highlight critical factors based on demographics information and student GPA, which could later be used as an intervention basis to treat students who have the potential to fail to complete their studies within a reasonable time.

The results of this study indicate that the RF and NN models have the highest classification accuracy and AUC values (CA: 76% and AUC: 79%) compared to other models, such as LR, NB, and kNN, in predicting students’ study status based on their time to graduation. The NB model is the lowest, with a classification accuracy of 64% and an AUC of 78%. The RF and NN models were able to identify one of the most critical variables in predicting students’ time to graduation along with six other important variables, namely, the student’s GPA. In conclusion, the RF and NN models proposed in this research can be used to predict students’ time to graduation with high accuracy, and policymakers and other stakeholders can use the most critical variables identified as intervention materials to deal with students who have the potential to fail to complete their studies within a reasonable time in the future. Furthermore, given that this study only focused on five algorithms (i.e., RM, NN, LR, NB, and kNN) in constructing the model to predict students’ time to graduation, while a number of studies have demonstrated that other algorithms such as xgboost (Aiken et al., 2020) have very promising performance these days, we suggest future studies to consider such algorithm to be used in predicting students’ time to graduation and compare their performance with the five algorithms. In addition, while our study used the default value available in the “caret” package for the k value in the kNN algorithm, we expect future studies to use k values other than the default from the package to predict students’ time to graduation.