Development of a digital employee rating evaluation system (DERES) based on machine learning algorithms and 360-degree method

1 Introduction

Effective employee motivation is a significant concern for companies and enterprises today. A key element in achieving this is the implementation of rating systems, which identify employee characteristics and aim to improve team efficiency.

The rating system is a system for identifying the characteristics of employees, aimed at improving the efficiency of the team.

The term “ratіng” can be іnterpreted as a measure that descrіbes an entіty and allows іt to assess the probabіlіty of advantage or domіnance over other entіtіes іn a competіtіve envіronment. One can say that a person or a group of people has a hіgh ratіng when the supremacy of a gіven subject(s) (partіcіpant(s) іn the ratіng process) іs hіgher than іts competіtors (other partіcіpants іn the ratіng). Today, the phenomenon of ratіng can be observed іn varіous fіelds: economіcs, fіnance, polіtіcs, cultural lіfe, sports, etc. [1].

By enhancing rating systems, organizations can achieve a significant increase in staff motivation, leading to improved performance and enhanced teamwork [2].

Consider the following situations where it is essential to monitor the employees:

1. The appearance of staff turnover. If a company experiences a high turnover rate, it is essential to monitor employee turnover closely. High turnover can indicate underlying issues such as dissatisfaction, lack of engagement, or poor work environment. By monitoring turnover, organizations can identify patterns, address root causes, and take necessary measures to improve motivation and retention.

2. Lack of performance evaluation system in the company. The lack of a robust performance evaluation system underscores the need to address shortcomings in the existing evaluation processes in order to foster employee motivation. Without an effective evaluation system in place, organizations face several challenges that can hinder employee engagement and also the overall performance.

3. Making managerial decisions in personnel policy. Rating systems provide valuable data and insights that enable managers to make effective personnel policy decisions. By leveraging the information obtained from these systems, managers can assess performance, identify talent, drive development, provide feedback, and create a positive work environment. This ultimately contributes to organizational success, employee motivation, and the achievement of strategic objectives.

4. Increasing the level of conflict in the team. Rating systems enable the identification of performance discrepancies, promote objective feedback, address biases, and provide a basis for conflict resolution within teams. By leveraging the insights provided by these systems, managers can proactively mitigate conflicts, promote a positive team dynamic, and create an environment conducive to collaboration and productivity.

5. Identification of employee complaints about working conditions. Monitoring employee feedback and concerns regarding working conditions is vital for fostering employee engagement, retention, and satisfaction. It helps ensure a safe and healthy work environment, enhances productivity and performance, builds a positive organizational reputation, and enables continuous improvement. By actively listening to employees and taking action on their feedback, organizations can create a supportive and conducive workplace that benefits both employees and the overall success of the organization.;

6. Lack of motivation among employees. Rating systems help address motivational challenges by providing clarity, recognition, growth opportunities, feedback, fairness, and accountability. By incorporating these elements into the rating process, organizations can inspire and motivate employees, fostering a positive work environment and driving individual and collective performance.

7. Expansion or renewal of the staff of company managers. By utilizing rating systems, organizations can make data-driven decisions during company expansion or restructuring phases. These systems provide valuable insights into employee performance, skills, potential, team dynamics, and progress, enabling decision-makers to align resources, identify talent, address skill gaps, and ensure a successful transition during periods of organizational change.

8. Changes in the personnel of the company. Monitoring and evaluating employees during personnel changes or transitions play a vital role in maintaining performance, aligning roles, managing team dynamics, supporting change, succession planning, enhancing employee engagement, and upholding performance standards. It enables organizations to navigate personnel changes effectively, minimize disruptions, and ensure a smooth transition while maximizing employee productivity and job satisfaction.

9. Strengthening the position of the company due to the personnel reserve. Rating systems play a vital role in identifying and developing potential future leaders within an organization. By providing objective assessment, aiding in talent identification and succession planning, offering development opportunities, facilitating performance feedback, providing success metrics, and supporting leadership pipeline management, rating systems enable organizations to identify, nurture, and groom individuals with leadership potential, ensuring a strong leadership bench for the future [3].

These situational factors highlight the need for effective rating systems to address these challenges.

The assessment of personnel according to the traditional rating systems are carried out as follows: for each completed task, the specialist receives a certain, pre-specified number of points. After specific period, for instance, a month, the staff rating is formed based on the scored points. At the end of the month, the points are summed up, and the enterprise receives the productivity information of employees.

This approach has the following drawbacks: paper based, does not provide full rating analysis, does not make suggestions or predictions, assesses only task management skills. Managers should be able to easily identify the current and projected competencies of their employees. Thus, it is important to provide enterprises with digital tools for evaluating employees based on well organized assessment model and artificial intelligence for data analysis.

The assessment model was based on known 360-degree method with additional features built by authors. 360 degree is a method in which employees receive anonymous, confidential feedback from their coworkers [4]. This method provides employees with multi-perspective and anonymous feedback, enriching their understanding of their performance and facilitating their professional development. By considering a broader range of perspectives, this approach enhances the sense-making apparatus within organizations, promoting a culture of continuous learning, collaboration, and improvement.

For data analysis, authors use regression machine learning. Employing regression machine learning algorithms in data analysis for predicting professional growth or decline offers the advantage of data-driven decision-making. It provides organizations with valuable insights into employee development needs, helps identify high-potential individuals, and supports effective talent management strategies. By leveraging advanced analytical techniques, organizations can enhance the sense-making process by making more accurate predictions, optimizing resource allocation, and fostering a culture of data-driven decision-making.

This research aims to bridge the gap between traditional paper-based rating systems and the need for more comprehensive, digital tools that provide a thorough analysis of employee ratings, offer suggestions and predictions, and assess a wider range of competencies beyond task management skills. The identified research gap lies in the absence of an assessment model that incorporates a 360-degree feedback approach while utilizing regression machine learning algorithms for data analysis.

By combining the 360-degree method, which provides multi-perspective and anonymous feedback from coworkers, with regression machine learning algorithms for data analysis, the research provides the following benefits:

Improved evaluation model: The research introduces an assessment model that goes beyond traditional rating systems by incorporating 360-degree feedback. This model enables employees to receive comprehensive feedback from multiple perspectives, enhancing their understanding of performance and supporting their professional development.

Enhanced decision making: By employing regression machine learning algorithms for data analysis, the research facilitates data-driven decision-making in organizations. This approach provides valuable insights into employee development needs, helps identify high-potential individuals, and enables effective talent management strategies.

Filling the research gap: The research addresses the existing gap in the literature by combining the 360-degree method and regression machine learning, which had not been extensively explored previously. It provides a novel approach to employee evaluation and contributes to the understanding of how advanced analytical techniques can be applied to improve rating systems.

Practical applications: The research’s focus on digital tools and artificial intelligence for evaluating employees highlights the practical implications of incorporating technology into performance assessment. It offers organizations a framework for utilizing technology to enhance the accuracy, efficiency, and objectivity of employee evaluations.

2 Literature review

2.1 Employee evaluation using 360-degree method

Performance evaluation has traditionally been restricted to a feedback process only between employees and managers [5]. However, as the emphasis has switched to teamwork, development of employees, and service quality, the focus of human-resource management has been shifted to the employee evaluation from the whole circle of sources illustrated in Figure 1.

Figure 1

360-degree feedback sources.

The circle of feedback sources in Figure 1 consists of supervisors, colleagues, subordinates, customers, and one’s self. There are no legal or regulatory restrictions on utilizing a range of rating sources other than the employee’s supervisor to evaluate performance. According to research works [5,6], evaluation methods that include various rating sources produce more accurate, dependable, and reputable data. As a result, the Office of Personnel Departments in the United States promotes the use of various rating sources as an effective approach of evaluating performance for formal assessment and other evaluative and developmental objectives.

In a observational study [6] conducted between September 2017 and May 2018, the results showed that the mean scores for each of the items in the 360-degree evaluation were high, with the highest grades being observed in the evaluations made by peers and patients (a mean of 9.1 out of 10.0). On average, the 360-degree evaluation method yielded grades 0.067% higher than the previous evaluation method (p ≤ 0.001). Students and teaching staff encountered difficulties in the evaluations made by users/families and other members of the healthcare team (nursing assistants and physicians), although they rated the overall proposal as being very powerful in terms of educational value. Thus, the research concluded the 360-degree evaluation method as being an innovative, motivating, and integrating approach to the acquisition of competencies with a focus on excellence [6].

In the paper focused on leadearship [7], this process of 360-degree feedback is proved to provide a more valid assessment of the extent to which a leader is charismatic, intelligent, inspirational, and considerate [7].

3 Research methods

After conducting a comprehensive review of existing methodologies, we have successfully developed advanced models dedicated to the evaluation and prediction of employee ratings. This section provides a detailed description of the workings of these models. By analyzing the strengths and limitations of existing approaches, we have designed the models to tackle the complexities and challenges associated with employee rating assessment. The following subsections will explain the underlying mechanisms and processes.

3.2 Employee rating prediction model

As a part of DERES system, research on developing prediction model of employee’s future ratings using machine learning algorithms was carried out.

3.2.1 Dataset preperation

For the training of the developing prediction model, the research uses simulated and real datasets of employees rating information.

As a real dataset, the research uses “IBM HR Analytics Employee Attrition & Performance” – open database created by IBM data scientists. It contains general and performance information of IBM employees. The original dataset consists of 35 features describing 1,470 employees. However, for the research purpose, the number of features was reduced to 7 (Figure 2). This dataset was used to create simulated data of employee ratings.

Figure 2

“IBM HR Analytics Employee Attrition & Performance” - dataset description.

The simulated data of employees have competences described in Table 1 as their features (Figure 3). The data rows are created using the IBM dataset. The target (predicted) value is “Overall Rating.” As the mean values of the properties are close to 2.5, it suggests that the data in the dataset are evenly distributed, indicating that it is balanced.

Figure 3

Simulated dataset properties with their description.

3.2.2 Key hyperparameters

Hyperparameters are values or settings that control the behavior of the learning algorithm and influence how the model is trained and its performance. Properly tuning hyper parameters is crucial for achieving optimal model performance.

3.2.2.1 Hyperparameters of SVR

SVR uses different kernel functions to transform the input data into a higher-dimensional space. The kernel hyperparameter defines the type of kernel to be used, such as linear, polynomial, radial basis function (RBF), or sigmoid. The choice of the kernel significantly impacts the model’s ability to capture complex relationships in the data. In this research, as a kernel we used RBF, as the data exhibit complex relationships that cannot be adequately captured by linear models.

The “C” hyperparameter controls the trade-off between maximizing the margin (distance between the support vectors and the decision boundary) and minimizing the training error. A smaller C allows more violations of the margin, potentially leading to a wider margin but allowing some training errors. A larger C penalizes violations more strongly, leading to a tighter margin and potentially lower training error. Its value was set to 0.5 to balance the advantages and disadvantages.

3.2.2.2 Hyperparameters of RFR

The hyperparameter “n_estimators” specifies the number of decision trees to be used in the RF. Increasing the number of estimators can lead to a more robust and accurate model, but it also increases the computational cost. The value of 200 displayed the best metrics result.

3.2.3 Evaluation metrics

In order to compare and select the best-suited algorithm for the model, the following regression evaluation metrics were used:

1. MSE was used to calculate the average squared difference between observed and expected values for every prediction [16];

2. Mean Absolute Error (MAE) was used to identify the average absolute error (the difference between the measured and real values);

3. R ² (R-Squared) was used to measure the proportion of variation explained by an independent variable or variables in a regression model for a dependent variable.

3.2.4 Results and discussion

The machine learning methods used for developing rating prediction model were trained with the simulated dataset of employees and tested for performance. The results of these tests can be observed in Table 3.

Table 3

Evaluation metrics results

Metric	SVR	RFR
MSE (%)	0.32	0.27
MAE (%)	4.40	2.47
R 2	0.98	0.98

As seen in Table 3, both algorithms showed approximately the same results of perdicting the target value on the testing dataset. However, MAE score of the RFR algorithm was about two times lesser than SVR’s. Thus, RFR algorithm was chosen to be used in the employee rating prediction model.

4 Architecture and technologies

The architecture of the information system for rating analyses has been developed, consisting of a web application running on the Laravel platform, employee prediction API and database (Figure 4). Laravel is a full-stack web-development tool that uses Model–view–controller (MVC) software design pattern [17].

Figure 4

Architecture of the information system module for employee rating analysis.

Laravel 8 continues the improvements made in Laravel 7.x by introducing Laravel Jetstream, model factory classes, migration squashing, job batching, improved rate limiting, queue improvements, dynamic blade components, tailwind pagination views, time testing helpers, improvements to artisan serve, event listener improvements, and a variety of other bug fixes and usability improvements.

As a web-server the system uses localhost. MySQL was used as a Database Management System (DBMS). MySQL is an open-source relational database management system (RDBMS). A relational database organizes data into one or more data tables in which data types may be related to each other, these relations help structure the data [17,18].

Security of data was handled by Laravel. Laravel automatically generates a CSRF “token” for each active user session managed by the application. This token is used to verify that the authenticated user is the one actually making the requests to the application.

Employee rating prediction API, run by python web-server, is used to predict employee future ratings using employee rating prediction model described in the research [19,20,21,22]. It receives employee’s current month ratings for each competence from the main web-server, inputs the data to the employee rating prediction model, sends back to server output of the prediction model as a result.

The Employee Evaluation Service (EESC) is made available subsequent to the organization’s representative completing the registration process within the system (refer to Figure 5). Once the registration is completed, the enterprise will be granted access to the comprehensive EESC. It is worth noting that an organization may comprise an arbitrary number of departments, with each department being endowed with the capability to conduct employee evaluation surveys.

Figure 5

Description of employee evaluation service.

Moreover, it is essential to emphasize that every employee is exclusively affiliated with a single department, thereby ensuring a distinct and unambiguous evaluation process for each individual employee. Consequently, evaluations are conducted separately for each employee, enabling a focused and targeted assessment of their performance.

Employee evaluation survey is an assessment tool, which is used by DERES to evaluate employees. The department representative can build employee evaluation survey manually, thus providing flexibility. By default, EEDRW is chosen.

The Employee Evaluation Process (EEP) entails a structured workflow designed for conducting comprehensive employee-by-employee evaluation surveys, as visually represented in Figure 6. Within the EEP, evaluators diligently assess each individual employee, and the outcomes of these assessments are meticulously recorded in a dedicated database. This database serves as a valuable repository of evaluation data, which subsequently becomes the foundation for generating a comprehensive report. The ultimate output of the EEP is the employee’s rating report, which provides a detailed and comprehensive account of the evaluation results pertaining to a specific user (employee) throughout their entire tenure within the organization [23,24]. The report encapsulates a comprehensive overview of the employee’s performance over the entire duration of their employment, offering a valuable insight into their strengths, weaknesses, and overall contribution to the organization.

Figure 6

Employee evaluation process.

5 Research results

For the implementation of the DERES system and its further analysis, a local Kazakhstani business-company – “Mebius Group” – used the developed system (DERES) from January to September of 2021. The company consists of two departments: IT-support and web-development departments. During the testing period, seven employees were assessed and evaluation results were calculated using the DERES system (Figure 7).

Figure 7

Organization employees’ rating results for 2021.

The graph in Figure 7 shows the results of organization departments’ average ratings for each month. During strict lockdowns, due to the pandemic situation in Kazakhstan, the organization was forced to switch to online working (from March to the end of July). In these months, employees’ ratings significantly declined. The minimum values for web-development department were calculated in July (3.2 points), while IT-support department obtained its minimum rating in May (2.45 points).

Starting from August, ratings started increasing. Thus, IT-support department reached its peak in September with 4.4 points. Whereas, rating of 3.87 was calculated for web-development at the end of September. The researchers assume that the improvement in departments’ ratings is due to two main reasons.

1. Return to offline work. A study published in the Academy of Management Discoveries journal found that offline work facilitated the development of social connections, trust, and shared understanding among team members. These social interactions were associated with higher levels of cooperation, coordination, and overall team performance [25]. This indicates that DERES assists managers in making informed decisions about which department should have the option to work online, considering their past performance and demonstrated results in online work;

2. By increasing work motivation through competition between employees of different departments. This is facilitated by the transparency and constant availability of the ratings: any employee in the system can view the ratings of his/her colleagues or employees of other departments. Such transparency and objectivity in assessing the work of company employees using DERES showed an increase in the productivity of team.

The DERES provides constantly updated short-term information about the employee’s rating, including his/her own evaluations and the evaluations of other employees on assessment questions (Figure 8). It shows which competencies should be improved by the employee as well as the suitability of the employee for the position or salary the subject currently has. The graph in Figure 8 showed that the assessed user’s weak points are adaptability and working-online.

Figure 8

An example of an employee’s EEDRW rating scores.

The next advantage of the developed system is that it can identify if an employee who has overestimated himself. From Figure 8, one can observe that the assessed employee gave more points to characteristics such as adaptability, working online, relationships, creativity compared to his/her teammates’ opinions. However, the assessed user underestimated his/her task management and personal development skills.

In Figure 9, the report illustrates the ratings of the selected employee by each month of the year with the predictions made by employee ratings prediction API for the upcoming months.

Figure 9

Employee’s ratings by month (for the year 2021) with predicted ratings.

Employee’s predicted ratings are dependant on the features’ weights described in Figure 10. From the list, one can observe that high personal development and excellent relationship with colleagues increase the chance of obtaining higher ratings in the near future.

Figure 10

Rating prediction model’s feature importance list.

6 Discussion of the results

Comparing DERES with traditional employee evaluation showed that DERES allows employers to evaluate employees using specialized software developed by the authors. The main advantage of DERES over traditional evaluation systems is that it reduces time and cost for evaluation process (Table 4).

Table 4 describes time differences spent on employee evaluation activities in traditional and digital DERES versions of employee rating evaluation systems. The values are estimated by the authors.

For example, in a company of 22 employees, using Table 4, each employee spends 60 + 30 + 30 = 120 min for evaluation process in traditional evaluation systems [26,27]. The overall time spent for one manager to complete evaluation of all employees with detailed analysis is 120 multiplied by 22 is 2,640 min or 44 h. Multiplying that by the manager’s hourly pay (10 USD on average) an estimated cost to the organization will be 440 USD per month. Whereas with DERES, it will cost 12 times less in terms of money and time.

7 Conclusіon

According to the authors, conducting a rating analysis of employees is one of the ways to increase the productivity of enterprises, this is also shown by a detailed literary analysis of works devoted to ways to increase the efficiency of an enterprise. This article proposes the DERES developed by the authors, the main advantages of which are: constant availability to employees of the enterprise for rating analysis, transparency of results, the possibility of a comparative analysis of various departments, efficiency, flexibility, and forecasting future employee ratings. Based on the developed DERES, a rating analysis of a real business company was carried out with recommendations for improving the efficiency of employees. The architecture of the information system has been developed. In the course of the study, the following methods were applied: the 360-degree method and the RFR method. When developing a rating prediction model, the analysis of regression machine learning algorithms showed that RFR showed slightly better results compared to the support vector regressor. The results of the developed rating forecasting model identified personal development and relationship skills as the most important skills in the production team. The results of a practical rating analysis conducted for a real company showed how remote work affected the work opportunities of verified employees.

Research and analysis made it possible to establish which employee competencies require more attention on the part of the employee, as well as the employer in order to improve the efficiency of the enterprise.

In general, with the help of the developed information system DERES, an employer on an ongoing basis can identify and conduct a rating analysis of his employees, predict their future working capacity, and identify and eliminate the causes of a decrease in the working capacity of their employees in order to increase the productivity of the enterprise [28]. The results of the research are used to evaluate the employees of an enterprise company “Mebius Group.” The obtained estimates allow analyzing the professional activities of employees on an ongoing basis and thereby improving the efficiency of the enterprise [29].