INTRODUCTION
The first article of the Korean Industrial Accident Compensation Insurance Act states that its purpose is for injured workers to return to society, as well as to compensate workers and facilitate their rehabilitation.
1 Since return-to-original-work (RTOW) is considered the most desirable for injured workers in return to society,
2 the Korean government has set as a policy goal to support individuals returning to original work. Previous studies of return-to-work (RTW) have identified factors related to this outcome, including sex, age, company characteristics, working period, injury type, period for medical care,
3 average wage, disability grade, job group (manual worker or not), company size,
4 education, consultation experience,
5 comorbidities,
6 and socio-economic status.
7 A systematic review
8 concluded that these factors affect RTW after musculoskeletal injury, and another study
9 reported that self-efficacy also plays a role in RTW. Since complex factors have been demonstrated as predictors of RTW, it is difficult to use all of these factors practically.
Previous studies have aimed to identify meaningful factors using several statistical methods, e.g., multivariate logistic regression (MLR) has been widely used as a methodological standard in medicine.
10 However, machine learning techniques are currently in practice in medical sciences especially for problems on prediction. A problem on prediction is a type of classification problem that can be solved with machine learning techniques. Additional machine learning techniques are becoming widely used, some of which have demonstrated better predictive ability on certain classification problems.
1112
This study aims to establish prediction models of RTOW and comparing each model and to apply machine learning techniques to identify factors related to RTOW. By comparing the characteristics of the predictive models, a simple and/or accurate model is identified. This analysis, coupled with a discussion of the advantages and disadvantages of machine learning models, is expected to help establish policy on industrial accidents.
DISCUSSION
This study showed that machine learning techniques can predict the probability of RTOW by four years from the end of the official recovery period. The RF model showed the best performance, but the other models showed almost comparable performance. This is the first study using machine learning techniques to predict RTW. Machine learning techniques have a more complex mathematical basis than classical MLR modeling; however, since they do not assume linearity, machine learning models are more flexible and require fewer variables.
Predictive models with machine learning techniques are anticipated to be adopted more frequently in medicine and social sciences in the future. Prediction of various outcomes by machine learning techniques has already become common in medicine, such as mortality,
12 surgical outcomes,
24 and psychiatric illness.
25 This study used similar methodologies of these previous trials: comparing variables with conventional statistical methods, establishing machine learning models with selecting all the variables or selected variables for efficiency, and hyperparameter tuning process to maximize the predictive accuracy.
When optimizing hyperparameters of RF and SVM models, we used the grid search mechanism which is a traditional method with some weaknesses. The grid search does not guarantee that the results were mathematically the most optimal. Therefore, novel methodologies such as random search, Bayesian optimization, gradient-based optimization, and evolutionary optimization are being developed and applied currently. This study did not used these current methods for hyperparameter tuning because of limited software resources of the R package caret. However, although our methodology could not guarantee a mathematical robustness, the grid search mechanism has been widely used, so that we estimate that our results are not far from the very optimized values. We also expect that this simple method could be applied practically in KCOMWEL.
If our models be applied in practice of KCOMWEL, it would be possible making an individual policy of RTW of an injured worker; for example, providing an early reemployment administration service to a worker who is expected to fail of RTOW. However, there might be considerations other than a simple prediction. In this study, we just showed the possibility of prediction and it is necessary more detailed discussion on availability of the prediction.
Comparing all the models introduced in this study, the RF model showed the best performance (
Table 5) in terms of accuracy significantly (
Supplementary Table 3). However, since this model used all variables in its analysis and its tuning process is automated, and RF sometimes yields unstable results by each trial, its practical applications are limited.
The SVM model showed results that were similar to those of the MLR model, even if they used only four variables in the analyses. This feature of SVM modeling — i.e., better efficiency with fewer variables
11 — has potential practical advantages. A modeling method using SVM is referred to as a ‘black box’ model because of its mathematical complexity
26; thus, the interpretation is limited.
Classical MLR modeling yields ORs of independent variables that affect dependent variables. The theoretical assumption behind this method is that ORs approximate risk ratios if the sample size is sufficient. However, regarding epidemiology, it is still unknown what is the meaning of features selected in SVM and RF. Future efforts should investigate how to interpret factors identified by other machine learning techniques in the context of social epidemiology.
Even if the accuracy is inferior, the results of the DT model seem to represent an acceptable compromise of practice and interpretation, because the mathematical methodology used to discriminate variables that maximize information gain is simpler than that used in other machine learning techniques. In particular, the DT model in this study showed a one-way path that is very easy to implement in practice. If an employee is not a temporary or daily worker, maintains a relationship with the company during the recovery period, has earnings in the upper quartile, and is satisfied with the employer's convenience, the DT model predicts that he/she will return to his/her original work with a sensitivity and specificity of 67% and 80%, respectively. Therefore, the DT model's four questions could be used when assessing how to best support patients from industrial accidents.
The differences between the techniques were not prominent; only 0.025 of difference of accuracy between the superior model (RF) and the inferior one (DT). These results are similar to a comparison study that showed statistically not significant differences of methodologies with naïve Bayes, RF, DT, SVM, and MLR even though the MLR model showed the best performance.
27 It means that there is not so much to expect in choosing the most elaborate method. It is necessary to select an appropriate model by advantages and disadvantages in order to predict a worker's possibility of RTOW. For example, RF model for the best accuracy; DT model for clear and easy-to-use standard; SVM model for a simple composition of necessary variables.
Regarding the characteristics of important predictors for RTOW, all models selected similar variables as important for predicting RTOW, although some differences were observed among the models. Four variables — earned income, maintenance of relation with company, satisfaction level with employer, and working status — were selected in all models. These variables can be classified as ‘variables related to the company itself.’ While nearly all the variables except these important variables were also significantly associated with RTOW in the χ2 analyses, no significant effects were observed for any of the variables in the machine learning models.
This finding means that the strongest component influencing RTOW and its durability is a worker's company. This finding is consistent with a previous study
3 that identified workplace characteristics as significant predictive factors for RTW. Another study from the first PSWCI
14 reported that sex, age, regular recovery assessment, and opinion of duration of medical treatment were related to RTW, similar to the findings of our study. However, the data obtained from the longer follow-up in our study suggest that company-related factors were more influential than individual or supportive factors.
As well as the four variables, formal contract for employment and average working hours per day were selected in the MLR model and age in the RF model. However, the additional variables were not more significant than the four variables in each model while the four variables showed little differences among the models and internal fluctuations. Therefore, although there were some differences due to technical methods, all models indicated that variables related to the company itself would influence RTOW. In this point, staffs and medical practitioners who are working for patients aiming RTW should consider the characteristics of the patients' original companies.
Earned income showed the biggest ORs in the MLR model and had a positive gradient, as all models selected it as an important variable. This feature should be discussed in the context of social justice, e.g., whether workers' compensation insurance is adequate in the social security system. KCOMWEL provides various services and programs for RTW, as mentioned in the Introduction. Of these services and programs, consultation for RTW with medical doctors and work ability assessment referral for RTW were associated with RTW in the χ
2 analysis (
Table 3). However, this effect was not observed in the MLR model or in the others. This finding suggests that these services were closely associated with company-related factors, i.e., the services necessary for RTW were provided to workers who already had a good base for RTW. If the services had similar effects on all workers, related factors should have been detected in the machine learning analyses. This result is strongly supported by the segmented labor market in Korea: a worker's situation before injury influences the quality of employment after RTW.
28
This study has some limitation. It did not use variable weights, because it is unclear how to handle these weights when developing training models. Weights are used to assure representativeness, but it is unknown how these weights should be used to guide training and test datasets. Moreover, machine learning techniques are not typically used with weighted variables. The follow-up rate was 83%, and this study did not consider dropouts. Therefore, the results of our study are not necessarily applicable to all workers nationwide. However, the results of previous studies and the results of the MLR model without weighted variables in this study were similar.
We showed that, by using only four to six variables at the end of a worker's recovery period, the probability of RTOW can be predicted with an accuracy of 76%–78%. Since additional factors that were not included in our study might change the predictive performance of the models, future studies are needed to investigate and predict RTW more precisely.