PDF
Citation
Print
INTRODUCTION
Since the outbreak of coronavirus disease 2019 (COVID-19) in Wuhan, China in December 2019, it was rapidly followed by worldwide outbreaks.123 As of March 22, 2021, the World Health Organization (WHO) reported a total of 123,877,740 COVID-19 cases globally, with an average mortality of 2.2%. In many patients, the disease is mild or self-limiting, however in a considerable portion the disease is severe and fatal. Consequently, it is vital to be able to identify in advance those patients who are at greatest risk of mortality, to enable prompt referral to appropriate care settings, to try and improve outcomes.
Recently, clinical scores to predict the occurrence of critically ill patients and/or a fatal outcome with COVID-19 were developed in a cohort of 1,590 Chinese patients treated in more than 575 centers throughout China.4 Liang et al.4 identified 10 independent predictive factors (abnormal chest radiograph, age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil-to-lymphocyte ratio, lactate dehydrogenase and direct bilirubin) which were used to produce a risk score which had a mean area under the curve (AUC) of 0.88 in both the development (95% confidence interval [CI], 0.85–0.91), and validation cohort (95% CI, 0.84–0.93), for estimating the risk a hospitalized patient with COVID-19 will develop critical illness. Further reports have shown methods or new severity scores to assess disease severity and mortality of COVID-19 infection (Table 1).56789 However, it is important to understand that the mortality of COVID-19 varies according to race and ethnicity, and therefore the accuracy of risk scores is not necessarily transferrable between countries.10 Therefore, the present study aimed to develop a novel COVID-19 in-hospital mortality risk score (hereafter referred to as COVID-Mortality Score), based on data rapidly obtainable soon after hospital admission in South Korea.
Table 1
Current overview of COVID-19 severity or mortality prediction models
| Author/setting/outcome | Score variables | Sample size (No. of participants for validation set) | Predictive performance |
|---|---|---|---|
| Liang et al.4/data from China, patients with confirmed COVID-19/in-hospital mortality | 10 variables (chest radiographic abnormality, age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil-to-lymphocyte ratio, lactate dehydrogenase, direct bilirubin) | 1,590 (710) | C index 0.88 (95% CI, 0.84–0.93) |
| Altschul et al.5/data from US, patients with confirmed COVID-19/in-hospital mortality | 6 variables (age, oxygen saturation, mean arterial pressure, blood urea nitrogen, C-Reactive protein, international normalized ratio) | 4,711 (2,356) | C index 0.80 (95% CI, 0.79–0.82) |
| Gao et al.6/data from China, patients with confirmed COVID-19/in-hospital mortality | 14 variables (age, sex, fever, sputum, consciousness, respiration rate, oxygen saturation, chronic kidney disease, number of comorbidities, platelet, blood urea nitrogen, D-dimer, albumin, lymphocyte) | 2,520 (1,080) | C index 0.96 (95% CI, 0.95–0.98) |
| Yadaw et al.9/data from US, patients with confirmed COVID-19/in-hospital mortality | 3 variables (age, minimum oxygen saturation, type of patient encounter) | 3,841 (1,210) | C index 0.91 (95% CI, 0.88–0.94) (retrospective), 0.94 (95% CI, 0.90–0.97) (prospective) |
METHODS
Data sources and processing
We obtained medical records from laboratory-confirmed hospitalized cases with COVID-19 reported to the Korea Centers for Disease Control and Prevention (KCDC) between January 2020 and April 2020. All 120 hospitals in Korea that were assigned to treat COVID-19 patients submitted the clinical data of all their hospitalized cases with laboratory-confirmed COVID-19 infection to the KCDC by April 30, 2020. All patients with COVID-19 were diagnosed and treated according to the guidelines published by the KCDC (http://www.cdc.go.kr). According to the KCDC guidelines, laboratory confirmation for COVID-19 infection was defined as a positive result on real-time reverse-transcription polymerase-chain-reaction assay of nasal and oropharyngeal swabs. All the patients analyzed either died in hospital or were discharged home despite the limitations that we could not acquire the information for in-hospital mortality due to other causes except COVID-19 or underlying diseases. We collected the data of all COVID-19 patients on clinical status at hospitalization (clinical symptoms and signs, complete blood count (CBC) findings, disease severity, and discharge status). The patients' data were collected up to death or discharge from the hospital. Ordinary variables were converted into separated dichotomous variable. We randomly selected a development and validation set using a 7:3 ratio, respectively. Imputation for missing variables was considered if missing values were less than 30%. We used predictive random forest algorithm for the imputation.
Potential predictive variables
The 35 potential predictive variables included the following patient characteristics at hospital admission: demographic variables and body mass index (BMI), medical history, clinical signs and symptoms, and CBC findings. Demographic variables such as age and sex and BMI were collected for the study. Medical history included diabetes mellitus, hypertension, heart failure, cardiovascular disease, bronchial asthma, chronic obstructive pulmonary disease, chronic renal disease, malignancy, chronic liver disease, autoimmune disease, and dementia. Clinical signs and symptoms included categorical and continuous variables as follows: systolic blood pressure (SBP), diastolic blood pressure (DBP), heart rate, body temperature, fever, cough, sputum, sore throat, rhinorrhea, myalgia, fatigue, dyspnea, headache, unconsciousness, nausea/vomiting, and diarrhea. CBC findings were included as follows: hemoglobin, hematocrit, lymphocyte, platelet, and white blood cell (WBC).
Outcomes
We adopted in-hospital death as the end point because death is the most serious outcome of COVID-19.
Selection of variables and construction of scoring system
We included all 3,940 patients hospitalized with COVID-19 in the development set for selection of variables and development of mortality score. As previously stated, the 35 variables were involved in the selection process. Least Absolute Shrinkage and Selection Operator (LASSO) regression was used for the initial variable selection. For datasets with a low events per variables ratio, LASSO is more appropriate than the stepwise regression analysis, and it is more satisfactory for regression models with high-dimensional predictors. This is a logistic regression model that obtains the subset of predictors that minimized prediction error for a quantitative variable. In penalized regression, it is needed to specify a constant λ to adjust the amount of the coefficient shrinkage. With larger penalties, the estimates of weaker factors shrink toward zero so that only the strongest predictors remain in the model.1 We used LASSO regression augmented with 10-fold cross-validation for internal validation. The best covariates that minimized the cross-validation prediction error rate were defined as the minimum (λ min). The R package “glmnet” statistical software (R Foundation) was used to perform the LASSO regression. The variables identified by LASSO regression analysis that were independently significant in logistic regression analysis were used to generate the risk prediction model (COVID-Mortality Score). The COVID-Mortality Score was generated on the basis of coefficients from the logistic model. We used the following equation to estimate the probability: probability = exp(Σβ × X)/[1 + exp (Σβ × X)].
Assessment of accuracy of prediction model
The accuracy of COVID-Mortality Score was analyzed for using the AUC of the receiver-operator characteristic curve. For internal validation, we used 200 bootstrap resamplings. Statistical analysis was performed with R software (version 4.0.2, R Foundation), and P < 0.05 was considered statistically significant. To validate the generalizability of COVID-Mortality Score, we used data from the 1,688 patients not included in the development set.
RESULTS
Characteristics of the development set
In the development set, on hospital admission, 134 of 3,940 patients (3.4%) were admitted directly to the intensive care unit (ICU), with the rest (96.6%) admitted to the general ward. The end point of mortality occurred in 4.3% (n = 169). Patients who died were more likely to be men, and were more likely to have a BMI < 18.5, hypertension, an admission to the ICU and more comorbidities than those who lived (Table 2). Fever (23.1%), cough (42.0%), sputum (29.0%), sore throat (15.8%), rhinorrhea (10.9%), myalgia (16.3%), fatigue (4.0%), dyspnea (12.3%), headache (16.8%), unconsciousness (0.6%), nausea/vomiting (4.5%), and diarrhea (9.1%) were the commonest symptoms. CBC findings of the development set are also presented in Table 2. When CBC findings such as WBC, lymphocyte, platelet, and hemoglobin were evaluated as continuous variables, they did not exhibit a significant U-shaped pattern.
Table 2
Clinical characteristics and complete blood cell findings among patients in the development set who did or did not develop mortality
Predictor selection
The 35 variables measured at hospital admission (Table 2, see method) were included in the LASSO regression. After LASSO regression selection (Supplementary Fig. 1), 17 variables remained as significant predictors of death, including age, sex, BMI, SBP, heart rate, body temperature, comorbidities including rhinorrhea, dyspnea, unconsciousness, diabetes mellitus, hypertension, chronic obstructive pulmonary disease, malignancy, WBC, lymphocyte, platelet, and hemoglobin. Inclusion of these 17 variables in a logistic regression model resulted in 13 variables that were independently statistically significant predictors of death and was included in risk score. These variables included age by three groups: 60–69 years (odds ratio [OR], 3.63; 95% CI, 1.64– 8.01; P = 0.001), age 70–79 years (OR, 6.12; 95% CI, 2.84–13.16; P < 0.001), age ≥ 80 years (OR, 21.24; 95% CI, 9.65–46.74; P < 0.001), men (OR, 1.67; 95% CI, 1.04–2.67; P = 0.034), BMI < 18.5 (OR, 3.38; 95% CI, 1.64–6.95; P < 0.001), diabetes mellitus (OR, 2.10; 95% CI, 1.33–3.31; P = 0.001), malignancy history (OR, 2.78; 95% CI, 1.14–6.79; P = 0.025), dementia (OR, 2.67; 95% CI, 1.49–4.78; P < 0.001), rhinorrhea (OR, 0.27; 95% CI, 0.08–0.91; P = 0.035), dyspnea (OR, 4.03; 95% CI, 2.50–6.48; P < 0.001), unconsciousness (OR, 25.10, 95% CI, 6.55–96.18; P < 0.001), WBC (OR per 103 μL, 1.10; 95% CI, 1.04–1.17, P < 0.001), lower lymphocyte proportion (OR per %, 0.92; 95% CI, 0.89–0.94; P < 0.001), lower platelet count (OR per 104 μL, 0.90; 95% CI, 0.88–0.93, P < 0.001), and lower hemoglobin level (OR per g/dL, 0.81; 95% CI, 0.72–0.92; P = 0.001) (Table 3).
Table 3
Multivariable logistic regression model for predicting development of death in patients hospitalized with coronavirus disease 2019 in Korea
The performance, validation and optimization of COVID-Mortality score
The clinical characteristics and CBC findings of the validation set are presented in Supplementary Table 1. The AUCs of the model with the development set and validation set were 0.97 (95% CI, 0.85–0.91, Fig. 1) and 0.96 (95% CI, 0.84–0.93, Fig. 2), respectively. When the scoring system was optimized for > 90% sensitivity by stratifying age groups, the accuracy was 81.0% with 91.7% sensitivity and 80.5% specificity in the development set. From the validation set, accuracy was 80.2% with 86.1% sensitivity and 80.0% specificity. The optimized scoring system was utilized for the construction of the online risk calculator (https://www.diseaseriskscore.com). The online risk calculator determined whether the patient belonged to high-risk group or low-risk group and presented hazard ratio and high rank percentage for mortality. The model-derived score thresholds used for the optimized scoring system were 0.51 for age less than 40, 0.00252 for age 40s, 0.00176 for age 50s, 0.02302 for age 60s, 0.09532 for age 70s, and 0.13311 for age equal or more than 80.
DISCUSSION
This study developed and validated a clinical risk score (COVID-Mortality Score) to predict mortality among patients hospitalized with COVID-19 infection. Importantly the 13 variables required for calculating the risk of mortality using this score are all generally readily available on hospital admission. Practically, if the patient's estimated risk for death is low, the clinician may choose careful monitoring, whereas high-risk estimates might require aggressive treatment or immediate ICU care. In this context, we optimized the scoring model to achieve higher sensitivity, to the detriment of accuracy, given the potentially lethal clinical outcome of COVID-19.
Furthermore, this score-based model could assist clinicians when making decisions. For example, clinicians may treat patients with a high-risk score more intensively in an emergency, if resources and ICU beds were limited. Older age, sex, BMI, diabetes mellitus, malignancy, dementia, rhinorrhea, dyspnea, unconsciousness, WBC, lymphocyte, platelet, and hemoglobin were all included in the COVID-Mortality Score. Previous studies have found several of these variables to be risk factors for severe illness related to COVID-19 (Table 1). Wu et al.11 found that older age and more comorbidities were associated with a higher risk of developing acute respiratory distress syndrome (ARDS) in patients infected with COVID-19. Recently Liang et al.4 developed a risk score based on characteristics of COVID-19 patients at the time of admission to the hospital to predict a patient's risk of developing a critical illness. They found that from 72 potential predictors, 10 variables were independent predictive factors and were included in a risk score which had a mean AUC of 0.88 in both the development (95% CI, 0.85–0.91) and validation (95% CI, 0.84–0.93) cohorts. Some of the variables in the Chinese model such as age, dyspnea, unconsciousness, and cancer history were also included in the COVID-Mortality Score, and despite its development in a Korean population, which could limit its generalizability to other areas of the world, the present results show that the AUC is similar at 96 to 97%. Nevertheless, the current COVID-Mortality Score will be needed to validate externally with heterogenous baseline characteristics cohorts because of the limitation of only available predictors in current data.
While mortality prediction is neither perfect nor absolute, having a simple score to predict how severe a patient's illness is and their hospital course, will aid admitting and emergency room physicians in triaging the severity, and predicting the prognosis of COVID-19 infection, which we are realizing has a very broad-spectrum of severity. This can also be used to guide recommendations for palliative care consultations early in a patient's hospital course.
Although this study includes a large sample size for constructing the risk score and a relatively big sample for validation, the data for score development and validation are entirely from Korea and are limited only in specified predictors and mortality as outcomes, limiting the generalizability of the risk score in other areas of the world. However, despite these differences in race, the risk score remained valid in predicting in-hospital mortality.
In conclusion, we developed a risk score to estimate the risk of developing mortality among patients with COVID-19 based on 13 variables commonly measured on admission to hospital in this study. This score could help identify patients in need of more supportive treatment or assist with optimizing the use of medical resources.
XML Download