This study was conducted prospectively in the medical ICUs at three tertiary teaching hospitals and one secondary hospital from January 2015 to February 2016. Eligible patients were those who died in a medical ICU. Attending doctors (resident, fellow, or attending physician) and nurses caring for these patients at the time of death were asked to complete the translated QODD questionnaire within 48 hours of the death. The raters were permitted to complete the questionnaires only a limited number of times per month (a maximum of three times per month for a doctor and two times per month for a nurse).

All completed questionnaires were collected in a designated storage box in the ICU. The raters were also asked to complete a brief questionnaire about their own information (e.g., age, sex, religion, education, length of experience). The decedents’ demographic characteristics (e.g., sex, age, previous medical history) and clinical information (e.g., reason for ICU admission, severity, hospital day, length of ICU stay, life support equipment, resuscitations within 24 hours) were obtained from their medical records.

We translated version 3.2 of the QODD questionnaire, which is designed for self-administration by a healthcare professional who cared for a patient who later died in a hospital or ICU setting [11]. We obtained permission to use the questionnaire from one of the developers prior to commencement of the study. Two experts translated the original questionnaire into Korean individually and then integrated their individual translations into a single translation. This version was retranslated into English by two additional experts, and an English language expert compared that version with the original. The final Korean version of the QODD questionnaire is included in the Supplementary Material.

The QODD questionnaire has four subcategories with items to be rated: patient experiences at the end of life (15 items), medical care at the end of life (four items), experience at the moment of death (three items), and overall ratings of care (two items). For each item, all raters were asked to evaluate the quality of each experience on a scale of 0 (“terrible”) to 10 (“almost perfect”). Twenty designated items are used for calculating the QODD score, which involves summing valid ratings, dividing by the number of valid items and finally multiplying the result by 10. The QODD score ranges from 0 to 100, with a higher score indicating a better quality of death for the patient [11].

The raters were also asked to complete a visual analogue scale (VAS) questionnaire to validate the QODD questionnaire. The VAS questionnaire consisted of two items for rating: “effect on the patient’s quality of death by medical treatment and the patient’s experience in the ICU” and “overall quality of ICU care by medical professionals.” Each item was rated on a scale from 1 (excellent) to 10 (inadequate). The mean value of valid items multiplied by 10 was designated as the “VAS score.”

All data were presented as median (range) or median (interquartile range) for continuous variables and number (percent) for categorical variables. We used IBM SPSS version 22.0 (IBM Corp., Armonk, NY, USA) for the statistical analyses. The validation process consisted of construct validity, internal consistency, and interrater reliability. Construct validity is defined as the extent to which a test assesses the concept intended to be measured [12]. To assess construct validity, we calculated the Pearson correlation coefficient between the QODD score and the 100-VAS score. Internal consistency estimates were related to item homogeneity or the degree to which the items on a test jointly measure the same construct [13]. We assessed internal consistency by calculating the Cronbach’s alpha if a given item was deleted. A Cronbach’s alpha that exceeded 0.8 indicated good internal consistency. The interrater reliability is the extent of agreement among raters. The interrater reliability between doctors and nurses was assessed by calculating intraclass correlation coefficients (ICCs). ICCs that exceed 0.75 indicate an excellent agreement, while those below 0.40 suggest poor agreement. A P-value <0.05 was considered statistically significant.

This study was approved by the Institutional Review Board of each hospital. Written informed consent was obtained from medical staff at the time of each patient’s death. The requirement for written consent from the patients was waived because their personal information was not used. We conducted this study in compliance with the Declaration of Helsinki.

Of the 255 total patients with a mean age of 66.6 years, 39.6% had an underlying malignancy at the time of ICU admission (Table 1). Respiratory failure was the leading cause of ICU admission, and 20.0% of patients were admitted for post-resuscitation care. Forty-eight (18.8%) had received cardiopulmonary resuscitation within 24 hours of death. Documented do-not-resuscitate orders were in place for 206 patients (80.8%).

In total, 177 medical staff members completed 416 questionnaires during the 12-month study period, including 66 doctors: 52 residents (78.8%), nine fellows (13.6%), and five attending physicians (7.6%). The characteristics of the respondent ICU medical staff members are detailed in Table 2.

The results of the assessment of construct validity are presented in Table 3. The QODD score and subcategorical items were positively correlated with the 100-VAS score (Pearson correlation coefficient, 0.348; P<0.001).

Table 4 contains the results of the assessment of internal consistency. The calculated Cronbach’s alpha exceeded 0.8 for all items, which showed good internal consistency and item heterogeneity.

Three hundred six questionnaires that were completed by both doctors and nurses and described 153 patients were used for assessing interrater reliability. the ICC for the QODD score was 0.077, and there was no statistical significance (P = 0.312) (Table 5).