PDF
Citation
Print
Highlights
• Screening scores for Diabetes & Chronic Kidney Disease are available in USA & Korea.
• These scores performed well in contemporary national samples (vs. original samples).
• Korean adults tend to have lower cardiovascular risk than American adults.
• Korea is not a "safe" zone for prehypertension, prediabetes & undiagnosed diabetes.
• Screening scores could be useful tools for self-assessment and health education.
INTRODUCTION
Prediction models and risk assessment tools have been a clinical research focus for a few decades, with some promising news, but not without controversy or skepticism [1-4]. Their value remains uncertain such that they are used infrequently in practice. Although screening or prediction of many medical conditions can benefit public health, early identification of individuals at risk for type 2 diabetes mellitus (DM) and chronic kidney disease (CKD) is important for two reasons [5,6]. First, these conditions may be undiagnosed, challenging the need to screen for current or future events in individuals with or free of asymptomatic or occult disease. Second, both conditions are increasingly prevalent in many regions and closely-related or inter-dependent (with several shared risk factors and underlying mechanisms), with significant global health implications for itself as well as based on their associations with other comorbidities including cardiovascular disease (CVD) [7]. Currently, some models are widely available in research community and public domain, and provide health education via self-assessment of disease risk using a short checklist of simple questions (excluding lab data or difficult medical terms), with broad endorsement from the medical community [8-10].
In this study, we aimed to compare and validate ‘four screening tools’ to help identify undiagnosed DM (alone or with preDM) and CKD being used in the United States and Korea. The original models were developed using data from up to two decades ago [11-15] with some adaptations over years. Therefore, we evaluated the currently used models/questionnaires using the most recent national survey data collected from the two countries, with comprehensive statistical measures; let us call 2×2×2 (two medical conditions by two countries’ models by two countries’ populations/datasets) temporal and cross-validation and comparison—we focused on associational measure (via regression fit) and diagnostic performance (high vs. low risk based on the risk test). As a secondary aim, we aimed to compare these two countries’ population health in terms of DM and CKD and the related factors, which also coincide with CVD risk factors. Our study is timely as DM and CKD (and their associated risk factors) are suggested as key medical conditions among patients with coronavirus disease 2019 (COVID-19) [16], and diverging performance in COVID control has been contrasted between United States and Korea [17,18]. During the COVID-19 crisis, the California Department of Public Health (CDPH) recommends the “prediabetes quiz (taking <1 minute)” in connection with COVID-19 on the internet, TV and highway billboards, and used the prediabetes screening score to be described below [19-21]. Finally, we sought to address a criticism regarding widely-promoted web-based risk tests and “medicalization” of pre-DM [8,22,23].
METHODS
Populations, data sources, models, and variable definitions
We evaluated and compared DM and CKD screening scores used in United States for health education and disease surveillance at the American Diabetes Association (ADA), American Medical Association (AMA), and/or Centers for Disease Control and Prevention (CDC), among other authorities/agencies [8-10,19,24-26], and the Korean counterparts. Particularly, the CDC started to use DM and CKD screening scores from 2019 on their websites [9,10]. The four models were originally developed/derived from nationally representative health surveys, the National Health and Nutrition Examination Survey (NHANES) and Korea National Health and Nutrition Examination Survey (KNHANES) from 1999 to 2009; the final presentation of questionnaires and scoring algorithms are reasonably comparable and standardized, consisting of integer score (maximum score of 12). Predictors include demographics (age, gender), family and personal medical history (e.g., DM, hypertension, CVD), obesity measures (body mass index [BMI], waist circumference), and lifestyle variables (physical activity, smoking, alcohol). See the scoring algorithms in Box below; original models can be found in original publications [11-13,15,27], and original and adapted/tailored versions of models and questionnaires have been used in practice. Those in the box are the most widely used versions in practice [8-10,19,21].
For the United States study, we used NHANES 2015–2016 and 2017–2018, and for the Korean counterpart, we used KNHANES 2016, 2017, and 2018. The study population was adults ≥19 years old. For the DM study, individuals with known DM were excluded, and those with known kidney failure/disease were excluded for the CKD study, as our study outcome was ‘undiagnosed’ cases, where ‘known’ means already diagnosed so participant knew the disease status at the time of survey. In practice, unlike DM, awareness of CKD is very low, so using a questionnaire to ascertain CKD will lead to significant underestimation of the population with CKD. Thus, it is possible that individuals with known CKD were included in our CKD analysis due to limited information collected. We addressed this issue in sensitivity analysis by treating persons with estimated glomerular filtration rate (eGFR) <15 or <30 mL/min/1.73 m2 as a surrogate for CKD known to patients but not queried or captured in survey.
We also considered the combined outcome of DM and pre-DM because it has been reported that early detection of pre-DM may play a major role in disease reversal or delay [24,28-30]. Of note, the same scoring algorithm has been justified for undiagnosed DM alone, or DM and pre-DM together, employing same or different cutpoints to define at-risk groups in practice [10,12,19,31]. Currently, United States uses the same risk score for DM and pre-DM [10,19,24].
Outcome measures were based on current clinical practice guidelines in United States and Korea: briefly speaking, glucose (with/without fasting), glycosylated hemoglobin (HbA1c), and 2h-oral glucose tolerance test (OGTT) for DM and pre-DM; and eGFR based on Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) standards for both United States and Korean studies. Korean version of CKD-EPI was used as sensitivity analysis. For other variables, we also used widely accepted definitions used in clinical practice and previous research [5,24,32-35]. Of note, KNHANES did not collect data on gestational DM, anemia treatment, (congestive) heart failure, and OGTT, and both surveys did not collect data on peripheral artery or vascular disease (phrased ‘circulation disease in legs’ in the questionnaire), unlike older versions of NHANES. Of note, NHANES collected self-reported weight and height and obesity status (normal vs. overweight) but we did not use these variables in our primary analyses. See Appendix 1 for variable definitions.
Statistical analyses
We used descriptive statistics to summarize nationally representative samples based on complex survey designs: mean±standard error for continuous variables and percent for categorical variables. To represent the entire country’s adult population and compute national health statistics (e.g., mean age, prevalence of medical condition), design features—weights (i.e., interview, medical exam and fasting subsample weights for multiple years, as appropriate), clusters, and strata—were accounted for. We followed the NHANES and KNHANES analyses guidelines as well as previous publications [11,13,31]. In the descriptive and inferential health statistics (presented in Table 1), known cases of DM and CKD were not excluded in order to represent the entire adult population in the nation. In the remaining screening score or risk test-related analyses where undiagnosed case is the outcome of interest, known cases were excluded.
First, we fitted the four risk score models in the originally intended populations for temporal validation, i.e., United States models in the United States sample and Korean models in the Korean sample. Next, we performed cross-validation, i.e., United States models in the Korean sample, and Korean models in the United States sample, previously described as a ‘2×2×2 approach.’ Multiple logistic regression was used to estimate odds ratio (OR), 95% confidence interval (CI), and P value for each predictor and outcome, along with area under the receiver operating characteristic (ROC) curve (AUC). For diagnostic performance, we computed percent (%) of participants who scored high (≥5 for DM and ≥4 for CKD), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), likelihood ratio positive/negative (LR+/–), Youden index, and AUC, along with ROC curve and calibration plot.
Sensitivity analyses were conducted for checking the robustness of the primary analysis results to adaptations/modifications that have been or could be considered in practice; that is, subgroup analyses based on demographics, under different data availability, with different weighting, or variations in outcome definition, and AUCs were reported as a key prediction summary measure. When findings were inconsistent or counterintuitive, we sought to provide scientific explanation(s) of the underlying mechanism. Of note, when we evaluated regression models and the final risk test (high vs. low score), final results were qualitatively similar in weighted and unweighted analyses, especially, third decimal point changed in AUC. SAS version 9.4 (SAS Institute Inc., Cary, NC, USA) was used for data analyses.
Ethics approval
The University of California Institutional Review Board (IRB) determined that IRB review is not required because the research is based on publicly available, de-identified data (IRB ID: 1653151-1). Written informed consent by the patients was waived due to the use of publicly available data in our study.
RESULTS
Characteristics and general health status of American and Korean adults
The NHANES includes 11,000 adults in the United States and KNHANES includes 18,560 adults in Korea. Table 1 presents the characteristics of the adult population of similar mean ages, approximately 47.5 years old. The United States population demonstrated 12.1% of known DM and 3.2% of known CKD; the corresponding estimates for the Korean population were 7.7% and <0.5%, respectively. It is noteworthy, but not surprising, that obesity distributions were markedly different, although prevalence of DM and pre-DM (based on lab tests) was similar in the two countries. This offers more evidence toward postulated differences between Western versus Asian populations in terms of DM etiology, particularly, in genetics, BMI cutpoints, different obesity measures, and lifestyles [28,36-39]. Prevalence of hypertension (but not prehypertension), CVD, CKD, and family history of DM were higher in United States. Interestingly, the prevalence of physical activity and smoking (as binary variables) and mean hemoglobin (14.2 g/dL) were very similar in the two countries. Difference in alcohol consumption may be attributed to varying approaches to phrasing of queries and information collected/available or cultural differences. Among outcomes-related variables, the mean of HbA1C was 5.6% to 5.7%; prevalence of DM based on lab tests was 10% to 11% (where approximately 20 and 34% of total DM cases were newly diagnosed in the United States and Korea, respectively), and prevalence of pre-DM was 32% to 34% in the two countries. In contrast, 6.8% of the United States population and approximately 2.8% of the Korean population had eGFR<60 mL/min/1.73 m2.
Risk model fits in newest survey data in the United States and Korea: model performance
Overall, logistic regression showed direction, magnitude, and statistical significance of associations (assessed by OR, CI, and P value), qualitatively consistent with the original model developments. Although DM, the most important risk factor along with hypertension, was not statistically significant for CKD (which may seem counterintuitive), it may be explained by a statistical phenomenon called ‘multicollinearity’; in simple/unadjusted logistic regression, DM yielded OR, 2.95 (P<0.001) and was correlated with hypertension and proteinuria. Different directions in the relationship between alcohol consumption and DM in the United States versus Korea are interesting, although a protective effect of light or moderate alcohol use in DM risk prediction has been reported [40], which might further reflect differences and difficulty in measuring/quantifying consumption and/or lifestyle factors that could be region- or population-specific. AUC was 0.75 for the USA-DM model in the United States population and 0.77 for the K-DM model in the Korean population. AUC was higher for the CKD counterparts (0.89 for USA and 0.91 for Korea), where high AUCs, even higher than the values reported in the original model developments, are partly due to very strong age effect. These results are summarized in Tables 2 and 3, and Supplementary Fig. 1.
Calibration based on observed versus predicted probabilities was satisfactory, showing a similar pattern in the two countries for the same disease. For DM, these probabilities ranged from 0% to 15% based on deciles, with slight overestimation in the highest risk group (for the 10th decile, 15% predicted vs. 13% observed), and for CKD, >50% of the participants were classified as low risk, with <5% of both predicted and observed probabilities, reflecting low risk in young age group (Supplementary Fig. 2).
Diagnostic performance of the risk tests
Diagnostic characteristics, using cutpoints for the high risk group, in the four models showed sensitivity 84%–96%, specificity 45%–65%, PPV 8%–16%, and NPV >99% in temporal validation. Some of these values were even higher in some external populations, i.e., cross-validation, but overall summary prediction/discrimination measure, AUC, was robust; e.g., lowest AUC 0.71 when the K-DM model was fitted to the United States population. With regard to regression model fits in addition to diagnostic measures, KNHANES consistently showed better discrimination (higher AUC) in all four models, which we did not expect. For example, the USA-DM model provided better discrimination in KNHANES than NHANES (AUC=0.78 vs. 0.75), likely offering extra-support toward external validity or transportability of the model or randomness. Discrimination was harder when DM/pre-DM are combined, compared to DM alone, since pre-DM is a middle group in risk spectrum. Uniformly attenuated regression coefficients but in the same direction for the composite outcome of DM/pre-DM seem to justify the same risk test/questionnaire for both conditions of the same kind, which reflects the current usage in practice. Temporal- and cross-validation results along with regression model fits for DM, DM/pre-DM combined, and CKD are presented in Table 4, and Supplementary Tables 1-4. PPV >50% for DM/pre-DM means that 1 out of 2 who scored high (≥5) is likely to have DM or pre-DM that is newly identified, suggesting a reasonable yield for pre-screening/self-risk assessment, where it is well known that PPV is a function of disease prevalence.
Sensitivity analyses
Supplementary Table 3 summarizes sensitivity analyses and their impact on discrimination. AUCs were quite robust in different settings, when samples were unweighted versus weighted; predictors were assessed via questionnaires only versus supplemented/informed by physical examinations; and for different genders and races. The older age group (≥60 years old) showed lowest AUC for DM, which may imply limited value or usefulness of these screening scores in this subgroup, as previously pointed out [8]. Interestingly, women and Koreans showed uniformly higher AUCs, which may deserve future investigation and elucidation. In one extreme scenario, omitting unmodifiable demographic factors and family history demonstrated AUCs >0.7. Lowest AUC (=0.60) for CKD in Koreans who were <40 years old might reflect a uniformly low risk in this subgroup. Results were robust to somewhat different outcome definitions employed in practice—for instance, glucose alone vs. HbA1c alone for DM; original United States CKD-EPI vs. Korean version of CKD-EPI vs. older Modification of Diet in Renal Disease (MDRD) equations for CKD [5,33]. Finally, prediction equations are provided in Supplementary Table 4 to calculate (updated) probability estimates for interested users or calculators.
DISCUSSION
Models of various complexity are increasingly available for DM and CKD; yet, few have examined their performance with extensive temporal and external validations. As our primary aim, we evaluated and compared the performance of the relatively widely available DM and CKD screening or risk models in the contemporary United States and Korean populations. Overall, discrimination and risk factor-outcome associations remained stable, with reasonably consistent findings between the original, old data used for model developments versus most recent data available. Minor variations were noted as follows: AUC was somewhat lower in the current validation (0.75) versus original publication (0.79) for the USA-DM model, and the corresponding values were 0.73 to 0.77 for the K-DM model; AUC was also higher in the current validation of the USA-CKD and K-CKD models (0.83–0.88 to 0.89–0.91). In contrast, cross-validation provided somewhat intriguing results. The United States models showed better discrimination in the Korean population, which may reflect random variation and/or strong external validity. To compare, prediction models for DM developed from another CDC dataset and machine learning techniques; for example, with 27 variables from >138,000 participants, showed AUC=0.72–0.79 [41]. Also, risk for female gender in CKD scores was attenuated in our study (e.g., weaker statistical significance or reversed direction), and this is partly explained by the eGFR formula used (MDRD, CKD-EPI, or Korean CKD-EPI). As such, “female” in the CKD score in both countries may be optional.
According to currently available risk tests, 30% to 60% of respondents may be classified as being at high risk of DM or CKD [8,10,19,24]. This may cause a legitimate concern that overestimating people at risk can create false alarm or challenge the utility. Depending on perceived burden or available resources, different cutpoints (say, 4, 5, 6) may be justified; indeed, adaptations/modifications in questionnaire design (e.g., translation, cultural adaptation or substitution of Asian BMI table) and different cutpoints for high risk designation in the DM and CKD risk scores have been suggested and incorporated [10,12,19,24,25,39,42]. Further, some groups may be more accepting and responsive to tests that are tailored to their population; we generally do not recommend the K-DM model for the United States population or the USA-DM model for the Korean population despite good numerical performance. We assert that the two countries can or should adopt different obesity measures and criteria, and should try to identify lifestyle and culturally appropriate risk or protective factors (physical activity, alcohol consumption, smoking—possibly gender-specific and how to capture/measure). For instance, Korea currently does not follow the United States revised guidelines on blood pressure thresholds in order to avoid over-classifying people as high risk (with possible over-treatment) and communicational challenge with the public, based on limited evidence among Asians (comprising 1% to 2% of the entire sample studied) [33,43]. Moreover, we should emphasize ‘modifiable’ or ‘controllable’ factors in most predictions; as an example, a very high AUC is not always ideal for prediction, especially when driven by deterministic factors—whether unmodifiable (e.g., age, family history, genetics) or a surrogate, imminent marker or early onset (e.g., elevated glucose or creatinine) —which may be better suited for diagnostics, instead of meaningful prediction [44,45].
As our secondary aim, we estimated and compared the prevalence of DM and CKD (and distribution of raw variables) along with well-known and validated risk factors. As expected, there were differences in the prevalence of some factors such as obesity, but some others—especially, prehypertension, pre-DM, and HbA1c level—were comparable in the two countries. Moreover, undiagnosed DM and proteinuria were more prevalent among Koreans (with different definitions/tests used for proteinuria). Thus, Korea is not a “safe zone” for hypertension and diabetes despite a relatively healthy national profile, excellent health care access, service and longevity [17,46].
As the tertiary aim, we considered potential adverse effects from the use of widely endorsed and available risk tests and possible medicalization of pre-DM [8]. It is true that a substantial proportion of individuals are designated as high risk (e.g., over 50% having a high score, 42% having prehypertension, and 34% having pre-DM), which may create unnecessary alarm that can lead to loss of trust or wasted medical resources. A possible (small and limited) solution may be to institute a higher cutoff (e.g., 6 instead of 5) or a priority or closer monitoring given to a subgroup with a higher level of HbA1c or glucose within pre-DM designation, as the range might be broad and monotonicity in risk is apparent [34,47]. Such adaptive measures may be wiser than abandoning risk tests totally or mechanical adoption.
Indeed, it has taken a long time to educate the general public regarding CVD and its associated risk factors, such as high blood pressure and cholesterol, partly through the Framingham risk score, a landmark risk model; yet many lay persons remain unfamiliar with key biomarkers such as HbA1c, creatinine, BMI, various obesity measures, or original/revised risk score itself. Recently, we may be seeing a momentum or progress in DM and CKD health education in many countries. These (standardized) screening tools have potential to promote disease awareness, health education, and possible risk reduction globally. Although risk assessment or prediction of CVD and cancer risk are equally important, this generally requires lab testing or complex information that may exceed a layman’s medical understanding. The easy-to-use DM and CKD risk assessment tools may improve accessibility and dissemination of solidly validated, widely relevant health-related information to the general public, and may help reduce health disparity gaps—the two goals of “Healthy People 2030.”
Limitations of our study should be noted. First, unavailability and lack of standardization of some variables or tests across countries may explain some inconclusive findings, as summarized in variable definitions in the Appendix 1. For example, the role of alcohol use on DM was not further elucidated; however, this can be a common occurrence when we combine disparate studies or heterogenous populations/datasets, yet may also be viewed valuable as it reflects real-world scenarios. Second, our investigation on temporal and cross-validation was based on the same types of survey data over >10 years apart; thus, good performance must be partly due to similarities in study design, and high internal consistency and high data quality from the two developed countries [17]. Diverse real-world settings (e.g., community, hospital, resource-poor) or special contexts (e.g., human immunodeficiency virus patients or other continents) may provide less ideal results [48].
Additionally, the following points are mentioned for consideration in similar line of research (e.g., based on cross-sectional survey data), not as limitations of this particular study. Causal or directional relationships (e.g., CKD to/from anemia) cannot be drawn from our study, which focused strictly on associations or correlations for cross-sectional events—DM and CKD present but unknown. Causation is not needed for screening and prediction, but the relevance, interpretability and meaningfulness of the models could improve user acceptance [45]. Also, risk score or self-assessment (especially, with low specificity and PPV) cannot replace professional medical advice and should remain a secondary or tertiary, or pre-screening or educational tool, with proper warnings to accompany any use. Because the outcome/cases in this study are “prevalent” events identified in a cross-sectional manner, risk in our study is technically odds or probability/prevalence. This approach is necessary in screening when the goal is to identify individuals with prevalent but currently undiagnosed disease. We have compared a version of the USA-CKD screening score to point of care urine and blood tests to identify individuals with CKD [49].
The main strengths of our study are large, nationally representative, contemporary multi-country data with key variables available (or reasonable proxies) for two prolific medical conditions. To that end, we were able to study similarities and differences between the two countries, representing different environments, with high statistical power. To our knowledge, this is the first study to attempt temporal- and cross-validation of the United States and Asian prediction/screening models.
In conclusion, our study supports high internal and external validity of the DM and CKD risk scores widely disseminated in United States and Korea and beyond, adding to the body of existing independent validation efforts. Perhaps our study provides support for country-specific models for DM, which is a trend in related research. These four models may be safely used as a supporting tool (online, pencil and paper, health fair) in a variety of different settings including those with limited resources, with possible adaptation or improvement. Despite active development and promotion by researchers (including machine learning and artificial intelligence-based models), the real impact of risk scores and prediction models are infrequently studied or reported. Wise use and appropriate adaptation or revision of currently available and effective models, as ADA/CDC/AMA/CDPH did for DM and pre-DM [8], aka ‘join forces’ for a similar goal, may be preferable to developing even more (similar, different, or confusing) models with minor improvements [1,2,4]. Risk assessment tools can be an essential component of patient-centeredness and communication and shared decision-making between clinicians and patients/families [50]. Rigorous assessment and reporting of value and potential harm should continue to garner patients’ and providers’ trust in research, to evaluate the real impact of the risk prediction tools on public and personal health, and to serve as a good example in translational research.
XML Download