Data on discharged patients and outpatient clinic visits were obtained from Ewha Womans University Seoul Hospital (a single secondary medical center) and Ewha Womans University Mokdong Hospital (a single tertiary medical center) in Korea. We selected 6,493 patients aged 3 40 years who were diagnosed with diabetic microvascular complications more than once based on the Korean Standard Classification of Disease. These classifications were reorganized from codes in the 10th version of the ICD (ICD-10) between January 2019 and December 2020. We randomly selected the diagnoses of 200 patients, 100 from each hospital (Fig. 1). No patients overlapped between the data of both hospitals. Clinical data for the participants during this period were stored in the Ewha Womans University Hospital electronic medical records (EMR) system and were retrospectively extracted from the hospital case notes. The case notes included information on diabetic microvascular complication diagnoses, medical history, symptom descriptions, neurologic examination findings, blood laboratory tests, urinary laboratory tests, nerve conduction studies, electromyography, fundoscopy, operative recordings, and prescriptions. This study was approved by the Institutional Review Board of Ewha Womans University Seoul Hospital (approval number SEUMC 2021-04-050).

We identified 7,065 patients with diabetic microvascular complication codes. The excluded patients comprised 572 who had only visited the hospital and were diagnosed with one of the relevant codes once or who did not undergo the gold standard tests. We then randomly selected 100 patients from each participating hospital for each disease. Among the 200 patients who visited the hospital more than once and were diagnosed with codes for each diabetic microvascular complication (diabetic nephropathy [E11.2, E12.2, E13.2, E14.2, or N08.3], diabetic neuropathy [E11.4, E12.4, E13.4, E14.4, or G63.2], and diabetic retinopathy [E11.3, E12.3, E13.3, E14.3, or H36.0]), the gold standard of diabetic nephropathy was defined as follows: (1) an estimated glomerular filtration rate of ≤ 60 mL/min/1.73 m² based on the Modification of Diet in Renal Disease study,15 (2) a urinary albumin-to-creatinine ratio of > 300 mg/g, or (3) end-stage kidney disease with hemodialysis.16,17 The gold standard of diabetic neuropathy was defined as follows: (1) abnormal nerve conduction study (NCS) results, with two or more different nerves with abnormal findings more than two standard deviations from the normal range in three or four of the parameters (motor and sensory amplitudes, conduction velocities, F responses, and latencies);18 (2) abnormal sudomotor test, such as a prolonged sweat response latency, a reduced or absent sweat response, or an elevated sweat volume;19 or (3) signs and symptoms of peripheral neuropathy, such as decreased sensation (e.g., vibration, proprioception, temperature, or pinprick sensations) in distal limbs in a “stocking and glove” distribution, and a decrease from a proximal to a distal gradient of the deep tendon reflex.20,21 Lastly, the gold standard of diabetic retinopathy was defined as cases with proliferative or nonproliferative retinopathy according to an ophthalmologist examination using fundoscopy.17,22

Five algorithms were analyzed: (1) only ICD codes (≥ 2 visits with diagnostic code and ≥ 3 visits with diagnostic code; two algorithms) and (2) ICD code and specific tests and/or prescriptions (three algorithms). For diabetic nephropathy, laboratory tests including blood chemistry or urine proteinuria were combined with the diagnostic codes for validation. In each analysis, eight patients who did not receive a blood chemistry test and 54 who did not receive a urine proteinuria test were excluded. For retinopathy, a fundoscopy, an ophthalmologist visit, prescription records for specific agents (ophthalmic solution with 0.1% bromfenac, calcium dobesilate hydrate, sulodexide, vaccinium myrtillus, and bevacizumab), and procedure execution data (retinal photocoagulation) were combined with the diagnostic codes for validation. In each analysis, one patient who did not receive fundoscopy, 15 who did not visit an ophthalmologist, and 38 who did not receive specific treatment were excluded. Lastly, diabetic neuropathy was validated using electromyography, including an NCS or sudomotor test, and prescription records of specific agents (pathogenic treatments [α-lipoic acid] or symptomatic treatments [γ-aminobutyric acid analogs such as gabapentin or pregabalin, serotonin-nonepinephrine reuptake inhibitors such as duloxetine or venlafaxine, and tricyclic agents such as amitriptyline or nortriptyline]). In each analysis, 74 patients who did not receive an NCS test and 44 who did not receive a specific treatment were excluded. Data extraction and classification of each disease were performed by different authors (diabetic retinopathy by T.J.S., diabetic neuropathy by M.S.P., and diabetic nephropathy by H.J.K.). The classification audit was performed by different authors by changing each diagnosis to minimize potential misclassification. M.S.P. and H.J.K. reviewed and discussed uncertain cases to reach a consensus.

To assess the percentage of correct diagnostic codes for diabetic microvascular complications, they were compared with the gold standard (in the case notes). The performance parameters included the positive predictive value (PPV), negative predictive value (NPV), error rate, sensitivity, and specificity. The PPV and its 95% confidence interval (CI) are used to quantify the diagnostic accuracy (number of correctly classified cases). PPV refers to the proportion of gold-standard diabetic microvascular complication cases relative to all of those identified with the diagnostic codes for diabetic nephropathy, neuropathy, and retinopathy in the Korean NHIS data. NPV was defined as the ratio of patients truly diagnosed as negative relative to all those who had negative test results. The error rate was defined as the proportion of patients with false results relative to all patients. Sensitivity and specificity analyses assessed the effect of reclassifying diagnoses using a combination of diagnostic codes and cofactors (specific tests and prescriptions for each diagnosis). Simple random sampling was used to select and allocate 100 samples for each disease from each hospital. The kappa coefficient was calculated after being independently investigated and compared by each of the two researchers. Each diabetic microvascular complication had an overall kappa value higher than 0.8, indicating an excellent degree of agreement between the researchers. All statistical analyses were performed using the open-source statistical package R (version 3.6.3; R Project for Statistical Computing, Vienna, Austria).

Of the 200 patients who visited the hospital more than once and were diagnosed with a diabetic nephropathy code, 142 had diabetic nephropathy according to the gold standard of diagnostic criteria. Diabetic nephropathy codes predicted the gold standard of diabetic nephropathy in 71.0% (PPV) of cases. Among them, 194 patients visited the hospital and were diagnosed with a diabetic nephropathy code more than twice, and had a slightly increased PPV than did the patients who visited the hospital more than once with a diabetic nephropathy code (71.0% for ≥ 2 hospital visits vs. 71.7% for ≥ 3 hospital visits) (Table 1, Supplementary Table 1). A comparison between blood tests including creatinine and urinary proteinuria examinations with diagnostic codes revealed that the latter had higher PPV (72.9% for diagnostic codes with blood tests vs. 80.8% for diagnostic codes with urinary proteinuria tests) and specificity (10.3% vs. 51.7%), and lower NPV (75.0% vs. 55.6%), error rate (27.0% vs. 26.0%), and sensitivity (98.6% vs. 83.1%). When both factors (blood and urinary proteinuria tests) were combined with the diagnostic codes, PPV (81.4% for diagnostic codes with blood and urinary proteinuria tests vs. 80.8% for diagnostic codes with urinary proteinuria), NPV (56.4% vs. 55.6%) and specificity (53.4% vs. 51.7%) increased slightly, whereas the error rate (25.5% vs. 26.0%) and sensitivity (83.1% vs. 83.1%) decreased compared with combining only one factor with the diagnostic codes. In both the tertiary and secondary hospitals, diagnostic codes in combination with other factors had a PPV of higher than 70% (Table 1).

Of the 200 patients who visited the hospital more than once and were diagnosed with a diagnostic code for diabetic neuropathy, 110 had the gold standard of diabetic neuropathy (PPV, 55.0%). Among them, 193 patients visited the hospital more than twice and were diagnosed with a diabetic neuropathy code, and PPV was slightly higher than that for the patients who visited the hospital more than once and were diagnosed with a diabetic neuropathy code (55.0% for ≥ 2 hospital visits vs. 56.0% for ≥ 3 hospital visits) (Table 2, Supplementary Table 2). When the diagnostic codes and NCS were combined, the PPV, NPV, error rate, sensitivity, and specificity were 76.2%, 81.1%, 22.0%, 87.3%, and 66.7%, respectively. When diagnostic codes and prescriptions were combined, the PPV, NPV, error rate, sensitivity, and specificity were 65.4%, 81.8%, 31.0%, 92.7%, and 40.0%, respectively. After applying the inclusion criteria, the PPV and specificity were maximized by combining primary diagnostic codes with NCS and prescription records of specific agents (PPV, 81.3%; specificity, 76.7%). In both hospitals, PPV was lower than 70% when only diabetic neuropathy codes were applied. However, including NCS (diagnostic codes with NCS or diagnostic codes with NCS and prescriptions) resulted in PPV increasing to nearly 80% in both hospitals (Table 2).

This study included 200 patients who visited the hospital more than once and were diagnosed with a diagnostic code for diabetic retinopathy from two different hospitals. Among those 200 patients, 154 (77.0%) were identified as having the gold standard of diabetic retinopathy. The patients included 183 who visited the hospital more than twice and were diagnosed with a diabetic retinopathy code, and PPV was slightly higher than that for the patients who visited the hospital more than once and were diagnosed with a diabetic neuropathy code (77.0% for ≥ 2 hospital visits vs. 79.2% for ≥ 3 hospital visits) (Table 3, Supplementary Table 3). Combining fundoscopy with the diagnostic codes increased PPV slightly, to 77.4%. Combining the ophthalmologist visit history with those results increased the PPV to 81.5%. Lastly, PPV (96.6%) increased the most when the use of specific medications for diabetic retinopathy or retinal photocoagulation was combined with ophthalmologist visits, fundoscopy, and diagnostic codes. The sensitivity of all cases exceeded 90.0%, but when all parameters were combined, the specificity significantly increased to 89.1% and the error rate significantly decreased to 9.9%. In each of the two hospitals, PPV was higher than 70.0% in all cases. When specific medications against diabetic retinopathy or retinal photocoagulation, ophthalmologist visits, fundoscopy, and diagnostic codes were combined, the PPV was more than 90.0% in both hospitals (Table 3).