PDF
Citation
Print
INTRODUCTION
Coronavirus disease 2019 (COVID-19), which originated in Wuhan, China in December 2019, is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [1]. More than 100 million people have been infected with SARS-CoV-2 and more than two million deaths due to COVID-19 have been reported worldwide in approximately one year [2]. The number of patients with confirmed disease includes only those who have been tested positive for SARS-CoV-2 following a hospital visit [3]. Therefore, the actual number of COVID-19 positive cases has been underestimated. To determine the size of the infected population and to establish quarantine measures, accurate serological testing is required.
Seroprevalence studies have been conducted in many countries, including the United States, the United Kingdom, Spain, and Korea [4-8]. In less than a year, several types of antibody assays have been developed worldwide. However, comparative studies on the performance of assays available in Korea to determine seroprevalence have not yet been conducted. The available antibody assays mainly use recombinant spike (S) proteins, nucleocapsid (N) proteins, receptor-binding domains, S1 antigens, and combinations of these antigens to detect IgG, IgM, and total antibody levels [9-16].
We evaluated the clinical performance of COVID-19 antibody assays available in Korea for seroprevalence studies. We further estimated the positive predictive values (PPVs) of individual and two combined assays using the sensitivities and specificities obtained from this study and the expected prevalence in Korea. We also investigated cross-reactivity using serum samples from patients with antibodies to various viruses and bacteria, autoimmune disease, or monoclonal gammopathy.
MATERIALS AND METHODS
Clinical samples
Serum samples, leftover from laboratory tests and designated to be discarded, from 398 patients diagnosed as having COVID-19 at two hospitals (Seoul Medical Center, Seoul, Korea and Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Korea) and the Korea Disease Control and Prevention Agency (KDCA) were collected between March and September 2020 and stored at –70°C until analysis. The dates of symptom onset and hospital admission were obtained retrospectively from the medical records at the two hospitals. Serum samples of 510 negative controls, collected before 2018 (pre-pandemic period), were obtained from the National Biobank of Korea, the KDCA, and the High-Risk Human Serum Bank of Chung-Ang University (Seoul, Korea). A total of 168 samples were tested for cross-reactivity, including 136 residual serum samples of patients with antibodies to other viruses (human (h)CoV-229E, -NL63, -OC43, and -HKU1; adenovirus; influenza A virus; influenza B virus; human metapneumovirus; parainfluenza virus type 1/2/3/4; respiratory syncytial virus; rhinovirus; Mycoplasma pneumoniae; Chlamydia pneumoniae; hepatitis A/B/C virus; Epstein-Barr virus; cytomegalovirus; herpes simplex virus; mumps virus; and varicella zoster virus) collected from two hospitals (Severance Hospital, Seoul, Korea and Hallym University Dongtan Sacred Heart Hospital). Serum samples positive for fluorescence anti-nuclear antibody, rheumatoid factor, and monoclonal gammopathy were collected from the same two hospitals.
Twenty-three serum samples positive for antibodies to SARS-CoV-1, Middle East respiratory syndrome (MERS)-CoV, hCoV-229E, -NL63, -OC43, and -HKU1, influenza B virus, respiratory syncytial virus, and adenovirus were purchased from Trina Bioreactives (Nänikon, Switzerland). Five serum samples from patients with monoclonal gammopathy were collected from the Korea Institute of Radiological and Medical Sciences Radiation Biobank (Seoul, Korea), and four MERS-CoV convalescent serum samples were provided by the International Vaccine Institute (Seoul, Korea) (Fig. 1). The Public Institutional Review Board designated by the Ministry of Health and Welfare (http://public.irb.or.kr/) approved the study protocol and waived the need for informed consent (P01-202008-31-002).
SARS-CoV-2 antibody assays
In August 2020, the KDCA received applications from 18 manufacturers to evaluate the performance of 21 SARS-CoV-2 antibody assays. It was not feasible to evaluate all 21 assays given the limited amount of serum samples; hence, a screening evaluation was required. After the Evaluation Committee did a preliminary assessment of the data submitted by the manufacturers, five assays were selected for performance evaluation: Elecsys Anti-SARS-CoV-2 assay on the Cobas e801 platform (Roche Diagnostics, Mannheim, Germany), Abbott Architect SARS-CoV-2 IgG assay on the Architect i2000SR platform (Abbott Laboratories, Abbott Park, IL, USA), Atellica IM SARS-CoV-2 Total assay on the Atellica platform (Siemens, Munich, Germany), SD Biosensor Standard E COVID-19 Total Ab assay (SD Biosensor, Suwon, Korea), and LG Chem AdvanSure SARS-CoV-2 IgG assay (LG Chem, Seoul, Korea). SD Biosensor and LG Chem ELISAs were performed using the Epoch Microplate Spectrophotometer and ELx50 Filter Microplate Washer (both from BioTek Instruments, Winooski, VT, USA). The principle, instrument, antibody detected, reagents used, cutoff value, sample volumes, and time to first results of all assays are listed in Table 1. All assays were carried out according to the manufacturers’ instructions. Most assays were performed at Hallym University Dongtan Sacred Heart Hospital by one laboratory technician and one scientific researcher, and the Atellica IM SARS-CoV-2 total assay was performed at Severance Hospital by one laboratory technician.
Performance evaluation of the five SARS-CoV-2 antibody assays
Sensitivity and specificity
The diagnostic sensitivity and specificity (and the 95% confidence intervals) of the five assays were calculated according to days after symptom onset using MedCalc software version 19.8 (MedCalc Software, Ostend, Belgium). The date of symptom onset was not available for some patients.
Cross-reactivity and interference
The potential cross-reactivity with other coronaviruses, such as SARS-CoV-1, MERS, respiratory viruses, and other viruses, bacteria, autoimmune diseases, and monoclonal gammopathy was evaluated using patient serum samples that were positive for antibodies against the relevant pathogen. For interference testing, Hb (10 mg/mL), bilirubin (0.4 ng/mL), a triglyceride mix (20 mg/mL), or zanamivir (10 mg/mL; all from Sigma-Aldrich) were spiked into serum samples that were negative for these antibodies. The results of spiked samples were compared with those of non-spiked samples and considered acceptable if the difference was <15%.
Agreement and correlation
The agreement and correlations between the results of the five SARS-CoV-2 antibody assays were calculated using MedCalc software and were reported as correlation graphs. For the Siemens Atellica IM SARS-CoV-2 Total assay, antibody titers <0.05 were considered 0, and those >10 were considered 10. The upper limits of detection values were determined to be 4.0 for the SD Biosensor assay and 4.5 for the LG Chem assay. Positive, negative, and total agreement between assays were evaluated using Cohen’s kappa statistics and categorized as poor (<0.00), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or nearly perfect (0.81–1.00).
ROC curve analysis
ROC curve analyses and areas under the ROC curve (AUC) calculations for the five assays were performed using the MedCalc software. P≤0.05 was considered statistically significant. For each ROC curve, we recalculated the cutoff values so that the Youden index (=sensitivity+specificity–1) was maximized, and the sensitivity and specificity were determined from the calculated cutoff values.
PPVs and negative predictive values (NPVs) for individual and two combined assays using varying seroprevalence settings
PPVs and NPVs of individual and two combined assays using the calculated sensitivity and specificity values and the predictive seroprevalence of 10%, 5%, 2%, 1%, and 0.1% were calculated using a US Food and Drug administration online calculator [17].
RESULTS
Sensitivity and specificity
The diagnostic sensitivities were 93.5% for the Roche assay, 92.2% for the Abbott assay, 95.7% for the Siemens assay, 98.0% for the SD Biosensor assay, and 97.0% for the LG Chem assay. The diagnostic specificities were 99.7% for the Roche assay, 99.4% for the Abbott assay, 100% for the Siemens assay, 99.3% for the SD Biosensor assay, and 97.5% for the LG Chem assay (Table 1). The positivity rate was low in the first week after symptom onset but was nearly 100% three to four weeks after symptom onset. The SD Biosensor assay showed the highest sensitivity and the Siemens assay showed the highest specificity.
Cross-reactivity and interference
The Abbott, Roche, and Siemens assays showed negative cross-reactivity for 33 species of viruses and bacteria in 168 positive serum samples (Supplemental Data Table S1). The SD Biosensor and LG Chem assays each showed three false-positive results in the cross-reactivity study. No samples showed interference for Hb, a triglyceride mix, bilirubin, albumin, or zanamivir.
Agreement and correlation
The agreement among all assays was nearly perfect (kappa>0.90) (Table 2); however, correlations among the assays were non-linear (Supplemental Data Fig. S1), and several samples exceeded the upper limit of detection in the Siemens, SD Biosensor, and LG Chem assays. The Siemens and SD Biosensor assays showed the highest agreement rate at 98.7% (95% CI, 97.8–99.3; kappa, 0.966). Discordant results among the assays are shown in Supplemental Data Table S2.
ROC curve and cutoff analysis
The AUCs calculated from ROC curve analysis were 0.976, 0.987, 0.984, 0.994, and 0.987 for the Roche, Abbott, Siemens, SD Biosensor, and LG Chem assays, respectively (Table 3). The optimal cutoff values, calculated using the Youden index method, were 0.188 cutoff index (COI) for the Roche assay, 0.44 index for the Abbott assay, 0.57 index for the Siemens assay, 0.404 OD for the SD Biosensor assay, and 1.163 S/CO (signal/cutoff) for the LG Chem assay, which differed from the original manufacturers’ cutoff values. Using the calculated cutoff values, the sensitivities of the Roche, Abbot, and Siemens assays increased (Roche, from 93.5% to 96.5%; Abbott, from 92.2% to 96.2%; Siemens, from 95.7% to 96.7%), whereas the specificities decreased (Roche, from 99.7% to 98.1%; Abbott, from 99.4% to 99.0%; Siemens, from 100% to 99.6%). By contrast, the sensitivities and specificities of the SD Biosensor and LG Chem assays did not significantly differ when using the manufacturers’ or calculated optimal cutoff values (Table 3).
PPVs and NPVs for individual and two combined assays using determined sensitivity, specificity, and seroprevalence
The lower the prevalence rate (from 10% to 0.1%), the lower is the PPV. The Siemens assay showed the highest specificity of 100% (95.2%; PPVs calculated using the lowest value of the 95% CI of the calculated specificity are shown in parentheses because the specificity was calculated as 100%, even at the lowest prevalence rate) among the five assays at a 10% prevalence and the highest specificity of 100% (15.1%) at a 0.1% prevalence (Table 4). When the predicted prevalence rate of 0.1% in Korea was considered, the PPV was as low as 24.2% for the Roche assay and 13.7% for the Abbott assay. However, the PPVs increased when two assays were both positive or when the orthogonal test algorithm (i.e., employing two assays in sequence when the first assay yields a positive result) [17] was used (Table 4).
DISCUSSION
The sensitivities of the five assays were all >93%, and the sensitivities were low in the first week after symptom onset but were nearly 100% three to four weeks after symptom onset. These results are similar to or better than those in previous studies [10, 12-20]. In a previous study, some SARS-CoV-2 antibody assays showed a sensitivity of 100%; however, that study used only samples collected two weeks after symptom onset [17]. The assay sensitivity may vary across studies depending on how many of the samples that are collected in the first week after symptom onset (especially on days 0–3) are included. In this study, the SD Biosensor and LG Chem assays had higher sensitivity but lower specificity than the Roche, Abbott, and Siemens assays. The Roche, Abbott, and Siemens assays use high cutoff values to increase specificity, according to the CDC guidelines [21]. The sensitivity and specificity and the cutoff may vary depending on the purpose of the assay. If the assay is used for diagnostic screening, a high sensitivity is preferred. However, the specificity should be high when the assay is used to investigate seroprevalence or the effect of vaccination. Laboratories can adjust assay cutoff values according to the intended purpose.
We extensively investigated cross-reactivity using serum samples of patients with antibodies to various viruses and bacteria, autoimmune disease, or monoclonal gammopathy. The Roche, Abbott, and Siemens assays showed negative results for 163 samples tested for cross-reactivity, whereas the SD Biosensor and LG Chem assays showed false-positive results in three out of the 163 samples. These false-positive results seemed to be due to nonspecific reactions as only one sample yielded a positive result in each species. We found no cross-reactivity in serum with anti-SARS-CoV-1 antibody, which is considered the most similar to the anti-SARS-CoV-2 antibody, or in hypergammaglobulinemia serum.
This study was the first to estimate the PPVs of individual and two combined assays based on the orthogonal test algorithm, using the sensitivities and specificities calculated in this study and the expected prevalence in Korea. As the prevalence in Korea is expected to be <0.1% at present, if only one assay was used to determine the seroprevalence, the PPV was as low as 24.2% (8.2%) for the Roche assay, 13.7% (5.9%) for the Abbott assay, 100% (15.1%) for the Siemens assay, 57.5% (10.8%) for the SD Biosensor assay, and 3.5% (2.3%) for the LG Chem assay (Table 4). When two assays were combined, the PPV increased to >95% for all combinations.
We evaluated the performances of SARS-CoV-2 antibody assays available in Korea during September 2020, and the assay reagents were still in development during this period; hence, the sensitivity and specificity of all assays could be improved, and the cutoff values could be adjusted. Presently, other SARS-CoV-2 antibody assays have been developed and are available in Korea.
One limitation of this study is that neutralizing antibody assays that may represent personal immunity were not included [16]. Because most vaccines use the S protein as an immunogen, SARS-CoV-2 antibody assays targeting the N protein, such as the Abbott and Roche assays evaluated in this study, would not be useful to evaluate the vaccine response. However, the use of combinations of assays targeting the N and S proteins may allow discriminating natural infections from vaccinations.
In summary, this study was performed to select appropriate SARS-CoV-2 antibody assays for implementation in a large-scale seroprevalence study in Korea. We estimated PPVs of individual and two combined assays based on the orthogonal test algorithm, using calculated sensitivity and specificity and expected prevalence of COVID-19 in Korea. Because the prevalence of COVID-19 in Korea is considered exceptionally low at present, we recommend using a combination of two or more assays rather than a single assay.
XML Download