PDF
Citation
Print
Introduction
Lung cancer is the leading cause of cancer-related deaths, accounting for 18% (1.8 million) of all fatalities globally in 2020 [1], and non–small cell lung cancer (NSCLC) constitutes over 85% of all lung cancer cases [2]. Although neoadjuvant chemotherapy has demonstrated survival benefits in resectable NSCLC [3], assessing the efficacy of neoadjuvant therapy on overall survival (OS), the gold standard primary outcome, is resource-intensive and time-consuming. Thus, there is a need for surrogate markers of OS in neoadjuvant clinical trials.
Studies across various cancer types, such as breast cancer [4], melanoma [5], pancreatic cancer [6], and bladder cancer [7], have shown that pathologic responses following neoadjuvant therapy serve as significant predictors of survival. In NSCLC, residual viable tumor (VT) ≤ 10% was reported to be associated with improved survival [8-10]. This specific condition, a major pathological response (MPR), has been proposed as a surrogate marker for OS [11]. Recent clinical trials of neoadjuvant immunotherapy and targeted therapy have increasingly adopted MPR as their primary outcome [12].
To standardize gross protocols and slide review methods for MPR evaluation, the International Association for the Study of Lung Cancer (IASLC) has issued multidisciplinary recommendations [13]. Additionally, an area-weighted VT (wVT, %) calculation approach, henceforth referred to as the MPR calculation tool, has been proposed to mitigate potential distortions in VT (%) values stemming from variations in the tumor bed area across individual slides [14]. However, further investigations are warranted to evaluate whether the proposed MPR assessment methods can reliably reproduce interobserver agreement, enabling MPR to serve as a robust surrogate marker for survival in clinical settings. Additionally, given the distinct pathologic responses observed across histologic subtypes, proposals suggesting the use of higher VT (%) cutoffs, such as 65%, for adenocarcinoma (ADC) [15-17] necessitate further validation through comprehensive survival analyses to determine optimal cutoff values.
The primary objectives of this study were to (1) investigate the factors influencing the interobserver reproducibility of the tumor bed and VT (%) assessments, (2) compare the VT (%) values calculated by area-weighted and unweighted methods, and assess their impact on MPR determination, and (3) determine whether optimal VT (%) cutoffs vary based on histological subtypes.
Materials and Methods
1. Study cohort
This retrospective study initially identified 141 patients who received chemotherapy or chemoradiation followed by surgical resection at Seoul National University Bundang Hospital (SNUBH) between January 2009 and December 2018. After applying exclusion criteria to confirm the neoadjuvant intention prior to surgery (S1 Fig.), 108 patients were included in the final analysis. Demographic, clinical, and pathologic data of all patients were extracted from the electronic medical records of SNUBH. Clinical and pathologic tumor-node-metastasis stages were re-evaluated according to the 8th edition of the American Joint Committee on Cancer staging manual [18]. In the case of ypT0N+, ypStage was assigned following the IASLC recommendations [13].
Follow-up commenced on the day of surgery. OS was defined as the time until death from any cause. Event-free survival (EFS) was the time until the earliest occurrence of progression, recurrence, or metastasis after surgery. Lung cancer-specific survival (LCS) was the time until death attributed to lung cancer. All patients were censored at the last follow-up date of June 2023. The median follow-up duration was 49.5 months, ranging from 3.5 to 148.6 months.
2. Histological assessment
For gross sampling, the entire mass was submitted for examination if the maximum diameter was ≤ 3 cm. If the tumor was not initially identified, additional sampling was performed in areas suspected to harbor tumor. For tumors > 3 cm, representative sections were sampled, including the cross-section with the largest diameter.
All available hematoxylin and eosin-stained slides of surgically resected primary tumor specimens were retrieved from the Department of Pathology at SNUBH. The slides were digitized using an Aperio GT 450 DX scanner (Leica Biosystems) at 40× magnification, with a resolution of 0.26 µm/pixel. Three observers with varying levels of expertise independently reviewed the whole-slide images (WSIs) using QuPath ver. 0.4.3: observer A, a medical doctor with 6 months of training; observer B, a junior pathologist; observer C, a senior pathologist specializing in lung pathology.
The evaluation of the tumor bed and quantification of VT (%) within WSIs adhered to the IASLC recommendations [13]. The “tumor bed” was defined as the area presumed to have originally harbored the tumor before neoadjuvant therapy, comprising three components: VT, necrosis, and stroma, collectively summing to 100%. Using QuPath’s annotation tools, observers A and B independently delineated the tumor bed borders on WSIs. All three observers, blinded to clinical information, independently assessed the VT (%), necrosis (%), and stroma (%) for each slide. Tumor cells with intact morphology were quantified as VT (%), while intratumoral stroma, such as papillary cores, was included in the stroma component. Assessments were performed in 1% increments to enable precise quantification.
Two approaches were employed to calculate VT (%) for each case: wVT (%) and unweighted VT (uwVT) (%). The unweighted method straightforwardly averaged the VT (%) values across all slides within a given case. In contrast, the area-weighted method adjusted the VT (%) based on the tumor bed area of each slide, aligning with the approach proposed by the MPR calculation tool [14]. The tumor bed areas used for wVT (%) calculation were determined through discussions among all three observers, based on the annotations of observers A and B. The average wVT (%) rated by the three observers was used to determine MPR status and the optimal cutoff.
3. Statistical analysis
All statistical analyses were performed using R software ver. 4.2.3 (R Foundation for Statistical Computing). The Wilcoxon rank-sum test was employed to examine the relationship between clinical and histopathologic features for continuous variables, while Pearson’s chi-square test or Fisher’s exact test was used for categorical variables. Spearman’s correlation coefficient was utilized to examine the relationship between two continuous variables.
To evaluate the concordance of tumor bed area delineations between observers A and B, the Dice coefficient and intersection-over-union (IoU) score were calculated. Denoting the tumor bed area annotations by observers A and B as A and B, respectively, the scores were computed as follows:
Where |A ∩ B| denotes the intersection, representing the overlapping region between the two annotations A and B, and |A ∪ B| denotes the union, encompassing the combined area covered by both annotations A and B.
The two-way mixed intraclass correlation coefficient (ICC) for absolute agreement was employed to determine the interobserver reproducibility of VT (%) assessments among the three observers. Fleiss’ kappa statistic was utilized to evaluate the interobserver agreement for MPR status. The Kaplan-Meier method was used to estimate survival rates over time, with differences between groups assessed by the log-rank test. In multivariate analyses using Cox proportional hazard regression models, factors exhibiting significance (p < 0.1) in univariate analyses were included, excluding those significantly associated with MPR status (p < 0.05). To identify the optimal VT (%) cutoff for survival prediction, maximally selected log-rank statistics implemented in the R package maxstat were used. Statistical significance was defined as a two-sided p-value < 0.05.
Results
1. Clinical and histopathologic characteristics
The study cohort consisted of 108 patients, with 86 males (79.2%) and 22 females (20.4%), and a median age of 62 years. The histologic subtypes included 64 patients (59.3%) with ADC, and 38 (35.2%) patients with squamous cell carcinoma (SqCC). Regarding neoadjuvant therapy, 95 patients (88.0%) received chemotherapy, while 13 patients (12.0%) underwent concurrent chemoradiotherapy. All patients received platinum-based neoadjuvant regimens, with 80 patients (78.4%) treated with a regimen containing gemcitabine. The median number of neoadjuvant cycles was three. The median time from the end of neoadjuvant treatment to surgery was 28 days (range, 8 to 60 days).
A total of 303 WSIs were reviewed, with an average of 2.81 WSIs per case (range, 1 to 11) (S2 Fig.). Fig. 1 presents representative slide images illustrating tumor bed annotations and the evaluation of VT, necrosis, and stroma components.
Eighteen patients (16.7%) were identified as MPR-positive (MPR+) using the 10% VT (%) cutoff, and a pathological complete response was observed in six (5.6%). Clinical and histopathologic information regarding MPR status is presented in Table 1. MPR+ group exhibited more favorable pathologic features compared to the MPR-negative (MPR–) group, including lower ypStage (p < 0.001), higher frequency of tumor size reduction (≥ 30%) (p < 0.001), more frequent nodal downstaging (p=0.008), reduced pleural invasion (p=0.008), reduced lymphovascular invasion (p < 0.001), reduced perineural invasion (p=0.039), and reduced tumor spread through air spaces (p < 0.001). However, there was no significant difference in the cStage between the MPR+ and MPR– groups (p=0.312), indicating that the tumor burden at baseline was comparable.
SqCC exhibited a significantly lower median VT (%) of 28% compared to 52% in ADC (p < 0.001) (S3 Fig.), resulting in a higher frequency of MPR+ in SqCC (26% vs. 11% in ADC, p=0.044). Additionally, SqCC cases demonstrated a lower ypStage (p=0.017), a higher rate of tumor size reduction (≥ 30%) (p=0.003), more frequent nodal downstaging (p=0.001), reduced pleural invasion (p < 0.001), and less tumor spread through air spaces (p=0.013) (S4 Table). EFS was also improved in SqCC (p=0.015) (S5 Fig.). However, no significant differences were observed between SqCC and ADC in terms of the initial cStage (p=0.387), type of neoadjuvant therapy (p=0.527), neoadjuvant regimen (p=0.216), or number of neoadjuvant cycle (p=0.151).
2. Interobserver agreement on tumor bed evaluation
The size of tumor bed areas evaluated by observers A and B showed median values of 1.32 cm2 (interquartile range [IQR], 0.66 to 2.11) and 1.21 cm2 (IQR, 0.64 to 1.97), respectively. A high level of concordance in tumor bed evaluation was observed, with a median Dice coefficient of 0.96 (IQR, 0.91 to 0.98) and an IoU score of 0.92 (IQR, 0.83 to 0.95) (S6 Fig.). There were no changes in the MPR status in any of the cases when wVT (%) was calculated solely based on the annotation of observer A or B.
The interobserver reproducibility of the tumor bed was related to the size of the tumor bed and histologic subtype (Fig. 2). The tumor bed area (cm2) was strongly correlated with the Dice coefficient (rs=0.4215, p < 0.001) and the IoU score (rs=0.4212, p < 0.001), indicating a positive relationship across the entire set of slides. The Spearman correlation coefficient was higher for SqCC (Dice coefficient rs=0.4963; IoU score rs=0.4957) than for ADC (Dice coefficient rs=0.3924; IoU score rs=0.3923), suggesting that changes in tumor bed size are more sensitively reflected in tumor bed interobserver reproducibility in SqCC. Meanwhile, in terms of histologic subtype, although the tumor bed area (cm2) did not exhibit a significant difference between ADC and SqCC (p=0.484), SqCC showed a significantly lower median Dice coefficient (0.93 vs. 0.97, p < 0.001) and IoU score (0.88 vs. 0.93, p < 0.001) than ADC. In summary, SqCCs tended to display lower tumor bed reproducibility scores, particularly when the tumor bed size was small.
For the Dice coefficients, the threshold defining lower-bound outliers were set to 0.805 according to the quartile value and IQR. Of the 303 slides, 28 (9.2%) had Dice coefficients below the threshold. When compared to non-outliers, these lower-bound outliers were predominantly SqCC (75% vs. 34%, p < 0.001), exhibited smaller tumor bed sizes (median, 0.83 cm2 vs. 1.36 cm2; p < 0.001), and had lower VT (%) values (median, 34% vs. 50%; p=0.005).
When reviewing the slides of lower-bound outliers, we identified common histological characteristics that posed challenges in tumor bed demarcation (Fig. 3). In SqCC, we observed void spaces within airway structures and/or adventitial tissue of bronchovascular bundles merging with the tumor bed, along with visual dispersion of tumor bed. For mucinous ADC, difficulties arouse in determining the tumor bed border within mucin. Extensive reactive changes surrounding the tumor bed also complicated the evaluation.
3. Interobserver agreement on VT (%) and MPR
The assessments of VT (%) and MPR assessments by the three observers demonstrated excellent concordance, with an ICC of 0.959 (95% confidence interval [CI], 0.913 to 0.978) and Fleiss’ kappa of 0.911 (95% CI, 0.86 to 0.96). There was unanimous consensus on MPR in 104 cases (96.3%). S7 Fig. presents a Bland-Altman plot illustrating the deviation of each observer’s VT (%) estimation from the mean VT (%).
The level of agreement in VT (%) assessment was influenced by the VT (%) value itself. The ICC varied depending on the VT (%) range. For cases with VT ≤ 20% or > 80% (n=36), the concordance was higher (ICC=0.996; 95% CI, 0.991 to 0.998) compared to the cases with VT (%) between 20% and 80% (n=72) (ICC=0.856; 95% CI, 0.632 to 0.930), indicating a higher level of agreement at the extremes of the percentages. Additionally, the standard deviation of the estimated VT (%) was significantly correlated with the VT (%) value. A scatter plot of modified VT (%), applying [VT] if VT < 50% and [100–VT] if VT ≥ 50%, along with the standard deviation, is shown in Fig. 4, with a Spearman correlation coefficient of 0.6550 (p < 0.001). This suggests that as VT (%) approaches 50%, the deviation tends to increase in magnitude.
Furthermore, we annotated the VT in QuPath on one ADC slide and one SqCC slide that exhibited the lowest agreement in VT (%) ratings. Each observer’s VT (%) ratings and the obtained VT (%) based on the annotated area are shown in S8 Fig.
4. Comparison of wVT (%) and uwVT (%)
We examined 81 cases with two or more available slides to compare the area-weighted and unweighted method for calculating VT (%) per patient case. A scatter plot comparing wVT (%) and uwVT (%) is presented in S9 Fig. The two methods exhibited an excellent level of agreement (ICC=0.973; 95% CI, 0.955 to 0.984) among these cases. The determination of MPR status using wVT (%) and uwVT (%) also showed a high level of agreement (kappa=0.95; 95% CI, 0.84 to 1), with only one case out of 81 exhibiting a differing MPR status. Based on the wVT (%) assessment, 11 patients (13.6%) were identified as MPR+, while one patient transitioned to MPR– in the uwVT (%) assessment, resulting in 10 patients (12.3%) classified as MPR+.
5. Optimal VT (%) cutoff and clinical significance of MPR
The optimal VT (%) cutoff values for OS (13.2%), EFS (7.7%), and LCS (13.2%) in the entire cohort were consistent with the 10% cutoff value (S10 Fig.). Within the ADC subgroup, the cutoffs for OS, EFS, and LCS were 7.7%, 15.2%, and 7.7%, respectively. In the SqCC subgroup, the corresponding cutoffs were 13.2%, 0.3%, and 13.2%, respectively.
Notably, a prominent peak was observed around 65% VT (%) in the standardized log-rank statistics, suggesting 65% as a potential candidate cutoff. In the entire cohort, the peak values were 64.1% for OS and LCS. In the ADC, the corresponding values were 65.4% for OS and 61.7% for LCS. However, when assessing the survival curves of the two groups categorized by the 10% and 65% cutoffs in ADC, it was evident that the 65% cutoff did not differentiate the dichotomized groups more effectively than the 10% cutoff (OS: p=0.222 vs. p=0.211; EFS: p=0.051 vs. p=0.008; LCS: p=0.364 vs. p=0.111) (S11 Fig.).
Higher VT (%) was associated with increased 3-year and 5-year cumulative overall mortality, cumulative event incidence, and lung cancer-specific cumulative mortality, as illustrated in S12 Fig. In the Kaplan-Meier estimates, MPR+ status, defined by a 10% cutoff, was significantly associated with improved OS (p=0.018), EFS (p < 0.001), and LCS (p=0.004) compared to MPR– status (S13 Fig.). In the multivariate analysis (Table 2), MPR+ status was an independent favorable prognostic factor for OS (hazard ratio [HR], 0.37; 95% CI, 0.16 to 0.87; p=0.023), EFS (HR, 0.25; 95% CI, 0.11 to 0.57; p=0.001), and LCS (HR, 0.16; 95% CI, 0.04 to 0.68; p=0.012). The results of univariable analysis are available in S14 Table.
Discussion
In this study, we analyzed the reproducibility of tumor bed and VT (%) assessments involving three observers; explored the implications of wVT and uwVT (%) in MPR determination; and identified optimal cutoff values for VT (%) in ADC and SqCC. The strength of this study lies in the quantitative analysis of reproducibility utilizing digital WSIs and in identifying histologic factors that influence this reproducibility.
In terms of tumor bed concordance, we found that reliable MPR determination can be achieved under standard conditions. The MPR status remained unchanged in all cases when using either observer A’s or B’s tumor bed to obtain wVT (%). An MPR reproducibility study conducted by the IASLC [19], although without extensive quantitative analysis, also reported that challenging situations in tumor bed identification had minimal impact on the calculation of the MPR.
However, we suggest closer attention when assessing tumor beds in SqCC with small size. In this study, we found that histologic subtypes and tumor bed size influenced the reproducibility of the tumor bed assessments. The disparity in concordance by histologic subtype can be explained by distinct location patterns of ADC and SqCC. ADC typically presents as a peripheral lung parenchymal mass, while SqCC arises from the bronchial epithelium, developing as a central mass near the hilum. Therefore, in SqCC, tumor beds are often situated near or intertwined with fibrotic bronchovascular bundles, airway spaces, or post-obstructive alterations, introducing complexity in tumor bed evaluation. A similar situation was mentioned by Saqi et al. [14] who noted the difficulty in cases of hilar tumors with minimal VTs, where the adventitia surrounding the bronchi and blood vessels can appear close to the tumor bed. Regarding tumor bed size, minor discrepancies in annotations of small tumor beds could resulted in relatively significant variations in concordance, explaining the positive correlation between the concordance of the tumor bed and its size. This trend was more pronounced in SqCC as well.
We observed that the reproducibility of VT (%) assessments was particularly high when the values were at extreme percentages (≤ 20% or > 80%). This finding aligns with the IASLC reproducibility study [19], which reported higher concordance in VT (%) ratings at both extremes. During the manual annotation of the two WSIs with the lowest VT (%) reproducibility, we noted that numerous small VT areas intermingled with stroma led to overestimation of VT (%) at low magnification, consequently decreasing concordance. Additionally, the classification of empty spaces encircled by tumor cells, as seen in acinar patterns of ADC, introduced variability in VT (%) assessment among observers. As per the IASLC recommendations [13] and the suggestion by Saqi et al. [14], VT should consist solely of well-preserved tumor cells, excluding intratumoral stroma-like papillary cores. However, precisely distinguishing tumor cells from stroma or tumor-encircled spaces during slide examination can be intricate and prone to observer variability, especially when these patterns occupy a significant portion of the tumor bed.
Integrating artificial intelligence (AI) algorithm trained to accurately segment and quantify VT could mitigate these challenges. A previous study by Dacic et al. [20] found a strong correlation and systemic difference between AI-measured and human-assessed pathologic responses, suggesting AI may perform better in differentiating tumor cells from intratumoral components. Therefore, incorporating AI support could aid in precise VT quantification and determine the clinical significance of including or excluding delicate intratumoral stroma or tumor-encircling spaces in the VT calculation.
The wVT (%) calculation method introduced by Saqi et al. [14] involves visualizing a rectangle that matches the tumor bed area and accounting for its width and length. In our study, we utilized the annotated and their automatically calculated tumor bed areas in QuPath for the wVT (%) assignment process. When comparing wVT (%) and uwVT (%), we found an excellent level of agreement between the two methods, aligning with the findings from the IASLC reproducibility study [19] that reported these two approaches as mutually replaceable.
However, before integrating the unweighted method into routine MPR assessment practices across institutions, it is crucial to establish evidence demonstrating its effectiveness in representing the area-weighted method. This is because the discrepancy between wVT (%) and uwVT (%) arises during the allocation of tumor bed areas to each slide, a practice that may vary across institutions. After providing adequate validation, the area-weighted method could be employed only in specific scenarios, such as when VT (%) is approximately 10%, thereby avoiding the cumbersome estimation of tumor bed area for every slide.
A higher VT (%) cutoff of 65% was proposed for ADC instead of the conventional 10% [15-17], based on the study results exhibiting a favorable neoadjuvant response in SqCC than ADC. SqCC was reported to be associated with a higher likelihood of achieving MPR [15,17,21,22] and improved EFS [23] compared to ADC, which is consistent with the results obtained in this study. However, in terms of the optimal cutoff, our findings strongly supported the well-established 10% cutoff across histologic subtypes. MPR+ with a 10% cutoff was a robust prognostic factor for improved OS, EFS, and LCS after adjusting for other clinical and histologic factors. High reproducibility at the extremes of VT (%) values further ensures reliability in the evaluation of MPR with 10% cutoff across multiple observers.
The primary limitation of this study was its retrospective, single-center analysis. Caution is needed when interpreting the results, given the study cohort with two different types of neoadjuvant therapy – chemotherapy alone or chemoradiation. While chemoradiation-induced pathologic response is recognized to be associated with survival [24-26], differences have been reported in the rate of achieving pathologic response [27,28] and the degree to which pathologic response reflects long-term survival [29,30] between the two modalities. In our study, MPR+ rates were 10.5% (10 out of 95) and 61.5% (8 out of 13), respectively, in patients who received chemotherapy alone and those who received chemoradiation, with 44% of the MPR+ group being treated with chemoradiation. Although MPR proved to be a highly effective predictor of survival in the entire cohort in this study, the limited number of patients made it challenging to compare the clinical implications of MPR in these two groups.
In conclusion, this study investigated factors influencing the reproducibility of MPR assessment and established optimal VT (%) cutoff values for ADC and SqCC. Tumor bed reproducibility remains robust with minimal effect on MPR, but attention is warranted when evaluating smaller tumor beds in SqCC. Standardizing the 10% cutoff in both ADC and SqCC provides notable advantages in terms of survival predictions and reliable assessments.
XML Download