PDF
Citation
Print
In 2022, thyroid neoplasm ranked as the seventh most prevalent cancer worldwide, with approximately 80%–90% of cases being papillary thyroid carcinoma (PTC) [1,2]. Histologically, PTC is a well-differentiated thyroid malignancy originating from follicular cells of the thyroid gland. The incidence of PTC has increased over recent decades [3,4]; however, the reasons for this increase have been the subject of considerable debate among global experts [5]. The major risk factors for PTC include ionizing radiation and obesity, the incidences of which have also continued to increase within the last decade [1]. However, the increasing attention to thyroid examination may also lead to greater detection of subclinical papillary carcinoma, as seen in developed countries such as South Korea and the United States [1]. Despite the increased incidence, mortality rates have remained unchanged, suggesting a favorable prognosis with a 10-year cause-specific survival rate in 73% of patients [3,4].
Establishing the diagnosis of PTC involves assessment of the gross pathology and histology of the thyroid gland. The gross appearance of the tumor may manifest as either solid nodules or cystic structures within the thyroid gland. Definitive diagnosis of PTC requires microscopic identification of papillary, follicular, and/or solid architectural structures, specific nuclear features, and the determination of the infiltrative pattern of the tumor [2]. Under the current World Health Organization (WHO) classification, various tumor diagnoses are encompassed under ‘encapsulated follicular-patterned thyroid lesions’ depending on the nuclear features and capsular and vascular invasion [2]. Reproducibility, particularly concerning the nuclear features of PTC and the presence of capsular and vascular invasion, was a significant concern among experts. Previous studies have shown considerable variation among pathologists when diagnosing the nuclear features of PTC, especially in tumors showing encapsulated follicular patterns [6-10]. Other studies have also highlighted challenges in the evaluation of thyroid tumor capsular invasion and minimal extrathyroidal invasion [11,12]. This variability encompasses both interobserver variability, which refers to differences between experts, and intra-observer variability, which pertains to variations within the same expert. The absence of well-defined evidence-based diagnostic criteria is the primary reason for this variability.
Various nuclear features have been utilized to diagnose PTC, including identification of tumor cells with high nuclear-to-cytoplasm ratio; nuclear enlargement; nuclear overlap, crowding, and elongation; irregular nuclear contours; intranuclear cytoplasmic inclusions; chromatin clearing; and multiple nucleoli located near the cell membrane [7]. To reduce variation in histological assessments, the WHO has adopted the three-point scoring system originally proposed by Nikiforov et al. [13] to evaluate nuclear features of PTC with low subjectivity. The scoring system consists of three measures: cell nucleus size and shape (enlarged, elongated, and crowded), nuclear membrane features (irregular nuclear contour, nuclear grooves, and nuclear pseudoinclusion), and chromatin characteristics (clear chromatin, chromatin margination to the cell membrane, and glassy nuclei). Each criterion contributes a score of 1, resulting in a total nuclear score range of 0–3. A nuclear score of 0–1 does not meet the criteria for typical PTC, whereas a score of 2–3 is considered typical. In 2018, the Asian Thyroid Working Group proposed a more detailed eight-point scoring system to evaluate the nuclear features of PTC [6]. In a study by Jung et al. [14], the eight-point scoring system was able to distinguish BRAF-like tumors from RAS-like tumors based on nuclear pseudo-inclusion and a high nuclear score.
Indonesia comprises more than 17,000 widely dispersed islands, which are organized into 38 provinces. Presently, Indonesia is home to approximately 860 pathologists who exhibit a wide array of educational backgrounds and professional experience. The presence of various medical education centers in Indonesia may also contribute to a higher likelihood of interobserver disagreement in the diagnosis of certain diseases.
To date, there have been no interobserver studies aimed at evaluating the extent of concordance among Indonesian pathologists in the assessment of the nuclear features and capsular and vascular invasion of PTC. The primary objective of this study was to assess the level of agreement among pathologists from various centers in Indonesia regarding the diagnosis of nuclear features as well as capsular/vascular invasion and the presumed molecular type of PTC cases.
MATERIALS AND METHODS
Study design
We conducted an interobserver agreement study in which 15 pathologists from multiple educational/hospital centers in Indonesia were recruited to perform an independent histology assessment of thyroid neoplasm cases. As depicted in Fig. 1, the participating pathologists were from Padang, Palembang, Bandar Lampung, Jambi, DKI Jakarta, Bandung, Yogyakarta, Surabaya, Semarang, Malang, Bali, Balikpapan, Makassar, Kupang, and South Halmahera.
This study consists of two rounds of histological assessment with two online training sessions in between. For histological assessment, the pathologists received digital slides of thyroid tumor specimens along with a questionnaire in which to record their histology conclusions. Basic demographic data about the pathologist’s age; sex; numbers of years in pathology practice, in a teaching hospital, and/or in thyroid consultancy; number of thyroid cases diagnosed per year; history of study abroad; the pathologist’s experience in diagnosing non-invasive follicular thyroid neoplasm with papillary-like nuclear features (NIFTP) in pathology practice; the routine acquisition of clinical and radiology information; and the pathologist’s prior experience with digital pathology slides were included in the initial questionnaire during the first round. In addition, pathology data including the method used for tumor capsular sectioning and the number of blocks submitted for a 4-cm thyroid nodule were also analyzed.
After the first round, all pathologists received a learning module (Supplementary Materials) and attended two online training sessions directed by the authors. To evaluate the impact of the training, a second-round assessment was performed two months after the first.
Digital slides of tumor specimens
The tumor specimens used in this study were thyroid neoplasms collected from Dr. Cipto Mangunkusumo Hospital, Jakarta, Indonesia. The histologic slides were stained with hematoxylin and eosin and were scanned using Aperio software (Leica Biosystems, Buffalo Grove, IL, USA) and converted into digital slides accessible for online viewing. Both rounds of assessment consisted of 14 cases of thyroid neoplasms. The second round comprised seven cases previously examined in the first round and seven new cases. We randomly mixed the sample cases in the second round to prevent memory bias. We determined the mutational status of BRAFV600E and RAS for all specimens following a previous procedure [15]. Seven of each BRAFV600E and RAS mutations were identified in each round.
This study evaluated the pathologists’ diagnoses of thyroid neoplasm, particularly their evaluation of the nuclear features of PTC. We asked each pathologist to assess the nuclear features based on the two nuclear scoring systems. As noted above, the three-point scoring system developed by Nikiforov et al. [13] encompasses three key parameters of nuclear size and shape, membrane abnormalities, and chromatin characteristics (Table 1). The evaluation of nuclear features was enhanced by a comprehensive eight-point scoring system modified from the schemes of Liu et al. [6] and Jung et al. [14]. The eight-point scoring system used in the current study encompassed the assessment of nuclear enlargement, nuclear crowding, nuclear elongation, irregular nuclear contour, nuclear grooves, nuclear pseudo-inclusion, chromatin clearing, and PTC-nuclear feature distribution. Specific scores were assigned to each of these features, as outlined in Table 2. The nuclear examination was conducted using adjacent benign follicular thyroid tissue as a comparative reference.
In addition to the evaluation of nuclear features, assessment of capsular and vascular invasion (categorized as invasive, non-invasive, or equivocal), molecular status (classified as BRAF-like or RAS-like), and whether the case was straightforward or doubtful was also completed.
Statistical analyses
The pathologists’ demographic information was presented in the form of categorical data, displayed as frequencies and percentages. The levels of interobserver agreement in the first and second rounds were analyzed using Fleiss kappa statistical analysis. The interpretation of kappa’s statistical result was as follows: 0.01–0.20, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81–0.99, almost perfect agreement. A kappa value of 1 indicated perfect agreement, while values <0 indicated pure chance alone. A stratified analysis based on endocrine expertise was also performed. All statistical analyses were performed using SPSS software ver. 20 (IBM Corp., Armonk, NY, USA).
RESULTS
The average age of pathologists in this study was 46 years, and the majority was female. Most participating pathologists were based in teaching hospitals, with an average of 10 years of practical experience. Nine respondents were pathologists who had passed national board examinations with a subspecialty in endocrinology. The number of thyroid cases diagnosed by the respondents ranged from 21 to 1,200 per year, with a median of 110 cases and interquartile range of 50–217 cases (Table 3). The tumor sampling techniques varied between the pathologists; some pathologists sectioned the entire capsule of follicular-patterned tumors, while all other pathologists sampled <8 blocks. Most of the pathologists were familiar with digital pathology.
The sample cases exhibited varying degrees of ambiguity (Fig. 2), which consequently influenced the interobserver agreement. The greatest level of agreement among pathologists was reported for a case with a positive RAS mutation, for which most pathologists provided a total nuclear score of 1. The assessment for other histological parameters such as capsular invasion also varied between the pathologists.
Pathologist agreement regarding the assessment of nuclear features
In the first round of assessments using the three-point system, the Fleiss kappa values were as follows: nuclear size and shape (κ=0.14), chromatin features (κ=0.21), and membrane irregularities (κ=0.28). These findings indicated slight to fair agreement among pathologists when evaluating each nuclear feature. When evaluating the total score for nuclear features using the three-point system, only slight agreement between the pathologists assessments was evident (κ=0.17). Using the eight-point nuclear scoring system, the Fleiss kappa values represented poor to fair agreement (Table 4), with the highest kappa values obtained for the assessment of nuclear pseudo-inclusion (κ=0.24) and nuclear elongation (0.20). The lowest kappa value was obtained for the assessment of PTC nuclear features distribution (κ=–0.02) and nuclear crowding/overlapping (κ=0.01). The concordance for the total score of nuclear features based on the eight-point system was poor (κ=–0.01).
Although not statistically significant (p=.052), there was an improvement in the average kappa agreement of the three-point nuclear score system following the training (Table 5). In the second (post-training) round, the pathologists displayed fair to moderate agreement when evaluating the nuclear size and shape (κ=0.30), membrane irregularities (κ=0.57), and chromatin features (κ=0.59). When evaluating the total score of nuclear features based on the three-point system, the pathologists displayed a fair level of agreement (κ=0.39). Furthermore, there was a significant improvement in the agreement when using the eight-point nuclear scoring system (p=.001). The pathologists displayed fair agreement when assessing most nuclear features and moderate agreement when evaluating irregularities of the membrane contour (κ=0.48). The only feature with slight agreement was nuclear grooves (κ=0.19). The agreement for the total nuclear score based on the eight-point system was slight (κ=0.05). As displayed in Fig. 3, total nuclear scores for both the three- and eight-point systems were lower in the second round than in the first round.
The study also compared the assessments of the 15 pathologists in the two rounds of slide assessments (Fig. 4). Four pathologists (Nos. 4, 7, 9, and 15) demonstrated uniformity in their average nuclear scores with the three-point assessment between the two rounds. Upon analysis of the scores obtained using the eight-point scoring system, two pathologists (Nos. 2 and 7) demonstrated marked differences in nuclear scoring assessment compared to the initial round, while only three pathologists (Nos. 4, 9, and 13) showed minimal alterations in the evaluative outcomes between the two rounds.
Stratified analysis based on the expertise of pathologists
Using the three-point system, the interobserver agreement for nuclear features was higher among endocrine experts compared to non-endocrine experts (Fig. 5). The greatest agreement among the endocrine experts was obtained when assessing chromatin features (κ=0.52), followed by membrane irregularities (κ=0.27) and nuclear size and shape (κ=0.22). In comparison, non-endocrine experts displayed fair agreement for membrane irregularities (κ=0.21) and slight agreement for both chromatin features (κ=0.10) and nuclear size and shape (κ=0.08). The endocrine experts displayed fair agreement for the total nuclear score of the three-point system (κ=0.26) relative to the slight agreement seen among the non-experts (κ=0.11).
Using the eight-point nuclear score system, both the endocrine and non-endocrine experts showed variation in their level of concordance (Fig. 5). The endocrine experts displayed better agreement regarding nuclear pseudo-inclusion (κ=0.31) relative to the non-endocrine experts (κ=0.18). However, the non-endocrine experts displayed more concordance when assessing other nuclear features, such as nuclear elongation, nuclear enlargement, irregular membrane contour, and chromatin clearing.
After the training, there was significant improvement in the agreement between the endocrine and non-endocrine experts when assessing nuclear features (Fig. 6). In the second (post-training) round, the experts displayed moderate agreement when assessing each nuclear feature using the three-point system: κ=0.52 for membrane irregularities; κ=0.44 for nuclear size and shape; and κ=0.44 for chromatin features. Furthermore, the non-endocrine experts displayed substantial agreement when assessing chromatin features (κ=0.69), moderate agreement for membrane irregularities (κ=0.57), and fair agreement for nuclear size and shape (κ=0.20). Both the endocrine experts (κ=0.34) and non-experts (κ=0.39) reached fair agreement when assessing the total nuclear score using the three-point system. Using the eight-point system, the agreement level showed considerable variation, as displayed in Fig. 6. The endocrine experts showed better agreement when assessing nuclear elongation, nuclear crowding/overlapping, nuclear pseudo-inclusion, and chromatin clearing compared to the non-experts.
Pathologists’ agreement for other histological parameters
This study also analyzed interobserver agreement for the assessment of additional histological parameters, including the presumed molecular type and the presence of vascular and/or capsular invasion (Table 6). The kappa statistics indicated substantial agreement for all pathologists (κ=0.74) regarding the presumed genetic type, with an almost perfect agreement level among endocrine experts. The accuracy of the molecular type identification ranged from 67%–100%, with 10 pathologists assigning 100% correct answers. The assessment of vascular invasion reached fair agreement among all pathologists (κ=0.35). A stratified analysis displayed better agreement (k=0.53) for the assessment of vascular invasion among the non-endocrine experts compared to the experts. The capsular invasion agreement was fair among all groups.
DISCUSSION
Obtaining consistency in thyroid neoplasm diagnosis presents a challenge to pathologists across the globe. Variation between assessments performed by individual pathologists may arise due to the application of different standard criteria in different geographic regions [9,10]. Pathologists in developed countries often employ ancillary tools such as immunohistochemistry and molecular examinations to assist in the establishment of a diagnosis. Although such methods are widely accepted, it is crucial to understand that the final diagnosis of thyroid neoplasm relies on the assessment of morphological features [16]. Our study confirmed that pathologists in Indonesia, influenced by their educational background, experiences, and viewpoints, demonstrated only moderate agreement when applying the standardized diagnostic criteria. We assessed concordance among pathologists from several cities in Indonesia with varied educational backgrounds and working environments when assessing the histopathological features of thyroid neoplasms. The discordances highlight the need for clear and uniform standards in pathological evaluation to reduce variability in interpretations.
The nuclear features of thyroid neoplasms range widely from nuclei with minimal atypia to prominent nuclear changes. The heterogeneity of the nuclear morphology is due in part to the presence of molecular alterations such as BRAFV600E and RAS mutations. Tumors with BRAFV600E mutations have more prominent PTC nuclear features, while those with RAS mutation show less noticeable nuclear features [14]. Additional genetic alterations may also contribute to the typical morphological features of PTC, including translocation of the NTRK gene, THADA-IGF2BP3 gene fusion, and DICER1 mutation [17-19]. The three-point nuclear scoring system is currently used by experts to diagnose NIFTP and papillary carcinoma. A total nuclear score of 0–1 favors a diagnosis of benign tumors while a total nuclear score of 2–3 favors NIFTP or carcinoma. This study found only slight to fair baseline concordance among Indonesian pathologists when assessing nuclear features using the three-point system. This result indicated lower concordance than was reported in an earlier validation study of the three-point nuclear system that showed good to substantial concordance among pathologists across California, the UK, and Japan [7]. Nonetheless, consistent with this earlier study, nuclear size and shape were the criteria with the lowest concordance, whereas membrane irregularity was the criterion with the highest agreement level [7].
Although the three-point system acts to standardize the diagnosis of nuclear features in thyroid tumors, continuous refinement is needed to accommodate the complexity of pathological features while ensuring an accurate and consistent diagnosis [7]. We also evaluated concordance among pathologists when evaluating nuclear features using the eight-point system. The baseline concordance among pathologists ranged widely from poor to fair. The pathologists demonstrated the greatest agreement for nuclear pseudo-inclusion. The three-point scoring system demonstrated better agreement than the eight-point scoring system, suggesting that its simplicity may facilitate more consistent assessments.
The training was significantly effective in enhancing the level of agreement among participating pathologists, particularly when using the eight-point nuclear scoring system. Following the training, moderate agreement was obtained for assessment of membrane irregularities, chromatin features, and irregular membrane contours. Interestingly, the feature of nuclear grooves yielded the greatest discordance, even after training. This finding contrasts with the results from a study by Elsheikh et al. [9], which identified nuclear grooves as having the second-highest level of agreement after nuclear clearing. Despite recent enhancements, the concordance achieved with the scoring system remains suboptimal, potentially attributable to variability in the experience of and thyroid case volumes among pathologists. The study conducted by Thompson et al. [7] demonstrated a minor trend in which pathologists practicing in the UK exhibited slightly lower accuracy than those in California and Japan.
The comparative analysis of performance scores for the 15 pathologists across two evaluation rounds provided insights into the dynamic nature of diagnostic assessments. This analysis of this line chart highlighted significant fluctuations in individual pathologist scores within each round, suggesting variability in the application of the scoring systems. These observations underscore the complexities of achieving uniformity in pathological assessment and highlight the need for ongoing training and calibration among pathologists to enhance the reliability of the processes used to diagnose thyroid neoplasms.
Another important finding was the correlation between the clinician’s area of expertise and the level of diagnostic agreement. Our findings suggest that endocrine experts often reach consensus on cellular characteristics, while non-endocrine experts demonstrate consistency in other aspects. According to Farmer et al. [20], expert input often significantly alters the initial diagnosis, with a very low kappa coefficient indicating substantial differences between assessments by expert and non-expert pathologists. Kerkhof et al. [21] reported moderate agreement among pathologists with varying levels of expertise when evaluating dysplasia, with greater concordance achieved when assessing more advanced cases. Notably, Thompson et al. [7] found that a uniform scoring system fosters agreement among pathologists in different subspecialties when diagnosing papillary carcinoma. Together, these findings suggest that, while expertise is irreplaceable, particularly in complex cases, the definition and adoption of clear, structured diagnostic criteria may provide common ground upon which to establish diagnostic objectivity and clarity, and interobserver agreement should improve.
Additional histological features contributing to the diagnostic challenges presented by PTC include capsular and/or vascular invasion. The categories of capsular invasion span non-invasive, questionable invasion, and unequivocally invasive forms. Zhu et al. [11] reported fair agreement among thyroid pathologists for non-invasive and invasive tumors, whereas questionable invasion elicited poor agreement, indicating that borderline tumors pose a significant challenge in the practice of thyroid pathology. The present study, which involved the evaluation of a mixture of encapsulated and invasive cases, resulted in fair agreement overall in determining capsular invasion. The identification of vascular invasion holds critical importance not only in establishing a definitive diagnosis, but also in prognostication and assessing the aggressiveness of the disease [22]. This study demonstrated fair to moderate agreement among pathologists when evaluating vascular invasion. The assessment of vascular invasion is complicated by several factors, including the presence of clefts that mimic vessels and artifacts related to floating tumor cells. Some immunohistochemistry markers (such as CD 31 or ERG) might help to evaluate this feature more robustly but were not used in this study.
The potential to utilize histological assessment to predict genetic alteration of thyroid tumors has been emphasized by several experts [15,19]. In the present study, the agreement among pathologists regarding the identification of tumors with BRAFV600E or RAS mutations ranged from substantial to nearly perfect. Ten pathologists involved in this study demonstrated 100% accuracy. The three-point scoring system exhibited 85%–94% accuracy for prediction of molecular alterations in NIFTP [19]. Notably, nuclear pseudo-inclusion is believed to be a distinctive feature of BRAFV600E-mutated PTC.
The major limitation of this study was the heterogeneous background and broad experience among pathologists. The small number of pathologists limited the generalizability of the findings to the large population of Indonesian pathologists.
Following the training, the overall agreement among pathologists when assessing nuclear features using the three- and eight-point scoring systems was slight to moderate. There was fair agreement regarding the identification of capsular invasion, whereas fair to moderate agreement was achieved for vascular invasion. The level of agreement was greatest for presumed genetic type. Despite enhancements to the scoring systems, the level of agreement attained in this study remained below optimal, possibly due to the diverse backgrounds of the individual pathologists. The pursuit of national standardization in pathological assessment, and especially with respect to the evaluation of nuclear features, is an ongoing effort that requires collaboration among pathologists to refine the applicable criteria, enhance training, and ultimately improve the reliability and validity of diagnostic processes in the field of thyroid pathology.
XML Download