Breast Ultrasound Computer-Aided Diagnosis: Analysis of Types of Errors

Min Kyung Jeong; Bong Joo Kang; Eunjeong Kim; Sung Hun Kim

doi:10.3348/jksr.2018.79.3.114

Journal List > J Korean Soc Radiol > v.79(3) > 1099989

Go to TopGo to Top Go to BottomGo to Bottom

TOOLS

Jeong, Kang, Kim, and Kim: Breast Ultrasound Computer-Aided Diagnosis: Analysis of Types of Errors

Original Article

J Korean Soc Radiol 2018;79(3):114-122.

Published online: 19 January 2018

DOI: https://doi.org/10.3348/jksr.2018.79.3.114

Breast Ultrasound Computer-Aided Diagnosis: Analysis of Types of Errors

Min Kyung Jeong, MD¹, Bong Joo Kang, MD^2,, Eunjeong Kim, MD², Sung Hun Kim, MD²

¹Department of Radiology, St. Vincent's Hospital, College of Medicine, The Catholic University of Korea, Suwon, Korea

²Department of Radiology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Korea

^∗Corresponding author: Bong Joo Kang, MD Department of Radiology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, 222 Banpo-daero, Seocho-gu, Seoul 06591, Korea. Tel. 82-2-2258-6253 Fax. 82-2-599-6771 E-mail:gmlionmain@gmail.com

Received 22 November 2017 Revised 2 May 2018 Accepted 31 May 2018

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (https://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Purpose

The aim of this study was to evaluate the diagnostic performance of breast ultrasound (US) computer-aided diagnosis (CAD) to distinguish between benign and malignant lesions and analyze features of lesions interpreted with errors retrospectively.

Materials and Methods

Three hundred and sixteen women with 375 breast lesions were enrolled. We assessed the accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Additionally, we evaluated the causes and patterns of the misinterpretation in the false positive and negative groups.

Results

The accuracy, sensitivity, specificity, PPV, and NPV of breast US-CAD were 80.3%, 83.3%, 79.8%, 37.7%, and 97.0%, respectively. There were 8 false negative lesions that were oval in shape and in parallel orientation. There were 66 false positive lesions. The greatest number of errors entailed inappropriate demarcation due to heterogeneous echogenicity, etc. The second exhibited suspicious features with good demarcation and description but were confirmed as benign histologically. The third entailed a benign lesion with suspicious features, such as abscesses. The smallest portion with good demarcations and descriptions indicating benign status exhibited possible malignancy as a final conclusion.

Conclusion

Breast US-CAD is expected to be helpful in avoiding unnecessary biopsies due to its high NPV. Therefore, operators need to know the characteristics of lesions prone to misinterpretation.

Keywords: Ultrasonography, Breast Neoplasm, Diagnosis, Computer-Assisted

REFERENCES

1.Wang X., Guo Y., Wang Y. Automatic detection of regions of interest in breast ultrasound images based on local phase information. Biomed Mater Eng. 2015. 26(Suppl 1):S1265-S1273.

2.Jung KW., Won YJ., Oh CM., Kong HJ., Lee DH., Lee KH. Cancer statistics in Korea: incidence, mortality, survival, and prevalence in 2014. Cancer Res Treat. 2017. 49:292–305.

3.Chabi ML., Borget I., Ardiles R., Aboud G., Boussouar S., Vilar V, et al. Evaluation of the accuracy of a computer-aided diagnosis (CAD) system in breast ultrasound according to the radiologist's experience. Acad Radiol. 2012. 19:311–319.

4.Kim K., Song MK., Kim EK., Yoon JH. Clinical application of S-Detect to breast masses on ultrasonography: a study evaluating the diagnostic performance and agreement with a dedicated breast radiologist. Ultrasonography. 2017. 36:3–9.

5.Chen DR., Chien CL., Kuo YF. Computer-aided assessment of tumor grade for breast cancer in ultrasound images. Comput Math Methods Med. 2015. 2015:914091.

6.Moon WK., Huang YS., Lo CM., Huang CS., Bae MS., Kim WH, et al. Computer-aided diagnosis for distinguishing between triple-negative breast cancer and fibroadenomas based on ultrasound texture features. Med Phys. 2015. 42:3024–3035.

7.Song SE., Seo BK., Cho KR., Woo OH., Son GS., Kim C, et al. Computer-aided detection (CAD) system for breast MRI in assessment of local tumor extent, nodal status, and multifo-cality of invasive breast cancers: preliminary study. Cancer Imaging. 2015. 15:1.

8.Shan J., Alam SK., Garra B., Zhang Y., Ahmed T. Computer-aided diagnosis for breast ultrasound using computerized BI-RADS features and machine learning methods. Ultrasound Med Biol. 2016. 42:980–988.

9.Cho E., Kim EK., Song MK., Yoon JH. Application of computer-aided diagnosis on breast ultrasonography: evaluation of diagnostic performances and agreement of radiologists according to different levels of experience. J Ultrasound Med. 2018. 37:209–216.

10.Choi JH., Kang BJ., Baek JE., Lee HS., Kim SH. Application of computer-aided diagnosis in breast ultrasound interpretation: improvements in diagnostic performance according to reader experience. Ultrasonography. 2017. 37:217–225.

11.Lee SE., Moon JE., Rho YH., Kim EK., Yoon JH. Which supplementary imaging modality should be used for breast ultrasonography? Comparison of the diagnostic performance of elastography and computer-aided diagnosis. Ultrasonography. 2017. 36:153–159.

12.Wang Y., Jiang S., Wang H., Guo YH., Liu B., Hou Y, et al. CAD algorithms for solid breast masses discrimination: evaluation of the accuracy and interobserver variability. Ultrasound Med Biol. 2010. 36:1273–1281.

13.Dromain C., Boyer B., Ferré R., Canale S., Delaloge S., Balleygui-er C. Computed-aided diagnosis (CAD) in the detection of breast cancer. Eur J Radiol. 2013. 82:417–423.

14.Landis JR., Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977. 33:159-174.

15.Tan T., Platel B., Twellmann T., van Schie G., Mus R., Grivegnée A, et al. Evaluation of the effect of computer-aided classification of benign and malignant lesions on reader performance in automated three-dimensional breast ultrasound. Acad Radiol. 2013. 20:1381–1388.

16.Lehman CD., Wellman RD., Buist DS., Kerlikowske K., Tosteson AN., Miglioretti DL. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern Med. 2015. 175:1828–1837.

17.Sahiner B., Chan HP., Roubidoux MA., Hadjiiski LM., Helvie MA., Paramagul C, et al. Malignant and benign breast masses on 3D US volumetric images: effect of computer-aided diagnosis on radiologist accuracy. Radiology. 2007. 242:716–724.

18.Yoo JL., Woo OH., Kim YK., Cho KR., Yong HS., Seo BK, et al. Can MR imaging contribute in characterizing well-circumscribed breast carcinomas? Radiographics. 2010. 30:1689–1702.

Fig. 1

Imaging findings from a 56-year-old woman who had ductal carcinoma in situ, an example of a false negative interpretation of CAD. CAD interpretation of this lesion as possibly benign with oval parallel, circumscribed hypoechoic features. CAD misinterpretation of ‘well-circumscribed’ breast carcinomas as benign lesions. CAD = computer-aided diagnosis

Fig. 2

Imaging findings from a 60-year-old woman who had an involuting fibroadenoma, an example of first type error of a false positive interpretation of CAD. This breast lesion was known as an involuting fibroadenoma. Heterogeneous echogenicity of this mass is applied to CAD, and due to its heterogeneity, the lesion shape is read as an irregular shape with a microlobulated margin, and the final assessment is possibly malignancy. CAD = computer-aided diagnosis

Fig. 3

Imaging findings from a 41-year-old woman who had fibrocystic change, an example of second type error of false positive interpretation of CAD. An irregular-shaped not parallel hypoechoic lesion is found on US. CAD reveals proper demarcation with irregular, not parallel, spiculated hypoechoic description with possible malignant conclusion. Due to its suspicious features, US-guided biopsy is found on US performed, and fibrocystic change is found on US confirmed. It is found on US an inevitable false positive case in the clinical setting. CAD = computer-aided diagnosis, US = ultrasound

Fig. 4

Imaging findings from a 61-year-old woman who had fibrocystic change, which is an example of fourth type error of false positive interpretation on CAD. US reveals an oval parallel circumscribed hypoechoic mass. CAD also describes this lesion as an oval parallel circumscribed hypoechoic mass without posterior features. However, the final assessment was possible malignancy. A US-guided biopsy was performed and fibrocystic change was confirmed. This was a real error in the interpretation of CAD. CAD = computer-aided diagnosis, US = ultrasound

Table 1.

Characteristics of 327 Benign Lesions and 48 Malignant Lesions

Pathology	Number
170 benign/borderline lesions
Atypical ductal hyperplasia	2
Intraductal papilloma	13
Radial scar	2
Phyllodes tumor	2
Flat epithelial atypia	2
Columnar cell hyperplasia	3
Sclerosing adenosis	3
Abscess	1
Chronic inflammation	3
Adenosis	2
Fat necrosis	4
Fibrocystic change	51
Fibroadenoma	62
Fibroadenomatous hyperplasia	5
Parasite	1
Sclerosing fibrosis	1
Stromal fibrosis	5
Tubular adenoma	2
Usual ductal hyperplasia	1
Hamartoma	3
Florid ductal hyperplasia	1
Inflammatory lymph node	1
48 confirmed malignancy
Invasive ductal carcinoma	35
Ductal carcinoma in situ	6
Mucinous carcinoma	5
Papillary carcinoma	1
Metaplastic carcinoma	1
157 breast lesions without changes for more than 2 years	157
Total	375

Table 2.

Interobserver Agreement between Radiologists and CAD System

Lexicon	Kappa Value	Agreement
Shape	0.526	Moderate
Oval
Round
Irregular
Orientation	0.590	Moderate
Parallel
Not parallel
Margin	0.377	Fair
Circumscribed
Indistinct
Angular
Microlobulated
Spiculated
Echogenicity	0.381	Fair
Anechoic
Hyperechoic
Complex cystic and solid
Hypoechoic
Isoechoic
Heterogeneous
Posterior features	0.385	Fair
No posterior features
Enhancement
Shadowing
Combined pattern
Final assessment
Possibly benign	0.422	Moderate
Possibly malignancy (≥ C4A)
Possibly benign	0.356	Fair
Possibly malignancy (≥ C4B)

C4A = category 4A, low suspicion for malignancy, C4B = category 4B, moderate suspicious for malignancy, CAD = computer-aided diagnosis

Table 3.

Diagnostic Performance of CAD System and Radiologists

	Pathology	Malignant	Benign	Total
CAD	Possibly malignant	40	66	106
	Possibly benign	8	261	269
	Total	48	327	375
	Accuracy (%)	80.3	PPV	37.7
	Sensitivity (%)	83.3	NPV	97.0
	Specificity (%)	79.8
Radiologist (Cutoff: C4A)	≥ C4A	47	99	146
	< C4A	1	228	229
	Total	48	327	375
	Accuracy (%)	73.3	PPV	32.2
	Sensitivity (%)	97.9	NPV	99.6
	Specificity (%)	69.7
Radiologist (Cutoff: C4B)	≥ C4B	35	7	42
	< C4B	13	320	333
	Total	48	327	375
	Accuracy (%)	94.7	PPV	83.3
	Sensitivity (%)	72.9	NPV	96.1
	Specificity (%)	97.9

C4A = category 4A, lo suspicion for malignancy, C4B = category 4B, moderate suspicious for malignancy, CAD = computer-aided diagnosis, NPV = negative predictive value, PPV = positive predictive value

Table 4.

Patterns of False Negative Interpretations of CAD System

No	Shape	Orientation	Margin	Echogenicity	Pathology	Size (cm)	Radiologists’ Category
1	Irregular	Parallel	Microlobulated	Isoechoic	Mucinous carcinoma	1.1	C3
2	Oval	Parallel	Circumscribed	Hypoechoic	Invasive ducal carcinoma	1.4	C4B
3	Oval	Parallel	Circumscribed	Complex cystic and solid	Ductal carcinoma in situ	1.8	C4A
4	Oval	Parallel	Microlobulated	Hypoechoic	Mucinous carcinoma	1.3	C4A
5	Oval	Parallel	Circumscribed	Hypoechoic	Mucinous carcinoma	0.3	C4A
6	Oval	Not parallel	Circumscribed	Isoechoic	Invasive ducal carcinoma	2.2	C4A
7	Oval	Parallel	Circumscribed	Isoechoic	Papillary carcinoma	0.3	C4A
8	Oval	Parallel	Microlobulated	Complex cystic and solid	Invasive ducal carcinoma	1.2	C4B

C3 = category 3, probable benign finding, C4A = category 4A, low suspicion for malignancy, C4B = category 4B, moderate suspicious for malignancy, CAD = computer-aided diagnosis

TOOLS

Similar articles