PDF
ePub
Citation
Print
Abstract
Deep learning has been applied to various medical data. In particular, current deep learning models exhibit remarkable performance at specific tasks, sometimes offering higher accuracy than that of experts for discriminating specific diseases from medical images. The current status of deep learning applications to molecular imaging can be divided into a few subtypes in terms of their purposes: differential diagnostic classification, enhancement of image acquisition, and image-based quantification. As functional and pathophysiologic information is key to molecular imaging, this review will emphasize the need for accurate biomarker acquisition by deep learning in molecular imaging. Furthermore, this review addresses practical issues that include clinical validation, data distribution, labeling issues, and harmonization to achieve clinically feasible deep learning models. Eventually, deep learning will enhance the role of theranostics, which aims at precision targeting of pathophysiology by maximizing molecular imaging functional information.
REFERENCES
1.RavıD. Wong C., Deligianni F, et al. Deep learning for health informatics. IEEE journal of biomedical and health informatics. 2017. 21(1):4–21.
2.Russakovsky O., Deng J., Su H, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision. 2015. 115(3):211–252.
3.Krizhevsky A., Sutskever I., Hinton GE. Imagenet classification with deep convolutional neural networks. Paper presented at: Advances in neural information processing systems2012.
4.Weber GM., Mandl KD., Kohane IS. Finding the missing link for big biomedical data. Jama. 2014. 311(24):2479–2480.
5.Esteva A., Kuprel B., Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017. 542(7639):115–118.
6.Rajpurkar P., Irvin J., Ball RL, et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS medicine. 2018. 15(11):e1002686.
7.Dhungel N., Carneiro G., Bradley AP. Automated mass detection in mammograms using cascaded deep learning and random forests. Paper presented at: Digital Image Computing: Techniques and Applications (DICTA), 2015 International Conference on2015.
8.Lakhani P., Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017. 284(2):574–582.
9.Litjens G., Kooi T., Bejnordi BE, et al. A survey on deep learning in medical image analysis. Medical image analysis. 2017. 42:60–88.
10.Choi H. Deep learning in nuclear medicine and molecular imaging: current perspectives and future directions. Nuclear medicine and molecular imaging. 2018. 1–10.
11.Wang H., Zhou Z., Li Y, et al. Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18 F-FDG PET/CT images. EJNMMI research. 2017. 7(1):11.
12.Kirienko M., Sollini M., Silvestri G, et al. Convolutional Neural Networks Promising in Lung Cancer T-Parameter Assessment on Baseline FDG-PET/CT. Contrast Media & Molecular Imaging. 2018. 2018.
13.Choi H., Jin KH., AsDN Initiative. Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. Behavioural brain research. 2018. 344:103–109.
14.Ding Y., Sohn JH., Kawczynski MG, et al. A deep learning model to predict a diagnosis of Alzheimer disease by using 18F-FDG PET of the brain. Radiology. 2018. 290(2):456–464.
15.Liu M., Cheng D., Yan W. Classification of Alzheimer's Disease by Combination of Convolutional and Recurrent Neural Networks Using FDG-PET images. Frontiers in neuroinformatics. 2018. 12:35.
16.Liu S., Liu S., Cai W, et al. Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer's disease. IEEE Transactions on Biomedical Engineering. 2015. 62(4):1132–1140.
17.Suk H-I., Lee S-W., Shen D., AsDN Initiative. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage. 2014. 101:569–582.
18.Li F., Tran L., Thung K-H., Ji S., Shen D., Li J. A robust deep model for improved classification of AD/MCI patients. IEEE journal of biomedical and health informatics. 2015. 19(5):1610–1616.
19.Choi H., Ha S., Im HJ., Paek SH., Lee DS. Refining diagnosis of Parkinson's disease with deep learning-based interpretation of dopamine transporter imaging. NeuroImage: Clinical. 2017.
20.Martinez-Murcia FJ., Gorriz JM., Ramirez J., Ortiz A. Convolutional Neural Networks for Neuroimaging in Parkinson's Disease: Is Preprocessing Needed? International journal of neural systems. 2018. 1850035–1850035.
21.Kim DH., Wit H., Thurston M. Artificial intelligence in the diagnosis of Parkinson's disease from ioflupane-123 single-photon emission computed tomography dopamine transporter scans using transfer learning. Nuclear medicine communications. 2018. 39(10):887–893.
22.Betancur JA., Hu L-H., Commandeur F, et al. Deep Learning Analysis of Upright-Supine High-Efficiency SPECT Myocardial Perfusion Imaging for Prediction of Obstructive Coronary Artery Disease: A Multicenter Study. Journal of Nuclear Medicine. 2018. jnumed. 118.213538.
23.Xu C., Xu L., Gao Z, et al. Direct detection of pixel-level myocardial infarction areas via a deep-learning algorithm. Paper presented at: International Conference on Medical Image Computing and Computer-Assisted Inter-vention2017.
24.Betancur J., Commandeur F., Motlagh M, et al. Deep learning for prediction of obstructive disease from fast myocardial perfusion SPECT: a multicenter study. JACC: Cardiovascular Imaging. 2018. 11(11):1654–1663.
25.Kim K., Wu D., Gong K, et al. Penalized PET reconstruction using deep learning prior and local linear fitting. IEEE transactions on medical imaging. 2018. 37(6):1478–1487.
26.Gong K., Catana C., Qi J., Li Q. PET Image Reconstruction Using Deep Image Prior. IEEE transactions on medical imaging. 2018.
27.Gong K., Guan J., Kim K, et al. Iterative PET image reconstruction using convolutional neural network representation. IEEE transactions on medical imaging. 2019. 38(3):675–685.
28.Zhu B., Liu JZ., Cauley SF., Rosen BR., Rosen MS. Image re-construction by domain-transform manifold learning. Nature. 2018. 555(7697):487.
29.Pfaehler E., De Jong Jr., Dierckx RA., van Velden FH., Boellaard R. SMART (SiMulAtion and ReconsTruction) PET: an efficient PET simulation-reconstruction tool. EJNMMI physics. 2018. 5(1):16.
30.Hwang D., Kang SK., Kim KY, et al. Generation of PET attenuation map for whole-body time-of-flight 18F-FDG PET/MRI using a deep neural network trained with simultaneously reconstructed activity and attenuation maps. Journal of Nuclear Medicine. 2019. jnumed.118.219493.
31.Han X. MR–based synthetic CT generation using a deep convolutional neural network method. Medical physics. 2017. 44(4):1408–1419.
32.Liu F., Jang H., Kijowski R., Bradshaw T., McMillan AB. Deep learning MR imaging–based attenuation correction for PET/MR imaging. Radiology. 2017. 286(2):676–684.
33.Leynes AP., Yang J., Wiesinger F, et al. Direct pseudoCT generation for pelvis PET/MRI attenuation correction using deep convolutional neural networks with multi-parametric MRI: zero echo-time and dixon deep pseudoCT (ZeDD-CT). Journal of Nuclear Medicine. 2017. jnumed. 117.198051.
34.Hwang D., Kim KY., Kang SK, et al. Improving the accuracy of simultaneously reconstructed activity and attenuation maps using deep learning. Journal of Nuclear Medicine. 2018. 59(10):1624–1629.
35.Xiang L., Qiao Y., Nie D, et al. Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing. 2017. 267:406–416.
36.Chen KT., Gong E., de Carvalho Macruz FB, et al. Ultra–Low-Dose 18F-Florbetaben Amyloid PET Imaging Using Deep Learning with Multi-Contrast MRI Inputs. Radiology. 2018. 180940.
37.Wang Y., Yu B., Wang L, et al. 3D conditional generative adversarial networks for high-quality PET image estimation at low dose. Neuroimage. 2018. 174:550–562.
38.Wang T., Lei Y., Tang H, et al. A learning-based automatic segmentation and quantification method on left ventricle in gated myocardial perfusion SPECT imaging: A feasibility study. Journal of Nuclear Cardiology. 2019. 1–12.
39.Belal SL., Sadik M., Kaboteh R, et al. Deep Learning for Segmentation of 49 Selected Bones in CT Scans: First Step in Automated PET/CT-based 3D Quantification of Skeletal Metastases. European Journal of Radiology. 2019.
40.Chen L., Shen C., Zhou Z, et al. Automatic PET cervical tumor segmentation by combining deep learning and anatomic prior. Physics in medicine and biology. 2019.
41.Zhong Z., Kim Y., Plichta K, et al. Simultaneous cosegmen-tation of tumors in PET–CT images using deep fully convolutional networks. Medical physics. 2019. 46(2):619–633.
42.Huang B., Chen Z., Wu P-M, et al. Fully Automated Delineation of Gross Tumor Volume for Head and Neck Cancer on PET-CT Using Deep Learning: A Dual-Center Study. Contrast media & molecular imaging. 2018. 2018.
43.Choi H., Lee DS. Generation of structural MR images from amyloid PET: Application to MR-less quantification. Journal of Nuclear Medicine. 2018. 59(7):1111–1117.
44.Kang SK., Seo S., Shin SA, et al. Adaptive template generation for amyloid PET using a deep learning approach. Human brain mapping. 2018. 39(9):3769–3778.
45.Ding Y., Sohn JH., Kawczynski MG, et al. A deep learning model to predict a diagnosis of Alzheimer disease by using 18F-FDG PET of the brain. Radiology. 2018. 180958.
46.Samarin A., Burger C., Wollenweber SD, et al. PET/MR imaging of bone lesions–implications for PET quantification from imperfect attenuation correction. European journal of nuclear medicine and molecular imaging. 2012. 39(7):1154–1160.
47.Choi H., Cheon GJ., Kim H-J, et al. Segmentation-based MR attenuation correction including bones also affects quantitation in brain studies: an initial result of 18F-FP-CIT PET/MR for patients with parkinsonism. Journal of Nuclear Medicine. 2014. 55(10):1617–1622.
48.Park J., Bae S., Seo S, et al. Measurement of Glomerular Filtration Rate using Quantitative SPECT/CT and Deep-learning-based Kidney Segmentation. Scientific reports. 2019. 9(1):4223.
49.Beam AL., Kohane IS. Translating artificial intelligence into clinical care. Jama. 2016. 316(22):2368–2369.
50.He J., Baxter SL., Xu J., Xu J., Zhou X., Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nature medicine. 2019. 25(1):30.
51.Saria S., Butte A., Sheikh A. Better medicine through machine learning: What's real, and what's artificial?: Public Library of Science. 2018.
52.Park SH., Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology. 2018. 286(3):800–809.
53.Redelmeier DA., Shafir E. Medical decision making in situations that offer multiple alternatives. Jama. 1995. 273(4):302–305.
54.Gal Y., Ghahramani Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Paper presented at: international conference on machine learning2016.
55.Wei Q., Ren Y., Hou R., Shi B., Lo JY., Carin L. Anomaly detection for medical images based on a one-class classification. Paper presented at: Medical Imaging 2018: Computer-Aided Diagnosis. 2018.
56.Schlegl T., Seeböck P., Waldstein SM., Schmidt-Erfurth U., Langs G. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. Paper presented at: International Conference on Information Processing in Medical Imaging2017.
57.Choi H., Kang H., Lee DS., Initiative AsDN. Predicting aging of brain metabolic topography using variational autoen-coder. Frontiers in aging neuroscience. 2018. 10.
58.Le QV., Ranzato MA., Monga R, et al. Building high-level features using large scale unsupervised learning. arXiv preprint arXiv: 11126209. 2011.
59.Bengio Y. Deep learning of representations for unsupervised and transfer learning. Paper presented at: Proceedings of ICML Workshop on Unsupervised and Transfer Learning2012.
60.Rasmus A., Berglund M., Honkala M., Valpola H., Raiko T. Semi-supervised learning with ladder networks. Paper presented at: Advances in neural information processing systems2015.
61.Odena A. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv: 160601583. 2016.
62.Choi H., Na KJ. Integrative analysis of imaging and tran-scriptomic data of the immune landscape associated with tumor metabolism in lung adenocarcinoma: Clinical and prognostic implications. Theranostics. 2018. 8(7):): 1956.
63.Shin H-C., Lu L., Kim L., Seff A., Yao J., Summers RM. Interleaved text/image deep mining on a very large-scale radiology database. Paper presented at: Proceedings of the IEEE conference on computer vision and pattern recogni-tion2015.
64.Gomez L., Patel Y., Rusiñol M., Karatzas D., Jawahar C. Self-supervised learning of visual features through embedding images into text topic spaces. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition2017.
65.Pan L., Cheng C., Haberkorn U., Dimitrakopoulou-Strauss A. Machine learning-based kinetic modeling: a robust and reproducible solution for quantitative analysis of dynamic PET data. Physics in Medicine & Biology. 2017. 62(9):3566.
66.Landau SM., Breault C., Joshi AD, et al. Amyloid-β imaging with Pittsburgh compound B and florbetapir: comparing radiotracers and quantification methods. Journal of Nuclear Medicine. 2013. 54(1):70–77.
67.Klunk WE., Koeppe RA., Price JC, et al. The Centiloid Project: standardizing quantitative amyloid plaque estimation by PET. Alzheimer's & dementia. 2015. 11(1):1–15. e14.
Fig. 1
The output of deep learning model as a predictive biomarker. A deep convolutional neural network (CNN) model was developed to differentiate brain PET of Alzheimer's disease from healthy subjects. This model was applied to another cohort, mild cognitive impairment patients to predict future cognitive outcome. The output of the model represents a probability of Alzheimer's disease, which can be used as a predictive biomarker for predicting cognitive outcome in preclinical disorders.
Fig. 2
A gap between training and real-world data. Most of deep learning models are developed by patients’ data with specific disorders and controls. The problem of deep learning application to the clinic is the difference between real-world data and the training cohort. Real-world data in the clinic included heterogeneous patients different from training cohorts. Furthermore, the distribution of disease and normal is considerably different. This data distribution issue become a bigger factor when deep learning aims at general population.
Fig. 3
Leveraging unlabeled data as a clinical routine for facilitating deep learning development. As labeling for medical data is too expensive and time-consuming, it is a bottleneck for developing deep learning models. Since it is relatively easy to collect heterogeneous image data obtained for clinical routine, unsupervised learning can leverage these unlabeled ‘dirty’ data. Unsupervised learning-based feature extraction can be transferred to relatively small cohorts which contain both labels and images to predict clinical outcome as well as differential diagnosis according to the clinical purposes.
Table 1.
Types of current deep learning applications for nuclear medicine and molecular imaging
XML Download