This article has been
cited by other articles in ScienceCentral.
Abstract
Objectives
Augmented reality (AR) technology has become rapidly available and is suitable for various medical applications since it can provide effective visualization of intricate anatomical structures inside the human body. This paper describes the procedure to develop an AR app with Unity3D and Vuforia software development kit and publish it to a smartphone for the localization of critical tissues or organs that cannot be seen easily by the naked eye during surgery.
Methods
In this study, Vuforia version 6.5 integrated with the Unity Editor was installed on a desktop computer and configured to develop the Android AR app for the visualization of internal organs. Three-dimensional segmented human organs were extracted from a computerized tomography file using Seg3D software, and overlaid on a target body surface through the developed app with an artificial marker.
Results
To aid beginners in using the AR technology for medical applications, a 3D model of the thyroid and surrounding structures was created from a thyroid cancer patient's DICOM file, and was visualized on the neck of a medical training mannequin through the developed AR app. The individual organs, including the thyroid, trachea, carotid artery, jugular vein, and esophagus were localized by the surgeon's Android smartphone.
Conclusions
Vuforia software can help even researchers, students, or surgeons who do not possess computer vision expertise to easily develop an AR app in a user-friendly manner and use it to visualize and localize critical internal organs without incision. It could allow AR technology to be extensively utilized for various medical applications.
Keywords: Virtual Reality, Medical Education, Three-Dimensional Imaging, Thyroidectomy, Augmented Reality
I. Introduction
Since it has the advantage of visualizing complex human structures in three dimensions, augmented reality (AR) technology has been developed over the last 20 years and its utilization has recently become more common in various medical applications [
12]. In particular, research on AR have been actively conducted regarding its application in the surgical field, which deals directly with human tissue [
3]. A search for the keyword, ‘Augmented Reality Surgery’ in PubMed shows that approximately 50 to 70 papers have been annually published on this topic since 2015.
The advantages of applying AR in the surgical field are as follows. The exact anatomical location of the organ to be operated on can be determined before the skin incision [
4]. It can provide safe and efficient surgery by visualization and localization of critical anatomical structures inside the human body, such as blood vessels, nerves, and bile ducts, which cannot be seen easily by the naked eye [
5]. In addition, it is important to have a proper surgical margin in the case of cancer surgery. In the case of a tumor located in an organ, it is possible to obtain a sufficient and safe surgical margin by visualizing the location of the tumor with AR [
6]. Finally, AR data for each patient can be used to help establish a preoperative plan, and it can be used in surgical education for students or surgeons [
27].
This paper describes methods for using Unity and Vuforia to realize AR and presents some scenarios in which AR can be used for medical applications.
II. Methods
1. Installation of Unity and Vuforia
AR can be achieved using Vuforia software development kit (SDK) in the Unity development environment. Unity is an editor for creating products such as 3D video games or animations, and Vuforia is an AR software platform created by Qualcomm. In the Vuforia version 6.5, Unity and Vuforia do not need to be installed separately because Vuforia is integrated with the Unity Editor and will work if the files are installed. The personal version of Unity is available free of charge and the download assistant file can be downloaded at the following web address (
https://store.unity.com/). The system requirements are Windows 7 SP1+, 8, or 10, or Mac OS X 10.9+.
The installation process is conducted as follows. Double-click the ‘UnityDownloadAssistant’ file and choose ‘Components’ on the pop-up screen, then check the ‘Vuforia Augmented Reality Support’ option additionally. Next, sign in with your Unity ID and access Unity with your account. Complete the license agreement and survey to finish the installation (
Figure 1).
2. Uploading Target Database
To use the AR features provided by Vuforia, one should first request a license key. To request a license key, log in to your account on the Vuforia homepage, and click the ‘Develop’ button to open the ‘License Manager’ window. Then click on the ‘Get Development Key’ button to request a license key (
Figure 2).
To implement AR, one of two image models should be selected to determine either ‘where it appears’ or ‘what appears’. Here, the ‘where’ in ‘where it appears’ is called a marker or target.
① On the Vuforia homepage, click the ‘Develop’ button to open the ‘Target Manager’, and then click the ‘Add Database’ button. Next, click the ‘Add Target’ button to select a target database. There are four target categories that can be selected: Single Image, Cuboid, Cylinder, and 3D Object as shown in Figure 3. After making a selection, enter the image's width (mm) in the ‘Width’ section. In this experiment, we selected ‘Single Image’ because we used a 2D image as a target.
② Click the ‘Download Database’, and set the development platform to ‘Unity Editor’ to download the unity package.
3. AR Process Using Unity and Vuforia
Run the Unity Editor and create a new project file. The Unity Editor workspace consists of the following four items as shown in
Figure 4.
Hierarchy: contains a list of every GameObject, asset file, and instance of prefabs in the current Scene,
Scene: shows the image screen that is currently being worked on,
Inspector: shows various types of properties and allows them to be edited,
Project: contains the Assets folder.
1) Basic settings (Figure 5)
① To import the registered target file, select ‘Assets’ and then ‘Import Package’. Subsequently, select ‘Custom package’ in the menu window and import the previously downloaded unitypackage file.
-
② To use Vuforia SDK's AR features, go to the menu and select ‘ARCamera’ and ‘Image’ as follows.
GameObject → Vuforia → ARCamera
GameObject → Vuforia → Image
③ Click the ‘PLAY’ button to augment the image of the webcam that is connected to user's PC.
2) Registering the license key (Figure 6)
To register the license key, select ‘Hierarchy’, and then select ‘ARCamera’ to open the Inspector menu. Click the ‘Open Vuforia configuration’ button at the bottom of the ‘Vuforia Behavior (Script)’ window. Copy and paste the issued license key into the ‘App License Key’ section.
3) Loading and activating the target image (Figure 7)
① Select ‘Hierarchy’ and then ‘ARCamera’. After that, click the ‘Open Vuforia configuration’ button. At the bottom of the Datasets menu, select the target file uploaded on the Vuforia homepage and check ‘Activate’.
② Select ‘Hierarchy’ and then ‘ImageTarget’. Click ‘Inspector’ and select the uploaded database folder name and the file uploaded in the ‘Data Set’ window and the ‘Image Target’ window below ‘Image Target Behavior (Script)’, respectively.
4) Loading overlay model file
5) Play
① In the ‘Scene’ window, adjust the position, angle, and magnification to arrange the desired positioning between the target image and the model to be overlaid.
② Click the ‘Play’ button at the top of the Unity Editor to run it.
6) Android platform usage example (Figure 8)
-
① After the previous process, use the menu to select the following:
File Build Settings → Platform → Android.
Next, click the ‘Open Download Page’ button to download the ‘Android Supporter’ file and install it.
② After installation, click the ‘Player Settings’ button and set the ‘Icon’, ‘API Level’, and other settings in the Android menu window at the bottom of the Inspector.
③ Click the ‘build’ button to create an .apk file and download this file to an Android mobile phone. The AR functions that operate on a PC can now be implemented on the phone.
III. Results
In this tutorial, a 3D model of the thyroid and surrounding structures was created from a thyroid cancer patient's CT DICOM file using Seg3D open-source software [
8].
Figure 9 shows the results of implementing AR using the PC version (Win 7, 64 bit) and a mobile phone running the Android OS. The constructed 3D model of the thyroid and surrounding structures was overlaid on 2D markers (around the neck) attached to a thyroid body model.
IV. Discussion
This paper described the overall AR process using the Unity Editor and Vuforia SDK and showed that AR can easily be used in medical applications, illustrated via an example in which AR was used in thyroid laparoscopic surgery. Even without expertise in image processing methods, such as registration techniques, this software can easily be used in AR research using various medical images. This tutorial is for beginners, and more detailed and in-depth usage guidelines and suggestions will be presented in the future.
Figures and Tables
Figure 1
Installation procedures of Unity and Vuforia.
Figure 2
Usage of License Manager to get development key, which is used in Unity environment for Vuforia configuration.
Figure 3
Usage of Target Manager to add target database. There are four target categories that can be selected: Single Image, Cuboid, Cylinder, and 3D Object.
Figure 4
Interface of Unity, consisting of Inspector, Scene, Project, and Hierarchy.
Figure 5
Basic settings.
Figure 6
Registering the license key.
Figure 7
Loading and activating the target datasets.
Figure 8
Build settings for PC and Android version.
Figure 9
Example of application of augmented reality technology: thyroid and internal organs on a robotic thyroidectomy training model through a smartphone display.
Acknowledgments
This work was supported by research fund of Chungnam National University. The authors thank Hyoun-Jin Joo for his technical assistance in developing the AR application.
References
1. Chang H, Choi M. Big data and healthcare: building an augmented world. Healthc Inform Res. 2016; 22(3):153–155.
2. Bernhardt S, Nicolau SA, Soler L, Doignon C. The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal. 2017; 37:66–90.
3. Khor WS, Baker B, Amin K, Chan A, Patel K, Wong J. Augmented and virtual reality in surgery-the digital surgical environment: applications, limitations and legal pitfalls. Ann Transl Med. 2016; 4(23):454.
4. Kim Y, Kim H, Kim YO. Virtual reality and augmented reality in plastic surgery: a review. Arch Plast Surg. 2017; 44(3):179–187.
5. Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, et al. Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: a case report. Medicine (Baltimore). 2017; 96(37):e8083.
6. Soler L, Nicolau S, Pessaux P, Mutter D, Marescaux J. Real-time 3D image reconstruction guidance in liver resection surgery. Hepatobiliary Surg Nutr. 2014; 3(2):73–81.
7. Barsom EZ, Graafland M, Schijven MP. Systematic review on the effectiveness of augmented reality applications in medical training. Surg Endosc. 2016; 30(10):4174–4183.
8. Hassan K, Dort JC, Sutherland GR, Chan S. Evaluation of software tools for segmentation of temporal bone anatomy. Stud Health Technol Inform. 2016; 220:130–133.
PDF
ePub
Citation
Print
XML Download