PDF
Citation
Print
INTRODUCTION
In medical education, the training of students through simulations is important because diagnostic procedures are difficult for physicians to learn. Virtual reality (VR) provides the possibility for students to experience real-world clinical situations that require diagnostic skill training.12345 Although the techniques required to use VR devices are important, there is a lack of educational environment to describe them. Also, knowing computer technology can help medical scientists develop software packages that are more beneficial to medicine.
For more immersive systems, various sensors have been proposed to assist with user experiences in the real world. These sensors use a certain device for measuring the distance from the user and recognizing user gestures.6 Kinect, Wii, Leap Motion, Playstation Move, and Myo are some typical examples.789101112 These motion-recognition sensors improve the immersion of user interaction because of their compact structure. However, their accuracy depends on the type of predefined gestures because they cannot recognize and find user actions accurately. We also defined in advance user gestures that represent similar behaviors of anatomists.
Several systems to visualize anatomical images have been introduced in the studies of medical sciences and training of medical students. The Anatomage Table13 provides virtual surgical training because it has been designed to have the form of an operating table. However, it does not provide a realistic stereoscopic model and special interface; it only provides a three-dimensional (3D) model and touch-screen interface. Lundstrom14 proposed a tabular computer that can visualize medical images for simulating a real operation. This proposed device is suitable for establishing a surgical plan because orthopedic surgeons have been involved in this particular project. However, its 2D results are less realistic than are stereoscopic images.
Most medical data captured during computed tomography, magnetic resonance imaging, and ultrasound are represented as 2D gray-scale images. Color anatomical datasets including Visible Human,1516 Visible Korean (VK),17181920 and Visible Chinese2122 represent human anatomical structures with real-color, high-resolution images. We used the VK dataset for our simulator to produce a high-definition (HD) image display. However, the size of these datasets reaches tens or hundreds of gigabytes (GB) even when we scan a tiny region of the human body, because a series of such images is regarded as a 3D texture sampled using a very high resolution. In our simulator, we crop the area of a specific region of the whole body to concentrate on the surgical training of the target organ.
Our virtual dissection simulator is also operated through user's gestures from motion-recognition sensors. When performing virtual dissection, some parts of the volume should be moved separately. This requires a type of internal representation to show the cross-sections made through anatomical operations. We need a method to produce an exploded view because both sides of an incised or excised part have an important meaning—the reason being an efficient description of the relationship between the incised or excised parts and the remaining parts exists.
Virtual endoscopy is a method for visualizing the pathological structures in pipe-shaped organs such as the colon, bronchus, and esophagus. Unlike optical endoscopy, there is no discomfort or side effect because the examination is noninvasive. We can observe a wide area inside an organ because the method produces perspective images while navigating the organ. To represent accurate movements of the virtual camera during user manipulation, making a reliable path is essential for diagnostic training. Before a virtual endoscopic simulation, we need a method for producing an exploded view to show the relationship between the separated incised or excised parts.
We propose an interactive anatomy simulator using motion-recognition sensors to improve the sense of immersion. This simulator is composed of two parts: 1) virtual dissection and 2) virtual endoscopy. It also provides HD stereoscopic images to increase the reality with high-resolution anatomical volume data. The stereoscopic images mean that it provides two images separately to the left and right eye for binocular vision. In our method, we made these images by using the GPU-based visualization method because using GPU can make two images fast.23 For gesture recognition, we use a Kinect device and its software development kit (SDK). To handle the motion of a virtual camera, we define an area—called a virtual control sphere (VCS)—around every camera position along the central navigation path. Because VCSs define the restricted area of the navigation camera, our simulator can move the camera along the central path, without touching the organ's wall. When the camera is free from user control, it turns back to the central path. Our simulator can be used to train medical students before they use actual endoscopic equipment.
The contributions of this article are as follows: 1) We provide a simulator with a more realistic and interactive interface than ordinary software which uses a normal input device such as a mouse. Using a gesture-based interface can be useful for reducing contagious infections. 2) It can be used to train medical students before they study anatomical structures. It can be used to observe the endoscope images from various angles by manipulating the motion of the virtual endoscope camera using motion recognition interface.
We describe the materials and methods applied in the “Methods” section, summarize some experimental results and discussions in the “Results” section, and finally, provide some concluding remarks regarding our work in the “Discussion” section.
METHODS
We use color images of the VK dataset. The resolution of these images is 2,468 × 1,407, and each pixel size is 0.2 mm × 0.2 mm. The interval between the images is also 0.2 mm.24 To select the target organ, anatomical structures should be segmented. The volume data, which include color and segment information, are reconstructed (as presented in Table 1). The resolution of each dataset is sufficient to be displayed on an HD screen. As graphics accelerator technology is radically advancing, it is possible to visualize 3D objects even when we use a large dataset. Although graphics accelerators are equipped with 6–8 GB of video memory, we crop the specific region for the target organ because the size of the volume is huge for the memory to accommodate at once. In addition, we separate this dataset into four channels (red, green, blue, and segmentation index) to avoid a large amount of memory usage at the same location.
Table 1
The resolution and capacity of datasets for our simulator
Our system consists of two parts: virtual dissection (Fig. 1) and virtual endoscopy (Fig. 2). It includes a gesture-recognition sensor module and a volume-rendering module. It provides an HD stereoscopic display to ensure an immersive environment. When the user stands in front of the system and manipulates the anatomy simulator using several predefined gestures, it provides an immersive environment because user operations are based on a noncontact interface.
Fig. 1
Result of virtual dissection. (A) Virtual dissection of the abdomen, (B) Three pipe-like organs in the abdomen, the user can select an organ to navigate inside using a motion sensor.
Fig. 2
Overall procedure of the virtual endoscopy simulator. (A) Along the navigation path has restricted region, (B) visual representation of user manipulation using a virtual camera near the navigation path.
Virtual dissection simulator
The virtual dissection part of our system is operated through user's gestures. When performing dissection, the volume data itself should be deformed and moved. We separate the volume data into several sub-volumes, with user-specific parameters under compute unified device architecture (CUDA). These volume data are used as the scale factor of the corresponding proxy geometry during the rendering phase. During the vertex process, this information is used as a parameter to scale the unit blocks. The scaled blocks are used as proxy geometry of the volume through a scaling operation. Because proxy geometry can be applied to skip an empty space and terminate an early ray in GPU ray casting during the fragment process, we can accelerate the rendering speed.
Large-sized medical content such as the Visible Male,24 Visible Female,25 and Visible Head26 is very useful in volume rendering. We made volume data using these image contents and cropped the 3D area against the region of interest in our system. The volume data, which only include color values, additionally require segmented information because we cannot recognize the shapes of objects on the basis of color values. These segmented data are created manually or semiautomatically.27 After segmentation, we can smooth out these segmented data with a Gaussian filter because the boundaries of objects achieved through manual segmentation are clumsy.
To begin with, we separate the volume data into several sub-volumes, with user-specific parameters under CUDA. Each sub-volume has its own resolution width, height, and length. These size data are used as the scale factor of the corresponding proxy geometry during the rendering phase.
During the vertex process, this scale information is used as a parameter to scale a unit block (1.0 × 1.0 × 1.0) into preferred block sizes. The scaled blocks are used as proxy geometry for a volume with a scaling operation. Because proxy geometry can be used to skip an empty space and terminate an early ray of GPU ray casting during the fragment process, we can accelerate the rendering speed. In the ray-sampling step, we render an image in a frame buffer, which refers to the multiple 3D textures of the sub-volumes separated on CUDA and a predefined opacity transfer function.
Virtual endoscopy simulator
Our virtual endoscopy simulator is also operated using user's gestures. To provide an immersive environment, the simulation system includes an HD stereoscopic display. After virtual dissection, we can select an internal organ to perform virtual endoscopy. The navigation path, which is composed of several control points, and the distance map2829 are prepared in advance because our endoscopy simulator is designed for medical training and not for diagnostic purposes. In the rendering step, we calculate the position of the virtual camera using rendering points by applying control points and distance values.
Computing navigation path using VCS
In the virtual endoscopy part of our simulator, we glide through the interiors of pipe-like organs, and therefore, computing the navigation path is important. We defined some control points using the center of gravity within an organ cavity.3031
Fig. 2 shows the overall procedure of the virtual endoscopy simulator. Along the navigation path has restricted region (Fig. 2A). The camera movement needs to be restricted because the camera should glide through the wrinkled wall of a tubular organ. If the degree of freedom of the camera movement increases, an unskilled person may lose track of the camera direction while navigating within the interior of a wrinkled wall. Therefore, we define a restricted region—i.e., the VCS (Fig. 2A)—for the camera movement, using a series of spheres along the navigation path. This region helps avoid a wrinkled wall during navigation, and it is defined by a distance r centered with the current position of the camera. The camera can move a distance d from the current position through user gestures. This distance of user movement d
user should be less than distance r of the VCS. The right side of Fig. 2B shows the process of camera movement using VCSs. During navigation, the camera can rotate in yaw and pitch and move forward and backward. In addition, it rotates at an angle within a field of view of 70° relative to the marching direction because this angle is the field of view of the endoscope camera. The speed of the camera movement is adjustable, depending on user navigation skills.
If we place VCSs uniformly along the navigation path instead of setting their size, when the size of a VCS has been fitted for a large inside area, the camera may clash with a wall in a smaller inside area. We have to avoid such clashes and observe as wide a range as possible. Therefore, we set the radius of VCSs adaptively using the diameter of the tubular organ.
The experiments were conducted on a system using an Intel Core i7-8700 CPU (3.70 GHz) and 16 GB main memory. An nVidia TITAN X (12 GB video memory) was used as the graphics processor. The Windows 10 operating system and a Kinect SDK 1.8 were also used for the gesture-recognition experiment.
RESULTS
We used tubular organs of the VK dataset, such as the respiratory tract (from the trachea to the segmental bronchus) and gastrointestinal tract (the esophagus, stomach, and large intestines), without small intestines (as provided in Table 1). We used these datasets at the resolution of the original dataset and cropped the dataset sufficiently for inclusion in the video memory.
Fig. 1 shows an example of virtual dissection. This helps us understand the internal structure of an organ itself and the interrelationships of the neighboring organs. After virtual dissection, we can select an organ to navigate inside. Yellow color indicates that the organ (in this case, the large intestines) has been selected by user gestures. When we select the respiratory tract, the user can fly through the inside of the respiratory tract, which is from the trachea to the left and right bronchi of each lobar bronchus.
Fig. 3 shows the results of virtual colonoscopy from the anus to the cecum. The camera movement is controlled through user gestures acquired from the Kinect device. During navigation control using such gestures, the camera can rotate in yaw and pitch and move forward and backward. A collision is detected when the user exits the VCS region, which is helpful in virtual endoscopy training.
Fig. 3
Virtual colonoscopy from the anus to the cecum (The arrow is forward direction in each section). (A) The colonoscope enters rectilinearly from the anus to rectum. (B) In the sigmoid colon, the colonoscopy moves in the sigmoid shape. (C) The colonoscopy moves in the descending colon. (D, E) In the descending, transverse, and ascending colons, the colonoscope passes straightly, but it turns in their junctions. (F) In the cecum, the colonoscope does not go to the ileum. Furthermore, the fold by the appendix can be shown.
The virtual colonoscope enters rectilinearly from the anus to rectum (Fig. 3A). In the sigmoid colon, the colonoscopy moves in the sigmoid shape (Fig. 3B). In the descending, transverse, and ascending colons, the colonoscope passes straightly, but it turns in their junctions (Fig. 3D and E). In the cecum, the colonoscope does not go to the ileum. Furthermore, the fold by the appendix can be verified.
The right-side column in Fig. 4 shows virtual bronchoscopy images generated with our simulator. These are rendered images obtained by navigating from the entrance of the trachea to the right bronchus. When we enter the trachea and reach the carina, there is the intersection of the left and right bronchi. Going further into the carina, we find that the left bronchus has a gentle slope and the right bronchus has a steep slope. If we go further, we see several branches, which are called the lobar bronchus. We can go to the segmental bronchus finally. We can also verify the bronchial wrinkles clearly through a realistic color and shape. In addition, a navigation map depicting the current position is provided at the bottom-left of each frame. The left-side column of Fig. 4 shows the results of virtual esophagogastroscopy. The camera movement is controlled using user gestures from the Kinect device. The user can move the navigation camera freely inside the VCS region. A collision is detected when the user escapes the VCS region, and this collision detection is helpful in virtual endoscopy training.
Fig. 4
Virtual esophagogastroscopy and virtual bronchoscopy. (A) The virtual esophagogastroscope enters in the stomach through the esophagus and (B) the cardia of the stomach. (C) The scope passes the longitudinal folds inside the stomach. (D, E) The virtual bronchoscope encounters two ways (right and left main bronchi) and a split spot (carina). (F) In right lung with three lobes, the scope encounters two entrances (middle lobar and inferior lobar bronchi).
DISCUSSION
There is not enough time to diagnose a disease during real endoscopy, and we cannot practice it as well. We should not practice on living things ethically, so our simulator is a solution for practice. During navigation, it is possible to effectively observe the entire area of the colon because the user can control the camera freely.
In most virtual anatomy software packages, a keyboard, a mouse, and joysticks are used. But user gesture, in a manner similar to this study, is more suitable for practicing real diagnosis and surgery. Several touchless interfaces using gesture recognition have been studied in the medical field.323334 Ruppert et al.32 implemented a gesture recognition system based on the Kinect device that enables a surgeon to touchless navigate through the image in the intraoperative setting using a computer. Chiang et al.33 presented a novel medical imaging observation system using gesture-based technique to build a touchless interactive environment. The gesture interface can help in teaching realistic clinical medicine and anatomy. Fig. 5 shows an operating person using our simulator and a Kinect device. The user can move the virtual camera during virtual endoscopy using user's gestures and perform virtual dissection.
Our method can observe the endoscope images from various angles by manipulating the motion of the virtual endoscope camera using motion recognition interface.
Several applications to visualize the internal structure of the human body have been introduced. However, these applications do not provide a realistic simulation environment or adequate training opportunities because they simply display a 2D image and use common interface. Our system includes an HD stereoscopic display device. Our method can provide a rendering speed of 70 fps even though a high-resolution stereoscopic device. Our interactive anatomy simulator that generates realistic images and provides gesture-based interactions is useful for educating in the realistic environment. It also includes an immersive display incorporating a motion-recognition sensor. We also provide three virtual endoscopy (colonoscopy, bronchoscopy and esophagogastroscopy) using the colon, bronchus, and stomach model.
In our previous study,35 we created 671 endoscopic images of the colon for virtual colonoscopy and labeled them and provided educational colonoscopy tutorial software only using these fixed images. Since images made along the central path of the large intestine have a limited viewing direction, this paper improved the viewing direction interactively near the central path according to the user's intention (Fig. 2). And the interface with the user was developed to utilize the user's gestures (Fig. 5).
Our simulator would be helpful in educating medical students because it provides an immersive environment. In addition, the gesture-based interface of our simulator would be useful for reducing contagious infections.
XML Download