Between September 2012 and April 2013, the case review system was developed by a team that comprised 2 teaching radiologists (K.H.L. and M.H.L. who had 10- and 2-year experience, respectively), a practitioner radiologist (H.W. with 5-year experience), a radiology resident (H.K.Y.), and 2 computer scientists (B.K. with 15-year experience in medical image processing and J.S. with 10-year experience in human computer interaction). The practitioner radiologist led the system development. A computer science student (J.J.) and a radiology technologist (J.H.J.) assisted in system implementation. The radiologists and radiology resident also served as case authors by preparing the case data for a pilot test of the system. All members of the development team participated in developing the system requirements and implementation (Table 1).

The system requirements were determined as follows. First, the system should allow the trainees to navigate through volumetric image datasets using advanced visualization techniques such as multiplanar reformation, volume rendering, and fly-through. Second, the system should be accessible from trainees' desktop computers, although volumetric navigation typically requires a high-end computing system. Third, the system should be scalable to accommodate an increasing amount of case data and increasing number of trainees. Fourth, management and updating the case data should not be demanding, even after the distribution of the software program for case reviews to many trainees. This was to cover a variety of fields in radiology and to keep pace with the rapid changes in knowledge and technology. Fifth, the system should encourage trainees to participate throughout the case review course. Sixth, the system should simulate a current busy practice environment.

The system comprised an image viewing system, case registry server, and client case review program (Fig. 1). We selected a commercially available cloud-based image viewing system because it met the important system requirements of volumetric navigation, accessibility, and scalability. We developed the case registry server to facilitate the maintenance of the case data. We used an interactive case challenge format for the client case review program to encourage participation by trainees (17); and a timer display was used in the client case review program to simulate a busy practice environment.

The image viewing system comprised a cloud image server (iNtuition CLOUD; TeraRecon, San Mateo, CA, USA) located in Seoul, Korea and a thin-client image viewer (AquariusNET v4.4.11.5; TeraRecon). The cloud image server stored the volumetric image datasets and processed the images as requested by the trainees. The client image viewers in the trainees' computers displayed the final processed images received from the server. The trainees could utilize all the advanced visualization techniques that were available in the image viewing system (18), including multiplanar reformation, volume rendering, and fly-through.

Owing to the use of a cloud-based image viewing system, we could avoid the hardware burden and an arduous installation process on many computers of trainees, and the cumbersome distribution of large volumetric image datasets to the trainees' computers, all of which might have discouraged the trainees from participating. In addition, the cloud image server provided a platform where the case authors could collaboratively upload, maintain anonymity, and annotate the image datasets via the Internet.

The case registry server was developed on the Ruby on Rails 3.1 platform (Rails Core Team, http://rubyonrails.org/core/). The server ran on a Linux system (Ubuntu 12.04 LTS; Canonical, London, UK) equipped with a 3.1 GHz 4-core Xeon CPU (Intel, Santa Clara, CA, USA) and 16 GB of main memory. The case registry server interacted with a web server (Apache 2.2.22; Apache Software Foundation, Los Angeles, CA, USA) and a database management system (MySQL 5.5.34; Oracle, Redwood Shores, CA, USA), which also ran on the Linux system. The case registry server stored the case data, including brief clinical findings, case questions, correct answers, and the identification numbers of the image datasets stored in the cloud image server, as well as the account information and response data of the individual trainees (e.g., their submitted answers and the review time for each case). The case registry server provided a web-based case management tool that allowed the case authors to upload and maintain the case data. The case registry server also provided a download link for the client case review program.

We developed a client case review program, which provided user interfaces for the trainees on the .NET Framework 4 platform (Microsoft, Redmond, WA, USA). In order to encourage the trainees to participate, the client case review program was developed in an interactive case challenge format. For each case, trainees were asked to answer several case questions, after which they were provided with the correct answers and key images, as well as annotations of important findings. A timer measuring the review time for each case was displayed to prompt the trainee to review the case as rapidly as possible, thereby simulating a busy practice environment.

In the pilot test, we decided to use cases of low-dose abdomen CT for the diagnosis of acute appendicitis. This application was considered suitable for demonstrating the advantages of the developed system. Appendicitis is a very common medical emergency and the preoperative CT reports are often made by non-expert radiologists (1920), but the reports play a critical role in patient disposition (21). An educational program that systematically familiarizes radiologists with the noisy low-dose CT images would facilitate the dissemination of the low-dose CT technique, which is promising (2223), although not yet widely accepted as standard practice (2425). While the image viewing system comprised many advanced visualization techniques, we selected multiplanar (26) sliding-slab average intensity projection (2728) in the pilot test, as the technique is known to help the diagnosis of appendicitis (293031).

We randomly selected 30 low-dose (estimated effective dose of 2 mSv) CT cases, including 14 with confirmed appendicitis and 16 without appendicitis, from the database of a previous study (22). The case authors prepared the case data, including brief clinical findings, case questions, and correct answers, and then uploaded them to the case registry server. Four questions based on a report form used in recent studies (2223) were provided for each case: the visibility of the appendix (on a 3-point Likert scale), likelihood of appendicitis (on a 5-point scale), likelihood of appendiceal perforation (on a 3-point scale), and any potential alternative diagnosis (as an open question).

Each CT image dataset had a section thickness of 2 mm and a reconstruction interval of 1 mm. The image datasets were anonymized and uploaded to the cloud image server in the Digital Imaging and Communications in Medicine format. A second copy of the image dataset was generated and stored in the cloud image server to provide feedback to the trainees. In the second copy, the case authors selected the set of transverse and coronal images that best depicted the appendix, and they annotated the selected images with arrows on the appendix and other important image findings. The case authors could also define the zooming, panning, and windowing as appropriate. This information was stored in the cloud image server, which allowed the trainees to review the annotated image datasets by starting from the selected images with the predefined setting.

The pilot system prompted trainees to review the 30 cases. The order of the 30 cases was randomized for each trainee. For each case challenge, the client case review program briefly displayed the clinical findings to the trainee. The system led the trainee to navigate through the volumetric image dataset using the multiplanar sliding-slab average intensity projection technique in the client image viewer. After the trainee submitted their answers, the client case review program displayed the correct answers and key images that were retrieved from the case registry server. The trainee was then asked to review the annotated image dataset to identify the appendix and other important image findings on the client image viewer, again using the multiplanar sliding-slab average intensity projection technique. After the trainee finished all the 30 cases, the client case review program displayed the trainee's diagnostic accuracy and average review time per case. The accuracy was defined as the percentage of correct diagnosis or exclusion of appendicitis based on the likelihood of appendicitis with grade 3 or higher as a positive response. The trainee was asked to reassess his or her failed cases until the revised accuracy exceeded 95% (Fig. 2, Movie 1 in the online-only Data Supplement). The detailed task flow during a case review cycle in the pilot system was provided in Fig. 3.

The pilot system was deployed as a hands-on workshop course in the Advanced Imaging Multimodality Schools and Seminars of European School of Radiology (32), which was held in Seoul, Korea in July 2013. In total, 49 radiologists with limited previous experience in low-dose abdomen CT participated in the course by using their own laptops to run the client case review program. Due to time constraints (40 minutes assigned to the hands-on workshop, including program installation), only 5 (10%) participants could complete all the 30 cases during the workshop course. The median number of cases reviewed by individual participants was 13 (interquartile range, 8-21) and the median per-case review time for individual participants was 91 (interquartile range, 67-119) sec. The course was rated as very good (mean ± standard deviation, 4.7 ± 0.7) according to the participants' responses regarding the satisfaction of the workshop course using a 5-point Likert scale (1, very poor; 2, poor; 3, satisfying; 4, good; 5, very good). The workshop course is available for public use, and interested readers can try it without charge after registering at http://www.locat.org/users/new?locale=en&no_phone=1&no_hospital=1&no_career=1&no_career_year=1&no_account=1&group_id=33.