With the widespread use of picture archiving and communication system (PACS) in the hospitals, the amount of medical image data is rapidly increasing. Thus, the more efficient and effective retrieval methods are required for better management of medical image information.

There are two ways that medical images are retrieved, text-based and content-based methods [1]. In text-based image retrieval, the images are retrieved by the manually annotated text descriptions and traditional database techniques to manage images are used. In content-based image retrieval, the images are retrieved on the basis of features such as color, texture, shape, and so on, which were derived from the images themselves.

So far, a variety of medical image retrieval systems have been developed using either method (text-based or content-based) or combining two methods; a rough classification of medical image retrieval methods is displayed in Figure 1.

Therefore, we described two technological approaches for medical image retrieval - past development and present status, and future perspectives.

Text-based image retrieval system can be traced back to 1970s. Text-based image retrieval system is prevalent in the search on the internet web browsers. Although text-based methods are fast and reliable when images are well annotated, they cannot search in unannotated image databases. Moreover, text-based image retrieval has the following additional drawbacks, it requires time-consuming annotation procedures and the annotation is subjective [2]. Text-based query commonly results in irrelevant images (Figure 2). Thus, to support effective image searching, retrieval methods based on the image content were developed.

In content-based image retrieval systems, images are indexed and retrieved from databases based on their visual content (image features) such as color, texture, shape, etc. Commercial content-based image retrieval systems have been developed, such as QBIC [3], Photobook [4], Virage [5], VisualSEEK [6], Netra [7]. Eakins [8] has divided these image features into three levels as followings: 1) Level 1 - Primitive features such as color, texture, shape or the spatial location of image elements. Typical query example is 'find pictures like this'; 2) Level 2 - Derived attributes or logical features, involving some degree of inference about the identity of the objects depicted in the image. Typical query example is 'find a picture of a flower'; 3) Level 3 - Abstract attributes, involving complex reasoning about the significance of the objects or scenes depicted. Typical query example is 'find pictures of a beautiful lady.'

The majority of content-based image retrieval systems mostly offer level 1 retrieval, a few experimental systems level 2, but none level 3.

Commonly used image features for content-based image retrieval were followings: 1) Color: color is one of the visual cues often used for content description, but most medical images are grayscale. Thus, color features are not used for medical image retrieval; 2) Texture: texture features mean spatial organization of pixel values of an image and used in standard transform domain analysis by tools such as Fourier transform, wavelets, Gabor or Stockwell filters. In the medical images, texture features are useful because they can reflect the details within an image structure; 3) Shape: shape feature has broad range of visual cues such as contour, curve, surfaces, and so on. Recently, many methods measures similarity between images using shape features has been developed.

Formerly developed commercial content-based image retrieval systems characterized images by global features such as color histogram, texture values and shape parameters, however, for medical images, the systems using global image features failed to capture the relevant information [9]. In medical images, the clinically useful information is mostly highly localized in small areas of the images, that is, the ratio of pathology bearing pixels to the rest of the images is small. Thus, the global image features such as color, texture, shape, etc. cannot effectively characterize the content of the medical images.

Initially, medical images were included in the content-based retrieval systems as a subdomain for trials [10,11]. Recent trials for content-based medical image retrieval were ASSERT system [12] for high resolution computed tomography (CT) images of the lung and image retrieval for medical applications (IRMA) system [13] for the classification of images into anatomical areas, modalities and viewpoints. Flexible image retrieval engine (FIRE) system handles different kinds of medical data as well as non-medical data like photographic databases [14].

In ASSERT system, the system lets the user extract pathology-bearing regions in lung images and these regions are then characterized by their grayscale, texture, and shape attributes, which are used for image retrieval.

The IRMA system (Figure 3) is a generic web-based x-ray retrieval system. It allows the user to extract images from a database given an x-ray image query. Local features and similarity measures are used to compute the nearest images. This system also developed a classification code for medical images to classify medical images based on four axes (modality, body orientations, body region, and biological system) [15]. The local features are derived from the previously classified and registered images that have been segmented automatically. IRMA analyzes content of medical images using a six-layer information model: raw data, registered data, feature, scheme, object, and knowledge. IRMA lacks the ability for finding particular pathology that may be localized in particular regions within the image.

In FIRE system (Figure 4), query by example image is implemented using a large variety of different image features that can be combined and weighted individually and relevance feedback can be used to refine the result [16].

The integration of content-based image retrieval system into the PACS system has been proposed [17,18]. However, an effective and precise medical image retrieval still remains a problem and recent researches aim at developing techniques that overcome this point.

Currently, work is underway to create tools to enable semantic annotation of images using ontologies, providing an opportunity to enhance content-based image retrieval systems with richer descriptions of images. The suggested ontology systems are controlled terminologies such as Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) [19], Foundational Model of Anatomy (FMA) [20] and Radiology Lexicon (RadLex) [21].

Considering the intrinsic difference between the text and image in representing and expressing information, there have been approaches to combine the text-based and content-based image retrieval. Techniques that perform the text-based method first [25,26] and two methods at the same time [27,28] were studied.

The hybrid image retrieval systems to incorporate external knowledge that is encoded in lexicons, thesauri and ontologies were suggested [29].

The new project, Annotation and Image Markup (AIM) for medical image annotation and markup is being developed to make all the key semantic (Figure 8) content of images machine-readable using controlled terminologies (mainly RadLex) and image markup standards [30].

In the domain of medical imaging informatics, a huge amount of image data is being produced. A lot of work has already been done to improve the image retrieval systems. One is text-based approach and the other is content-based. Each method has its own advantages and disadvantages. Text-based method is widely used and fast, but it requires precise annotation. Content-based approach provides semantic retrieval, but effective and precise techniques still remains elusive.

Recently, a new controlled vocabulary, RadLex was developed to provide standardized terms for images and combined text and content-based methods were developed. For improved semantic image retrieval, it is proposed that image retrieval techniques be effectively integrated with external knowledge, annotation tools, and image markup systems.

In the near future, it is expected that the semantic contents of medical images will be totally computationally-accessible and reusable by the application of ontology and the development of new convenient tools.