Generally, real world objects are captured by a camera and reproduced on a display. AR merges virtual objects with the real world, which requires transformation between the camera and real world coordinates. Before estimating the transformation, the characteristics of the camera must be defined. Pinhole model is a simple camera model that maps the 3D real world onto two-dimensional (2D) coordinates called the image plane. 3D points are mapped onto the 2D image plane by translating the point on a straight line towards the camera center until it intersects the image plane [89]. This mapping is called perspective projection, and the transformation between the image and real world coordinates can be represented as a projection matrix. Thus, camera calibration is the estimation of the projection matrix parameters for a pinhole model [101112].

Patient data for preoperative planning is 3D volume data taken from computed tomography (CT) or magnetic resonance imaging (MRI). Since it provides a view of the internal anatomy and target points for the surgeon, patient data should be registered with respect to a patient of the real world coordinates, which is called patient registration [5]. Point based registration is a reliable solution, where registration is performed with fiducials affixed on the patient. One set, consisting of more than four fiducial points, is registered to another set of corresponding points using a rigid transformation. However, the accuracy of fiducial based registration varies depending on the number of fiducials and measurement quality of each fiducial position, as well as their spatial arrangement [2]. To improve registration accuracy, iterative closest point (ICP) based surface matching is often used in combination with point based registration [131415]. However, careful selection and collection of 3D surface data is critical for final accuracy, usually expressed in terms of target registration error (TRE).

Object tracking is to estimate the spatial position of the camera or marker on surgical instruments, and is an essential component of a medical AR system. In AR tracking, the relative position of an object on the basis of the camera position is generally calculated. When given a calibrated camera with known intrinsic parameters, the relative position can be determined as a set of three or more paired points between the 3D and projected 2D coordinates [1112161718].

Combining these technologies, we can implement an AR system that overlays virtual objects on the endoscope or surgical microscope view.

A surgical navigation system was proposed to guide chronic total occlusion intervention. Conventional intervention for chronic total occlusion of the coronary artery depends highly on 2D X-ray images and the surgeon's experience. Therefore, large displacements between the surgeon's hand-eye coordination may be generated by discrepancies in position or orientation between the patient and the acquired images [2]. This can lead to misidentification of the coronary artery or incorrect positioning of the stenosis on the coronary artery [5]. The proposed system merged the 3D CT angiography model with X-ray images to provide 3D anatomical information to the surgeon [19].

An AR navigation system for bone tumor resection was proposed. Resection of a pelvic tumor is a major surgical challenge because blood vessels are complexly intertwined with nerves. Safety resection margins should be confirmed intraoperatively during the surgery. An AR navigation system would be helpful, providing intuitive visualized information of resection margins [20].

Sinus surgery is also endoscopic surgery. The main problem is difficulty locating a surgical instrument to a specific object seen through the endoscope [6]. Since the access route to paranasal sinuses is complex, complications such as blindness and cerebrospinal fluid leak can occur due to damage of the orbit and skull base. To solve these problems, an integrated system was developed consisting of an AR based surgical navigation system and endoscope holder [21].

For spinal surgery, the most important consideration is to correctly localize the surgical instrument inside the patient's anatomy. Therefore, AR based surgical navigation systems, which assist the surgeon to recognize patient anatomical structures, are widely accepted and have become a very an important research topic in this field [22]. However, although AR provides more intuitive visual information, inaccurate depth perception is a major issue. To improve the depth perception, an integrated VR and AR system was proposed that displayed in a single window with aligned view axes and provided the distance between surgical instruments and target organs [18].