Checklist for DP quality assurance program
Preface
• This checklist was developed based on the ‘Recommendations for Pathologic Practice Using Digital Pathology: Consensus Report of the Korean Society of Pathologists’ (2020.10) [
2]. It is highly recommended to refer to the recommendations for understanding the background information related to DP (
Supplementary Table S1).
• “DP” used in this checklist for DP QAP refers to a relatively narrow perspective of DP, that is primary diagnosis or consultation using WSIs via displays such as monitors instead of a microscope, by digitalizing all or a part of pathological samples. The DP QAP of the KSP is intended only for institutions conducting primary diagnoses using DP systems.
• “DP” is a dynamic imaging environment (or academic field related to this environment) that involves the acquisition and management of pathologic information, by converting microscopic glass slides into digital files, and the pathologic diagnosis and interpretation of those images using display devices (e.g., monitors). The scope of the application includes education, diagnosis, research, image analysis, archiving, retrieval, expert consultations, and data sharing. In this checklist, its meaning is limited to the primary diagnosis or consultation as part of the pathologic diagnosis process.
• “The digital pathology system” is a computer system that enables the collection, management, and interpretation of pathologic image data by digitalizing glass slides. It includes a whole slide scanner (WSS), computer workstation, operating and managing software (scanner operation software, image viewer, and image analysis software), and network system (server and network environment).
• “Telepathology” is a digital or real-time pathologic image communication environment using wired or wireless networks or a related academic field. Telepathology could be used either for consultation with specialists in a distant location or for the diagnosis of samples in a remote facility.
• “Whole slide image/imaging (WSI)” is a single high-resolution image file or associated technology scanned from a single glass slide using a WSS. WSI can be considered a high-resolution copy or mirrored image of a glass slide.
• “Focus stacking (Z-stacking)” is an image-processing technique that displays multilayer digital images acquired at varying focus levels to obtain a much greater depth of field. In samples with many 3-dimensional microstructures and cell clusters, such as cytology slides, it is difficult to obtain the appropriate depth of field with a single focus. Multiple images at slightly different levels of the z-axis should be combined using various image processing methods to generate a single image file.
• “Pathology picture archiving and communication system (Pathology PACS)” is a system that archives, processes, and transmits DP images in accordance with international standards such as the Digital Imaging and Communications in Medicine (DICOM) format. Pathology PACS consists of an image-viewing and archiving software, a mass storage device, and a computer hardware system.
• “A laboratory information management system, or laboratory information system (LIS)”, is a software system designed to manage information related to the overall operation and management of a laboratory.
• “Validation” describes the process of confirming whether equipment, reagents, and test methods that have already been verified can be appropriately applied to an individual laboratory according to certain standards before implementation. Validation should be conducted using documents that provide a high level of assurance.
QC of DP systems
W.01.001 Do you have laboratory guidelines for DP covering overall DP workflow and is it easily accessible? (4)
Yes ( ) No ( )
* Laboratory DP guidelines can include the following content according to the scope of application and individual needs.
- Principles/guidelines for the operation of DP systems (Standard Operating Procedures, SOP)
- Information on personnel managing and using DP systems (organizational charts and roles)
- Principles/guidelines for the maintenance and repair of all facilities and equipment in a DP system
- Principles/guidelines for the collection, storage, use, and disposal of digital image data
- Principles/guidelines for personal information protection, management, and security
- Principles/guidelines related to telepathology and the use of portable devices
- Principles/guidelines for coping with emergencies and disasters
- Principles/guidelines for education of all DP system users
- Principles/guidelines for QC in DP systems
- Principles/guidelines for validation of DP systems
- Principles/guidelines for the other operation in DP systems
W.01.002 Does the DP director regularly review and check all the guidelines for the DP system? (2)
Yes ( ) No ( )
* The director of the DP system should review and confirm all the guidelines at least once a year, leaving a date and signature.
W.01.003 If the guidelines are changed, added, revised, or partly discarded, does the person in charge of the DP system review and confirm their content? (2)
Yes ( ) No ( )
* When major equipment constituting the DP system is newly introduced/changed, the content of the corresponding guidelines should be changed or added accordingly.
* The director of DP systems should review and leave a date and signature whenever there is a change in the guideline content.
* The contents of the guidelines that are changed or discarded should be retained for at least 2 years.
* Any changes, abolitions, or additions to the guidelines should be dated.
W.01.004 When the DP system director changes, does the new director review and confirm all the guidelines? (4)
Yes ( ) No ( )
* The new director should review and confirm the guidelines, and leave the date and signature.
W.01.005 Is there a system to ensure that all employees using the DP system are aware of the guidelines? (2)
Yes ( ) No ( )
* In-house education for the guidelines should be implemented, and trainees should leave training dates and signatures.
W.01.006 Do you have an internal QC system for the DP system in place and implemented? (32)
Yes ( ) No ( )
* Internal QC of DP systems should include:
- Regular validation of DP systems
- Documentation, causal analysis, and statistics on errors that occur during whole slide scanning
- Maintenance check report for overall equipment of DP system
- Documentation and troubleshooting of errors between DP systems and LIS (or hospital information systems, HIS)
- Inspections related to privacy and personal information security
- Guidelines for follow-up measures of errors in internal QC
W.01.007 Do you regularly conduct internal QC of DP systems? (8)
Scoring: ______________
* Internal QC should be implemented periodically, daily, weekly, monthly, and quarterly, in accordance with internal regulations as well as validation after major changes in the components of the entire DP system due to the introduction of new equipment and others. If it is conducted irregularly (when 20%–80% is satisfied), half of the score (4 points) is given, and if it is conducted with less than 20%, 0 points are given.
W.01.008 Does the person in charge of the DP system review and resolve the internal QC results in documentation? (8)
Yes ( ) No ( )
Personnel in DP systems
W.02.001 Are the personnel managing the DP system adequate in number? (4)
Yes ( ) No ( )
* The management team of DP systems consists of a variety of people, including pathologists, technicians, personnel who perform whole slide scanning, IT managers, and others.
* Information on personnel managing DP systems (such as organization charts and roles) should be thoroughly prepared and updated.
* Personnel managing DP systems should understand the subjects related to implementation, management, and maintenance of DP systems. DP personnel should continuously educate themselves for SOP and understand the difference in the diagnosis process between DP systems and conventional microscopy.
* The appropriate number of personnel managing the DP system may be determined according to the scale and workflow of the DP system in individual institutions.
W.02.002 Is the education/training for personnel using/managing DP systems being documented? (4)
Yes ( ) No ( )
* Training programs for new personnel using/managing DP systems should be prepared, and regular training should be provided for existing members.
* New training should be provided to employees using/managing digital pathology systems in the following circumstances: (1) introduction/replacement of key equipment in the DP system and corresponding changes in the laboratory guidelines, (2) change in the operational scope and method in the DP system, and 3) validation is newly performed.
W.02.003 Are there personnel primarily assigned to whole slide scanning? (8)
Yes ( ) No ( )
* Scanning personnel are required to undergo a certain training period according to the laboratory DP guidelines.
The hardware and software used in DP systems
W.03.001 Is the WSS in the DP system approved by the Ministry of Food and Drug Safety? (4)
Yes ( ) No ( )
W.03.002 Is the performance of the WSS regularly checked and maintained by the manufacturer? (4)
Yes ( ) No ( )
W.03.003 Is the performance of the WSS and the quality of scanned WSIs regularly evaluated by DP personnel? (4)
Yes ( ) No ( )
W.03.004 Does the image database system guarantee that the identification information of the glass slide matches that of the digital image? (2)
Yes ( ) No ( )
W.03.005 Even if the version of the image-archiving software changes, is it still available to view and use the archived data with updated software without any technical difficulty? (2)
Yes ( ) No ( )
W.03.006 Does the image storage method take a form of backup or mirroring? (4)
Yes ( ) No ( )
W.03.007 Do the image display devices (e.g., monitor) have the appropriate image quality for primary diagnosis? (2)
Yes ( ) No ( )
W.03.008 Does the image viewing software properly implement overview image, annotation, and image comparison functions to suit the pathological workflow? (2)
Yes ( ) No ( )
* Considerations and recommended functional requirements for image display devices and image viewing software (
Supplementary Material S6) [
2].
W.03.009 Are DP systems appropriately linked to the LIS, electronic medical records (EMR), or HIS? (4)
Yes ( ) No ( )
Operation, management, and validation of DP systems
W.04.001 Is the laboratory SOP well prepared stating the following? (10)
1) Types of samples and staining methods to be used for primary diagnosis using a DP system (scope) Yes ( ) No ( )
2) Components of the DP system and SOP for each device Yes ( ) No ( )
3) Role and training records of the personnel using and managing DP system Yes ( ) No ( )
4) Guidelines on consultation and management of the referred patient samples using the DP system Yes ( ) No ( )
5) Guidelines for the storage, management, and disposal of DP image data Yes ( ) No ( )
6) Guidelines for privacy and information security Yes ( ) No ( )
7) Principles/guidelines related to telepathology and portable device use Yes ( ) No ( )
8) Guidelines for validation Yes ( ) No ( )
9) Plans/programs for QC Yes ( ) No ( )
10) Principles/guidelines for emergencies and disasters Yes ( ) No ( )
* 1 point to each item.
W.04.002 Is in-house validation of newly introduced devices of the DP system being conducted and documented? (12)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S8) [
2].
W.04.003 Is the validation study conducted under conditions that are consistent with the clinical use intended by the DP system manufacturer? (4)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S9) [
2].
W.04.004 Is the validation study designed to be as similar as possible to the actual clinical settings in which the technology will be used? (4)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S10) [
2].
W.04.005 Does the validation study cover the entire DP system? (4)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S11) [
2].
W.04.006 When there are significant changes in the composition of the DP system, is the revalidation on the whole DP system being conducted? Do you have guidelines for this? (8)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S12) [
2].
W.04.007 Is the validation intended to be conducted by at least one pathologist who has acclimated to the DP system? (4)
Yes ( ) No ( )
* For general hematoxylin and eosin (H&E) slides, frozen sections, cytology slides, and blood smears, validation must be performed on at least 60 samples for a single applicable field. For additional applicable fields such as immunohistochemical stains and special stains, validation could be performed by adding an additional 20 samples.
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S13) [
2].
W.04.008 Is the validation carried out using a comparative analysis of concordance between microscopic and WSI-based diagnoses made by a single observer (intra-observer variability assessment)? (4)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S14) [
2].
W.04.009 Did you have a washout period of at least 2 weeks to minimize the influence of recall bias during the validation? (4)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S15) [
2].
W.04.010 During validation, do you assess the data integrity of the image acquisition by verifying whether all tissues on the glass slide have been properly scanned to form the digital image? (8)
Yes ( ) No ( )
* Guidelines and considerations for validation needed for the implementation of DP systems and internal QC needed during operation (
Supplementary Material S16) [
2].
W.04.011 Is additional validation being conducted on the samples for the detection of microorganisms (e.g., Helicobacter pylori), the cytology slides (cell smears, liquid-based cytology, or blood smears), and the cases suspicious for lymphoreticular neoplasms? (4)
Yes ( ) No ( )
W.04.012 Are all the errors, its statistics, and its cause analysis being recorded during the whole slide scanning? (4)
Yes ( ) No ( )
W.04.013 Are WSIs being scanned at a minimum magnification of 20× (in case of H&E slides)? (4)
Yes ( ) No ( )
W.04.014 Do you manage regular inspection results for the image display device (e.g., monitor)? (4)
Yes ( ) No ( )
* Considerations and recommended functional requirements for image display devices and image viewing software (
Supplementary Material S18) [
2].
W.04.015 Is the data being stored according to the data-preservation period determined by the individual laboratory? (4)
Yes ( ) No ( )
Personal information protection and information security of DP image data
W.05.001 Do the guidelines include the instruction and regulation on the collection and management of personal information and information safety? (16)
Yes ( ) No ( )
* Digital pathological imaging data should be stored systematically with the help of information processing experts and easily found as needed, and methods for regulating access to appropriate security and information should be established to protect privacy.
* There should be appropriate guidelines for data sharing and wired/wireless transmission that include the extent to which personal information is shared and how it is protected.
* Issues related to telepathology, firewalls, protection of personal information, and mobile device use (
Supplementary Material S20) [
2].
W.05.002 Does the DP guideline specify the external personnel or institutions that can access medical information and data using DP systems? (4)
Yes ( ) No ( )
W.05.003 Is there a person in charge of information security for the DP system? (4)
Yes ( ) No ( )
W.05.004 Is there a suitable system to prevent the loss of personal medical information in the case of hardware or software failure in the DP system, other emergencies, or disasters (e.g., power outages due to natural disasters)? (4)
Yes ( ) No ( )
* Measures and guidelines should be implemented to prevent the spread of failures or disasters that cause problems, plan periodic data/information backups, and recover corrupted data/information.
Quality assurance program trial for DP
Four leading hospitals, Seoul St. Mary’s Hospital, Seoul National University Hospital, Yongin Severance Hospital, and National Cancer Center participated in the QAP trial for DP. At the 46th Annual Spring Meeting of the KSP and the 73rd Annual Autumn Meeting of the KSP, eight hospitals, including the Samsung Medical Center, Yonsei Severance Hospital, Kangbuk Samsung Hospitals, and Seoul Asan Medical Center, in addition to the aforementioned hospitals, shared institutional experience on DP implementation and feedback after participating in the trial [
17-
24]. In addition, 11 DP directors, WSS technicians, and DP users from these hospitals submitted a feedback survey.
Current status of DP implementation in Korea
As of 2022, 40 out of 214 pathology laboratories (18.7%) implemented DP for primary diagnosis, archiving, consultation, and research and 2–10 more laboratories are expected to implement DP within next 1–2 years. Majority of the laboratories with DP is using DP for research, followed by archive, and consultation. Only about 15% of these laboratories are being estimated to use DP for primary diagnosis. The feedback survey showed that these pathology laboratories are currently processing 40%–100% of the total H&E examinations digitally (5,000 to 50,000 cases annually per lab). These labs are scanning mostly a part of H&E, some special and immunohistochemical stains, rarely immunofluorescence, silver in situ hybridization, or fluorescence in situ hybridization and are planning to expand the coverage.
QAP trial results
Four leading hospitals, Seoul St. Mary’s Hospital, Seoul National University Hospital, National Cancer Center, and Yongin Severance Hospital, participated in the QAP trial for DP.
Table 5 summarizes the results. Average score was 209.75 out of 216 ranging from 204 to 216 with 1 to 2 missed items in each lab. All checklist items according to the hardware and software used in DP systems and personal information protection and information security of DP images were well prepared by all laboratories. Missed items were W.01.007, W.02.003, W04.001.004, W.04.011, and W.04.014, which do not affect the integrity of DP system seriously and can be improved in the future.
Feedback survey after DP QAP trial
Jang K-T from Samsung Medical Center highlighted focusing error, robotic motor dysfunction, localization error of the region of interest, software upgrades issue, compatibility between server and storage, and security software troubles as major issues during DP implementation [
20]. He mentioned fast and convenient accessibility and reduced time of tumor annotation for additional ancillary tests, convenient application in multidisciplinary conferences, and education as the major benefits of the DP system. He also pointed out that the current DP systems fall behind the users’ expectations because there are still many problems in incorporating the system into the conventional pathology workflow, in which thorough QAP is required.
Lee KB from Seoul National University Hospital mentioned delayed turn-around time (TAT) due to scanning, unfamiliarity with WSIs, and increased workload of pathology lab personnel as the most worried aspects during DP implementation [
24]. However, it turned out that the TAT could be reduced as the DP systems were expanded to the whole sample, including immunohistochemistry. The number of active user pathologists could increase over time, and gradual expansion of the DP system could increase the flexibility of the rapid workload increase. She highlighted the increased patient safety of the DP system, which is based on easy access to the archive, fast comparison of previous examinations, and instant intra- or interdepartmental consultation. She also mentioned that this could be very important for the general quality of pathologic diagnosis services.
Hong SW from Yongin Severance Hospital, Yonsei University introduced the advanced facilities of the digital hospital of Yongin Severance Hospital, including the Integration and Response Space, Real Time Location System, artificial intelligence-based chest X-ray interpretation system, mobile PACS, 5G mobile network-based server and storage, and DP solution, including voice recognition system, digital gross lab recording and photography system, and barcode/QR code sample tracking system [
19]. She also shared that most errors were found during the pre- and post-scanning quality checks, such as localization error of the region of interest, foreign body/air bubble during the mounting, and handling errors by lab personnel. Over a year and a half after DP implementation, the error rate of her laboratory decreased from 0.61 to 0.02 as the DP coverage increased from 75% to 100%. She also mentioned witnessing a significantly reduced TAT, especially for reviewing prior exams and intra-/interdepartmental consultation, as well as a possible reduction of workload and slide storage space.
Some surveyees did not seem to understand the DP checklist items properly and they raised questions, the need for revision of some items for better clarification, or the need for more educational opportunity on DP from the KSP. Overall, the surveyees replied that the DP checklist was helpful in preparing internal QC guidelines on DP and laboratory QAP on DP.