Postoperative pancreatic fistula (POPF) is one of the most fatal complications after pancreatoduodenectomy (PD) and is associated with postoperative hemorrhage, intraabdominal infections, and increased mortality [12]. The incidence of POPF remains high despite advances in surgical techniques and perioperative management [2]. Particularly for clinically relevant POPF (CR-POPF), defined by the grading system of the International Study Group of Pancreatic Fistula (ISGPF) [3], the reported incidence was 14.5% in a previous meta-analysis [4].

Many studies have analyzed the risk factors for POPF. Based on traditional risk factors, there were attempts to develop platforms for predicting POPF [56]. However, due to the limited predictive values of previous platforms [78], we developed new risk prediction platforms using nomograms [9] and artificial intelligence (AI) technology [10]. The nomogram consisting of 6 preoperatively available data provides a better insight into the risk factors and their contributions. The AI model was based on an in-depth analysis of risk factors using machine learning algorithms. The model finally included 16 preoperative and intraoperative variables. Both models are readily available in the form of a calculator at http://popf.smchbp.org and http://popfrisk.smchbp.org.

In this study, we performed external validation of the aforementioned platforms with multicenter datasets to verify the reproducibility and generalizability of the models and to determine their clinical utilities.

This study was approved by the Institutional Review Boards of Samsung Medical Center (Seoul, Korea; No. 2020-09-181), Seoul National University Hospital (Seoul, Korea; No. SNUH 2010-147-116), Ilsan Paik Hospital (Goyang, Korea; No. 2021-06-009), and Boramae Medical Center (Seoul, Korea; No. 30-2021-72). The study was performed in accordance with the Declaration of Helsinki and written informed consent was waived due to its retrospective nature.

The clinical demographics and surgical outcomes of 1,576 patients in the validation cohort are presented in Table 1. The patients’ mean age was 63.6 years, and 697 patients (44.2%) had underlying heart disease including hypertension. Preoperative endoscopic biliary drainage was performed in 825 patients (52.3%). Intraoperatively, 1,056 patients (67.0%) had soft pancreas and the mean diameter of MPD was 3.5 mm. CR-POPF was developed in 270 patients (17.1%).

In the external validation of the nomogram with 1,576 patients, the ROC curve was drawn with the AUC of 0.679 (95% CI, 0.645–0.713; P < 0.001) (Fig. 2A). Fig. 3 shows the values of the AUC after backward elimination. All 16 independent variables were entered into the analysis first and each variable was eliminated one by one. The maximal AUC was 0.672 (95% CI, 0.637–0.706; P < 0.001) (Fig. 2B), including the following 13 variables: the diameter of MPD, BMI, serum albumin, amount of intraoperative fluid infusion, age, preoperative platelet count, tumor location, portal vein resection, coexisting pancreatitis, serum lipase, neoadjuvant radiotherapy, ASA PS classification, and sex.

In the absence of a standardized management protocol for POPF until now, early prediction and recognition are crucial to identify patients at high risk of POPF requiring careful observation [2]. A series of traditional risk scoring systems, such as the original fistula risk score (o-FRS) [5] and the alternative fistula risk score (a-FRS) [6], have been used. However, several studies performed external validation and the predictability varied with the study population [781112]. Therefore, our institution suggested new predictive models using nomograms and AI [910] and this study validated the predictability of the new platforms with an external cohort.

The nomogram including 6 simple variables had an AUC value of 0.709 in the development process [9], and 0.679 in the external validation. A few other nomograms were recently proposed. Huang et al. [13] suggested a new nomogram with the following 3 variables: BMI, the diameter of MPD, and drain fluid amylase level (DFA) on postoperative day (POD) 1. The AUC value was 0.744 in the external validation. Another nomogram by Suzuki et al. [14] also included drain fluid lipase level on POD 1 and decreased rate of DFA, which was defined as a change in levels from POD 1 to 3. The accuracy of the nomogram was 0.810, as stated in the study. Compared to these recently developed nomograms, our nomogram showed limited predictive value in the external validation. The AUC value might have decreased in the process of external validation with highly heterogeneous data collected from different centers. But most of all, the predictive power of the variables in the nomogram could also be limited. Previous studies suggested that DFA is a strong predictive factor of POPF [1516], and DFA on POD 1 is currently widely accepted as an indicator for early drain removal after PD [17]. Considering that our nomogram is composed of only preoperative and intraoperative factors, updating the platform with DFA may improve the accuracy and predictability of the model.

Machine learning is a branch of AI technology designed to enable rapid analytical model building. It has been used in various medical fields including surgery. As far as we know, we invented the first AI-based prediction model for CR-POPF. The most remarkable advantage of machine learning is that it can identify complex structures in high-dimensional data [18] and detect latent variables, which are not directly measured using conventional analytical methods [19]. The new AI model yielded the maximal AUC value of 0.74 with 16 variables, and in the external validation with backward elimination, the AUC was 0.672 with 13 variables. This value is acceptable but leaves considerable room for improvement. First, there were quite a few missing values in both development and validation cohorts. Despite attempts to handle missing data in deep learning [20], it remains the main obstacle to model development. Also, the AI model includes a few variables, which cannot be objectively measured. For example, coexisting pancreatitis or pancreatic texture detected by surgeons intraoperatively can be highly subjective. In order to resolve this issue, several studies are underway to objectively measure those variables using preoperative images and machine learning [2122]. These efforts are expected to establish a foothold for improving predictability of the platforms in the future.

The current study has several limitations. Since this study is based on retrospectively reviewed multicenter datasets, the results might have been affected by selection and information bias. Some potential factors that could influence the development of POPF, such as anastomosis technique and postoperative drain management, varied considerably among the surgeons and the institutions. Also, as previously stated, the platforms were developed and validated with datasets including missing values, which reduces the sample representativeness and complicates the analysis. Further studies with prospectively collected high-quality data are needed to upgrade the platforms with improved predictability.

In conclusion, this study was performed to externally validate the previously developed prediction platforms for POPF. The results suggest the need for improvement and future studies to build better prediction models with higher accuracy.