Symptomatic gallstones, including biliary colic, are a common cause of emergency department visits [1,2]; however, studies evaluating the natural history of the biliary colic are limited. Notably, recurrent biliary colic is common; a Canadian study demonstrated that 6.7% of patients awaiting cholecystectomy for biliary colic required admission to the hospital prior to their scheduled surgery due to recurrent biliary symptoms [3]. In one cohort of 71 patients in Texas, 37.1% of the patients who did not undergo early scheduled cholecystectomy following an episode of biliary colic had a repeat emergency department visit and 17.7% had two or more additional visits [4]. However, population-based data from New York state suggests that almost half of the patients with biliary colic will not return to the emergency department or have scheduled surgery within 5 years [5]. Therefore, it is unclear which biliary colic patients will have recurrent symptoms and thus benefit from a scheduled surgery.

In contrast to biliary colic, several large studies have demonstrated a high rate of recurrent disease in the non-operative management of acute cholecystitis [6,7]. Further analyses have demonstrated that early surgery in acute cholecystitis results in improved outcomes including decreased post-operative complications, decreased conversion to open surgery, decreased re-operation rates, shorter length of hospital stay, and decreased rates of bile duct injury [8,9]. Similar findings and adequate data on biliary colic patients would help to make informed recommendations regarding the timeliness of surgical intervention in this subset of patients. Therefore, there is a need for a prospective evaluation of patients with biliary colic; a detailed and prolonged period of close clinical observation would clarify the rate of symptoms and predictors of symptoms. However, prospective studies are costly. For this reason, retrospective analyses of administrative health databases have been routinely used. Limitations of previous attempts to elucidate the natural history of biliary colic using administrative health data is that these studies rely on the identification of biliary colic via administrative diagnostic codes in isolation and may be prone to misclassification. The purpose of our study was to validate the use of International Statistical Classification of Diseases and Related Health Problems, 10th Revision, with the Canadian modification (ICD-10-CA) during emergency department visits in adequately differentiating biliary colic patients with gallstones from those with an alternate biliary diagnosis such as cholecystitis or common bile duct stones.

We performed a retrospective validation study using data routinely provided to the National Ambulatory Care Reporting System (NACRS) at two large academic hospitals in Toronto, Ontario, Canada. The province of Ontario is Canada’s most populous province with a population of approximately 14.6 million people. Ontario has a universally accessible, single-payer healthcare system.

NACRS captures demographic, diagnostic, procedural, and discharge data for all emergency department visits in Ontario. Trained health record coders read through the medical charts to code up to 16 diagnoses, these data are used to generate the NACRS records. NACRS is routinely used for health services research given the capacity to follow all healthcare interactions across any Ontario hospital, and it is the data source for a natural history of biliary colic evaluation in Ontario, Canada.

We reviewed charts of all adult patients presenting at the emergency department with ICD-10-CA codes K80.2x (calculus of gallbladder without cholecystitis) or K80.8x (other cholelithiases) as the most responsible diagnosis between January 1, 2012 and December 31, 2018. These ICD-10-CA codes were chosen because they were most likely to accurately reflect uncomplicated biliary colic with gallstones to the exclusion of other biliary pathologies based on coding definitions set out by ICD-10-CA (Table 1). Emergency department visits were selected to ensure patients were presenting with symptoms and that asymptomatic stones were not captured.

The deterministically linked patient charts were then individually abstracted and compared to the assigned ICD-10-CA code to assess the level of agreement for diagnosis coding. The study protocol was approved by the St. Michaels Hospital Research Ethics Board (no. 19-296) (Toronto, ON, Canada); informed consent was waived by the board. Individuals were excluded if they had previous bariatric surgery, were pregnant, or had no available imaging records.

All charts were abstracted by a clinically trained reviewer (M.M.) using a standard case reporting form and any discrepancies were resolved by a second reviewer (D.G.). Demographic, laboratory, and imaging data were gathered for each patient encounter. Data made available following the initial patient encounter in the emergency department, such as follow-up imaging, were not included. Patients were considered misclassified (i.e., not biliary colic with gallstones) if there was evidence of complicated biliary disease on clinical, laboratory, or imaging grounds. Cholecystitis and cholangitis were defined according to the 2018 Tokyo Guidelines [10,11] and pancreatitis was defined by the diagnostic criteria stated in the Atlanta classification [12]. In addition, patients were considered misclassified if there was an absence of gallstones or gallbladder on imaging, or previously documented cholecystectomy. Cases from the medical records found to be correctly classified by this method relative to abstracted records were considered “true positive” results. The misclassified cases were labeled as “false positive” results. Using this value and the total number of cases identified by the administrative codes, it was possible to determine the positive predictive value (PPV), together with its 95% confidence interval (CI) [13]. Since the examination of patients with ICD-10-CA codes for biliary colic only is highly selective, it would be expected that the proportion of patients with the diagnosis of interest would be significantly greater than that of the baseline population. Since PPV increases as the prevalence of the disease in a population increases, we can assume that this represents a “best-case scenario” for PPV in comparison to any population-based analysis.

A total of 991 patients were identified during the study period. The patients’ mean age was 45 years, and 683 (68.9%) were female. Using chart review as the gold standard, the PPV of administrative diagnosis codes in isolation for the diagnosis of biliary colic with gallstones in this population was 73% (95% CI 73%–74%). Amongst those misclassified, the most common reason was a lack of gallstones on imaging (49.8%). The second most common reason was meeting diagnostic criteria as per the Tokyo guidelines for acute cholecystitis (27.8%). Eleven percent of the misclassified patients had a prior documented cholecystectomy and 12.2% had choledocholithiasis. Three patients (1.1%) had more than one disqualifying finding (Table 2). There was no association between age or sex and misclassification.

The use of administrative codes for large studies is increasingly common and can yield an enormous amount of information using readily available data. However, this method relies on the assumption that the administrative codes used to identify a diagnosis or procedure were accurately assigned. The vast majority of administrative database research does not use codes that have been adequately validated to ensure that they represent the characteristic of the study interest [14]. This calls into question the validity of any results derived from these investigations and therefore is an important subject pertaining to the responsible use of any subsequently derived information.

Many surgical diagnoses are readily identifiable from administrative codes and have suitably high PPVs, such as malignant neoplasm of the colon (PPV 86%), or appendicitis (PPV 96%) [15]. We found that the predictive value of diagnostic codes for uncomplicated biliary colic with gallstones was suboptimal, with only 73% of cases examined carrying a true diagnosis of biliary colic with gallstones. Because a clinical diagnosis of biliary colic with gallstones is not made by the presence of a particular laboratory or imaging finding, but rather by clinical assessment and the exclusion of complicated biliary pathology, retrospective identification of the disease, even by trained abstractors, is difficult.

Unfortunately, large Canadian administrative databases routinely used for research such as NACRS do not collect the additional information that would be necessary to increase the diagnostic accuracy of ICD-10-CA codes such as imaging reports. Therefore, population-based administrative health data for the purposes of research should be complemented by chart abstraction, which is rich in objective clinical data such as imaging reports and blood work, to adequately identify cases of biliary colic with gallstones and other similar pathologies.

There are several limitations to this study that must be considered in its evaluation. Our study included only two large academic centres in Toronto, therefore, generalizability may be limited. Additionally, because there are no widely used diagnostic criteria for biliary colic with gallstones, the diagnosis must be verified through the exclusion of other similar diagnoses rather than by verifying the presence of biliary colic. In addition, characteristics of pain were not collected as they were found to be variably and not systematically present in the medical chart. The inherent complexity of the diagnostic process thereby increases the likelihood of inaccurate coding. Since the abstracted cases were selected using the codes associated with the diagnosis of interest, there was no comparative cohort without biliary colic in our study. As a result, additional test characteristics such as the sensitivity, specificity, and negative predictive value of these administrative codes could not be determined; however, our findings validate the need for additional input for accurate diagnosis.

Our findings suggest that the use of ICD-10 codes as the sole means of identifying biliary colic with gallstones to the exclusion of other biliary pathologies is subject to moderate inaccuracy in administrative health datasets routinely used for research. Previous investigations of biliary colic with gallstones utilizing administrative codes for diagnosis may therefore be prone to unforeseen bias.