The rapid evolution of natural language processing has improved large language models (LLMs), strengthening their capacity to comprehend and process complex linguistic information [1]. Consequently, LLMs have gained interest in the medicine for applications such as literature analysis, clinical education, and diagnostic support [2]. As clinical decisions directly influence patient outcomes, the field remains critically dependent on evidence-based practices. Accordingly, the integration of artificial intelligence (AI) tools such as LLMs into clinical workflows demands rigorous, objective validation for accuracy and reliability before deployment [3, 4].

Laboratory medicine serves as a foundational discipline supporting diagnosis, monitoring, and prognosis through the analysis of biological specimens. Standardized interpretation, quantitative methods, and repetitive tasks make it well-suited for data-driven learning and pattern recognition [4]. Rising test volumes and complexity have amplified specialist workloads, highlighting the need for AI tools to improve efficiency and diagnostic precision. In this context, AI can support decision-making, improve consistency, expedite diagnoses, and reduce errors [5].

AI and LLMs are increasingly investigated in laboratory medicine and have demonstrated potential in integrating laboratory data, predicting outcomes, and assisting diagnostics [6, 7]. Wu et al. [8] found AI effectively deliver laboratory information but may fail to address nuanced clinical questions or incorporate recent evidence. Girton et al. [9] reported that ChatGPT outperformed human experts in responding to patient questions on social media; however, they emphasized the importance of clinical validation [9]. Additional studies have explored the performance of LLMs in handling complex scenarios in laboratory medicine [10]. However, most investigations have addressed general clinical queries, and few have evaluated LLMs using the specialized knowledge required for board certification in laboratory medicine. We comprehensively evaluated LLMs using residency examination questions that reflect the rigor of board exams, curated by experts from subspecialties including clinical chemistry, diagnostic hematology, microbiology, immunology, transfusion medicine, and molecular diagnostics. We analyzed performance differences among LLMs and evaluated their capacity to address questions specific to laboratory medicine subspecialties. We also examined error types and sources to provide a comprehensive evaluation of advanced LLMs and their practical applicability in laboratory medicine.

We comparatively evaluated the knowledge and reasoning capabilities of 12 LLMs developed by OpenAI, Anthropic, and Google, using questions from the Korean Laboratory Medicine Residency Examination. Gemini 2.0 Pro demonstrated the highest overall accuracy, followed by GPT-4.5, GPT-4o, and Claude 3.7 Sonnet, collectively indicating strong performance among the latest models. Prior evaluations of GPT-4o in general medical domains using medical licensing examinations revealed accuracies of 90.4% for the United States Medical Licensing Examination (USMLE) [11] and 89.2% for the Japan Medical Licensing Examination (JMLE) [12]. Conversely, GPT-4o achieved only 74.7% accuracy in laboratory medicine, indicating reduced performance in specialized medical fields requiring in-depth domain-specific knowledge.

A time-series analysis by release dates (Fig. 1) revealed dynamic performance trends. Gemini 1.5 Pro underperformed when compared with contemporaneous flagship models (mid-2024), whereas its successor, Gemini 2.0 Pro (released in early 2025) achieved the highest accuracy among all models, representing a rapid, substantial improvement. Among smaller models, Gemini 2.0 Flash uniquely exceeded 70.0% accuracy; however, direct comparisons were constrained by the absence of comparably timed small model releases from OpenAI and Anthropic.

Performance analysis stratified by question difficulty revealed a consistent decline in accuracy with increasing complexity. This corroborates prior findings on LLM performance with other medical examinations, such as the USMLE [13] and JMLE [12]. Notably, the performance of the state-of-the-art models on high-difficulty questions differed significantly, with Gemini 2.0 Pro demonstrating a distinct advantage (67.6% accuracy). Their performance on low- and moderate-difficulty questions was more similar. The relationship between LLM performance and task difficulty is well documented [14], emphasizing the importance of carefully considering the complexity of intended applications when integrating LLMs into clinical workflows. High-difficulty questions within an examination require sophisticated reasoning and profound domain-specific knowledge. Gemini 2.0 Pro’s strong performance in this domain suggests its significant potential for applications in demanding clinical tasks, such as diagnosing rare diseases or interpreting complex clinical scenarios.

A comparison of LLM performance across six laboratory medicine subspecialties (Table 1, Fig. 2B) revealed substantial variation in model accuracy across the subspecialties. The performance of LLMs was reported to differ depending on the specific medical field involved [12, 15, 16] because of factors such as imbalances in LLM training sets or inherent difficulties in effectively representing knowledge within certain specialized domains. Notably, for transfusion medicine, accuracy was markedly reduced across all evaluated LLMs. Even Gemini 2.0 Pro, the top-performing model overall, achieved only 59.1% accuracy for this subspecialty. Similarly, GPT-4.5 and Claude 3.7 Sonnet achieved only 40.9% and 50.0% accuracy, respectively. This underperformance contrasted with the trend observed among human examinees, whose average scores for transfusion medicine did not differ significantly from scores for other subspecialties.

Two primary factors likely contributed to the low LLM accuracy in transfusion medicine. First, proprietary LLM training datasets predominantly contain general data, potentially underrepresenting specialized or recently updated knowledge in this relatively niche field. Second, transfusion medicine often requires a detailed understanding of region-specific regulations, clinical guidelines, and procedures [17]. Our error-cause analysis (Table 2) supports this explanation, as many incorrect responses were due to “lack of understanding of regional context.”

To the best of our knowledge, this study presents one of the first quantitative evaluations of current LLM limitations in transfusion medicine, based on direct comparisons across laboratory medicine subspecialties. Our findings indicate that effective LLM use in this field requires strategies to overcome limited specialized data and regional context gaps. Approaches such as retrieval-augmented generation (RAG) and fine-tuning can help address these issues [18]. Nonetheless, expert oversight remains essential for ensuring clinical safety and reliability.

In subspecialties other than transfusion medicine, the models demonstrated notably higher accuracy than human examinees. This observation suggests strong potential for LLMs in structured, knowledge-based tasks. However, such performance warrants careful interpretation. The models outperformed human examinees in five out of six subspecialties, reflecting enhanced capacity for structured problem-solving and factual recall using standardized formats. These results do not indicate superiority in handling ambiguous or complex real-world clinical scenarios. Clinical decision-making often involves integrating incomplete, conflicting, or context-dependent information—tasks that remain challenging for current LLMs. Thus, although the models exceeded residency-level performance under standardized settings (except in transfusion medicine), broader clinical competence cannot be confirmed. Human oversight and contextual judgment remain essential in clinical environments.

Consistency of responses, or reproducibility, is as critical as accuracy in evaluating LLMs in medical contexts [19]. Given that response variability is sensitive to hyperparameter settings, we quantified reproducibility under standardized experimental conditions. A greedy decoding strategy was applied, with the temperature set to 0 for all models [20]. Temperature governs the randomness level in an LLM’s token-selection process during response generation. At temperature 0, models select the highest-probability token, yielding deterministic output for identical inputs. However, even under greedy decoding, a degree of non-determinism can persist during practical LLM implementations [21–23]. In our evaluation, inconsistency rates ranged from 0% to 6.9%, depending on the model (Table 3). Notably, OpenAI models demonstrated higher variability than Google and Anthropic models. Potential sources of this residual variability include [24]: 1) limitations associated with floating-point arithmetic precision within graphics-processing units, 2) the inherent stochasticity of tie-breaking mechanisms activated for multiple tokens sharing the same highest probability score, and 3) non-deterministic elements present in the model architecture upstream of the final token-prediction stage, including probabilistic expert routing in mixture-of-experts models. Despite the importance of understanding these non-deterministic properties and quantitatively assessing their variability for responsible LLM adoption and utilization in healthcare [23], reproducibility assessment was frequently overlooked in previous research. A recent survey of 158 studies on medical LLM performance (November 2022-June 2024) found only 12.7% explicitly described reproducibility [25]. In this context, our detailed reproducibility assessment provides crucial empirical evidence for assessing the potential clinical applicability and reliability of current LLMs.

This study has the following strengths. First, we directly compared 12 proprietary LLMs, including those with both small and large architectures from major LLM developers under identical evaluation conditions, providing comprehensive and directly comparable data on LLM performance at a specific point in time, differentiating our study from those focused on single-model evaluations or more limited comparisons. Second, to our knowledge, this was the first study to directly compare LLMs and residents-in-training using the same examination questions in the laboratory medicine field, offering a valuable benchmark for gauging the expert knowledge level of LLMs versus human specialists. Third, question difficulty was objectively categorized using actual resident performance data rather than subjective or arbitrary criteria, enabling a more reliable analysis of how LLM performance varies with the difficulty level. Finally, by analyzing performance differences across subspecialties within laboratory medicine, we empirically identified a serious knowledge imbalance issue affecting current LLMs (particularly concerning transfusion medicine) and proposed its underlying causes and potential remedies, representing a key contribution to the field.

Nevertheless, this study has some limitations. First, the model selection process was limited by API availability and image input constraints during the evaluation period. Consequently, we did not comprehensively assess all major existing LLMs. The rapid pace of LLM development and the inherent delay associated with peer review and publication may limit the extent to which the reported findings reflect the capabilities of the most recent models available at the time of publication [26]. Second, changes in Korean healthcare during 2024, particularly those related to proposed increases in medical school quota [27, 28], substantially reduced the number of examinees participating in that year’s residency examination (109 in 2021, 116 in 2022, 123 in 2023, and 23 in 2024). Consequently, the difficulty classification standard (residents’ correct answer rates) derived from the 2024 resident response data may lack the representativeness of earlier years, An additional limitation concerns the language used for model evaluation. All questions were presented in Korean, which may have influenced model performance because of the predominance of English language training data in most LLMs. This language-specific factor may have affected performance differentially across various LLM architectures, depending on each model’s exposure to and handling of Korean medical terminology. Although our findings offer insights into LLM capabilities in the context of Korean laboratory medicine, caution should be exercised when generalizing these results to other languages or regions. Future studies using identical questions translated into multiple languages would be valuable to distinguish between limitations in domain knowledge versus language-specific processing capabilities. Such multilingual assessments would help determine whether observed performance differences stem from model architecture, pre-training data imbalances, or linguistic factors.

In summary, we comprehensively evaluated the capabilities of 12 proprietary LLMs to answer questions on the Korean Laboratory Medicine Residency Examination. Gemini 2.0 Pro demonstrated the highest overall accuracy, especially with high-difficulty questions. However, all models showed significantly reduced performance in transfusion medicine, likely owing to insufficient domain-specific training data and limited understanding of regional contexts. Non-deterministic response was observed with several models despite deterministic settings (temperature=0), highlighting the need to verify reliability in clinical contexts. Collectively, although recent LLMs exhibit considerable knowledge and reasoning abilities relevant to laboratory medicine, unresolved challenges remain including knowledge gaps in specific subspecialties and optimizing accuracy for highly complex domains. Responsible integration of LLMs into laboratory medicine practice will require continued technical enhancements, such as the application of RAG and fine-tuning, coupled with expert oversight.