PDF
Citation
Print
I. Introduction
Tuberculosis (TB) remains a critical public health challenge in Korea. In 2021, Korea ranked first in TB incidence among Organisation for Economic Co-operation and Development nations [1]. Treatment success rates remain below the World Health Organization average [2], hindered by barriers including insufficient TB knowledge and social stigma [3]. Therefore, understanding public perceptions and concerns about TB treatment in Korea may prove beneficial. This approach enables the development of targeted educational interventions and psychological support strategies, thus improving treatment outcomes [4].
Studies on Korean perceptions of TB have been constrained by small sample sizes [5–8] with inadequate representativeness of the Korean population. This study leveraged Naver Knowledge-iN, a Korean online question-and-answer platform featuring extensive user-generated content, to provide a larger dataset than previous research. Moreover, this platform allows access to spontaneous public questions about TB treatment, clarifying barriers often overlooked by traditional methodologies such as structured questionnaires.
Large language models (LLMs) offer advanced capabilities for analyzing complex textual data, providing contextual understanding beyond traditional computational linguistics methods [9]. This study employs LLMs to comprehensively examine public questions regarding TB treatment, demonstrating their capability for contextual understanding compared to conventional text analysis techniques such as keyword matching or latent Dirichlet allocation (LDA).
This research had three primary objectives: (i) to investigate public perceptions and concerns surrounding TB treatment by analyzing queries about antitubercular medications; (ii) to examine drug side effects, infectivity, and social implications of TB during treatment, informing personalized educational and counseling interventions; and (iii) to evaluate the performance of LLMs as tools for extensive data analysis.
Through systematic analysis of public questions, this study provides relevant insights into strategies for improving TB treatment outcomes in Korea while demonstrating the methodological potential of LLMs in addressing public health challenges. The study presents a framework for integrating LLMs into healthcare research.
II. Methods
Figure 1 illustrates the overall study process.
1. Data Collection
Using web crawling, 44,174 questions containing the term “tuberculosis” (collectively, the “entire dataset”) were obtained from the Naver Knowledge-iN platform (2002–2024). Following initial data collection, questions mentioning antitubercular medication were extracted through a multi-stage process involving several OpenAI models: GPT-4omini-2024-07-18, GPT-4o-2024-08-06, and fine-tuned GPT- 4o-mini-2024-07-18. Supplement A details this method.
2. Data Analysis
1) Frequency analysis of antitubercular medication names
Twenty-four keyword lists, totaling 547 keywords, were developed to extract questions mentioning specific anti-tubercular medications. Each list (21 generic names and 3 combination drug lists) corresponded to one medication, including its various forms. Extracted questions were categorized based on the 24 lists, with multiple categorizations permitted. Supplement B details this protocol.
2) Thematic categorization
To identify relevant public concerns, questions mentioning antitubercular medication were thematically classified into seven predefined categories:
-Antitubercular medication management and challenges
-Effectiveness of antitubercular medication
-Interactions with other substances
-Infectivity and social implications
-Administrative and institutional aspects
-Translation
-Not applicable (N/A)
Supplement C describes these categories. Categorization employed OpenAI’s fine-tuned GPT-4o-mini-2024-07-18, trained on 147 manually reviewed examples. Multiple categorizations were permitted.
3) Analysis of Infectivity and Social Implications
Questions categorized under “Infectivity and Social Implications” were analyzed using a three-step process to identify thematic subgroups: text embedding generation via OpenAI’s text-embedding-3-large model, dimensionality reduction using uniform manifold approximation and projection (UMAP), and cluster analysis implemented with K-means clustering. Topic labels for the 10 groups were generated with GPT-4o-mini, and two researchers reviewed questions and labels for each group. Groups with distinctive features remained independent, retaining their labels. Conversely, those comprising general questions about infectivity during treatment without distinctive features were grouped as “General Questions.” Supplement D details the parameters for UMAP and K-means clustering.
4) Analysis of side effects
The analysis of side effects employed two parallel approaches: (i) general side effects were analyzed across all questions related to “Antitubercular Medication Management and Challenges” using GPT-4o-mini-2024-07-18 and GPT-4o-2024-08-06, and (ii) side effects mentioned with each first-line medication (isoniazid, rifampin, ethambutol, and pyrazinamide) were analyzed using GPT-4omini-2024-07-18. The analysis utilized a predefined list of side effects and TB symptoms [10] (Supplement E). To address challenges in distinguishing between side effects and symptoms, the analysis initially included both, subsequently excluding symptoms to isolate side effects.
3. LLM Accuracy Evaluation
1) LLMs vs. human researchers
The accuracy of LLMs was compared with human researchers across two tasks: (i) extraction of questions mentioning antitubercular medication by a series of non-fine-tuned GPT-4o-mini-2024-07-18, GPT-4o-2024-08-06, and fine-tuned GPT-4o-mini-2024-07-18 (Methods-1) and (ii) thematic categorization of questions by fine-tuned GPT-4omini- 2024-07-18 (Methods-2-3). For each task, researchers independently analyzed randomly selected 100-question samples. Inter-rater agreement among the researchers was calculated and compared with the concordance between LLMs and human consensus judgments.
2) LLMs vs. keyword matching
The accuracy of the same LLM series was compared with keyword matching in extracting questions mentioning antitubercular medication. Three progressively expanded keyword lists were utilized for extraction:
(i) List 1: 547 keywords representing generic and brand names of antitubercular medication, defined in Methods-2-1.
(ii) List 2: List 1 + {‘ 결핵약’, ‘결핵약물’, ‘결핵치료제’, ‘항결핵 제’, ‘결핵치료약’, ‘결핵 약품’, ‘결핵 약’, ‘결핵 약물’, ‘결핵 치료제’, ‘결핵 치료약’, ‘결핵 약품’} (Korean terms combining variations of “tuberculosis” and “medication”).
(iii) List 3: List 2 + {‘ 약’} (the general Korean term for medication).
Questions extracted via keyword matching with Lists 1, 2, and 3 were labeled Sets B1, B2, and B3, respectively. Questions extracted by LLMs via Methods-1 were labeled Set A. Six difference sets were generated: questions extracted by LLMs but missed by keyword matching (A-B1, A-B2, A-B3) and questions extracted by keyword matching but missed by LLMs (B1-A, B2-A, B3-A). From each difference set, 100 questions were randomly sampled. Two researchers independently evaluated these questions, resolving disagreements through discussion until reaching consensus. Inter-rater agreement among researchers was calculated and compared with the concordance between LLMs and human consensus judgments.
3) Fine-tuned vs. non-fine-tuned models
Fine-tuning accuracy was evaluated by comparing a fine-tuned GPT-4o-mini-2024-07-18 against a non-fine-tuned GPT-4o-2024-08-06 utilizing in-context learning. The comparison involved two tasks: extraction of questions mentioning antitubercular medication (Methods-1) and thematic categorization (Methods-2-2). For the first task, 100 questions randomly selected from the entire dataset were analyzed by both models, with the former trained on 110 extraction examples and the latter provided with six in-context examples. For the second task, 100 randomly selected questions mentioning antitubercular medication were analyzed by both models: the fine-tuned GPT-4o-mini-2024-07-18 (trained on 147 categorization examples) and another non-fine-tuned model (provided with six in-context examples). Researchers collaboratively reviewed the classifications by the two models until consensus was reached, counting the number of correct classifications.
4. LLM Efficiency Evaluation
1) Time and cost: LLMs vs. human researchers
The time and cost of LLMs in this study were compared with those of human researchers. To estimate hypothetical labor costs, this study postulated that human researchers performed 99,635 analyses across the entire process. Additionally, a conservative comparison was conducted with the simplest task: extracting questions mentioning any medication (Supplement A). The steps were as follows:
Step 1: 100 questions were randomly selected from the entire dataset.
Step 2: From these, researchers independently extracted questions mentioning any medication, with processing time recorded.
Step 3: Questions mentioning any medication were extracted by GPT-4o-mini-2024-07-18, again recording processing time.
Step 4: To extrapolate the total time for 99,635 analyses by human researchers, the average researcher time per 100 analyses was multiplied by 997.
Step 5: The hypothetical labor cost was estimated by multiplying total researcher time by 9,860 Korean won (KRW), Korea’s minimum wage.
Step 6: Token counts for input and output for each task across the entire process were measured.
Step 7: Total cost was calculated by multiplying token counts per analysis by their published prices and summing the results [11].
2) Cost: fine-tuned vs. non-fine-tuned models
The costs of the fine-tuned and non-fine-tuned LLMs were compared. To estimate the cost of the non-fine-tuned LLM, we postulated that Methods-1 and Methods-2-2, originally processed by fine-tuned LLMs, were instead handled by the non-fine-tuned model. The cost was calculated by multiplying the number of input tokens by the input token billing rate and adding the product of the number of output tokens by the output token billing rate.
5. Ethical Considerations
This study was approved by the Institutional Review Board of Inje University Busan Paik Hospital (Approval No. BPIRB 2025-05-031). The work qualified for exemption from further ethical review as it constitutes a retrospective analysis of publicly available data from which individual contributors are unidentifiable.
III. Results
Figure 1 illustrates the processes involved in the Results section.
1. Frequency Analysis of Antitubercular Medication Names
Among questions mentioning antitubercular medications by name (n = 919), rifampin (527 mentions, 31.8%) and isoniazid (524, 31.6%) were most frequently referenced, followed by ethambutol, pyrazinamide, Tubis, and Tubis2 (Supplement F).
2. Thematic Categorization
Of 10,044 questions referencing antitubercular medications, “Antitubercular Medication Management and Challenges” constituted the largest thematic category (5,717 mentions, 44.8%) (Figure 2). An example question is: “I skipped taking antitubercular medication yesterday. Will drug resistance develop immediately?”
3. Analysis of Infectivity and Social Implications
“Infectivity and Social Implications” questions (n = 583) were clustered into 10 thematic subgroups using text embedding, UMAP dimensionality reduction, and K-means clustering algorithm. Six subgroups had distinct topics, while four were indistinct. “Employment and Infection in Workplace” represented the largest distinct subgroup (114 questions, 20.6%), whereas the indistinct groups, collectively labeled “General Questions” and primarily concerning general infectivity, accounted for 50% of questions (Figure 3). Table 1 presents representative examples from each subgroup.
4. Analysis of Side Effects
Of 5,717 questions categorized under “Antitubercular Medication Management and Challenges,” 3,037 described side effects. Hepatotoxicity, dermatosis, and vomiting were most frequently reported (Figure 4). Regarding medication-specific side effects, hepatotoxicity was most common for isoniazid (33 mentions, 9.3%), dermatosis for rifampin (36, 8.6%), and visual impairment for ethambutol (21, 10.4%). Pyrazinamide displayed equal frequencies of nausea, dermatosis, and diarrhea (nine mentions each, 6.6%) (Table 2).
5. LLM Accuracy Comparison
1) LLMs vs. human researchers
Comparative analysis revealed that concordance between LLMs and human consensus was lower than the inter-rater agreement (Table 3).
2) LLMs vs. keyword matching
Compared to LLM-based extraction, keyword matching extracted fewer and less accurate questions. Figure 5 details the sizes and accuracies of each set.
3) Fine-tuned vs. non-fine-tuned models
Fine-tuned and non-fine-tuned models exhibited comparable accuracy (Table 4).
6. LLM Efficiency Comparison
1) LLMs vs. human researchers
To extract questions mentioning antitubercular medication from a 100-question set, human researchers averaged 498 seconds, whereas GPT-4o-mini-2024-07-18 required 20 seconds. The total cost for the study was USD 307.94 (Supplement G), while hypothetical human labor costs were USD 958.90 (converted from KRW 1,360,680 at an exchange rate of KRW 1,419/USD).
IV. Discussion
1. Main Findings and Interpretation
Frequency analysis of medications revealed a predominance of first-line drugs (isoniazid, rifampin, ethambutol, pyrazinamide) and fixed-dose combination drugs (Tubis and Tubis2), aligning with standard TB treatment in Korea. The higher frequency of isoniazid and rifampin reflects their central role during intensive and maintenance phases. Levofloxacin, cycloserine, kanamycin, and streptomycin followed. While levofloxacin and cycloserine remain prescribed for multidrug-resistant TB (MDR-TB), kanamycin and streptomycin have not been recommended for MDR-TB since 2020 [12–15]. Time-series analysis reflects this clinical practice change: most mentions of kanamycin and streptomycin before 2020 related to treatment, while post-2020 mentions typically did not. This alignment supports the validity of the Naver Knowledge-iN dataset as a reliable, representative sample of the Korean public.
The predominance of “Antitubercular Medication Management and Challenges” in the thematic categorization reflects the complexity of TB treatment regimens and patient concerns regarding side effects. The high question volume indicates knowledge gaps in medication management, potentially leading to poor adherence, incorrect medication intake, and premature discontinuation [3,16]. Thus, this category’s prevalence underscores the importance of comprehensive patient education [4].
Analysis of “Infectivity and Social Implications” revealed concerns about blood donation eligibility and immigration among patients with TB, aspects not identified previously [17]. Studies examining the social implications of human immunodeficiency virus [18], hepatitis B [19], and severe acute respiratory syndrome [20] reported discrimination in workplaces, educational and healthcare settings, familial relationships, and community interactions. Similar public anxieties about TB were identified: “Employment and Infection in Workplace” represented the largest distinct subgroup, followed by “Household Infection and Infection between Intimate Partners.”
Unlike the aforementioned diseases, individuals with TB retain eligibility for blood donation. Given the frequency of related questions, incorporating this information into patient education may be beneficial. Notably, Korean Red Cross Guidelines permit blood donation 1 month after completing TB treatment, suggesting that clear communication of such policies could support blood donation participation and TB treatment adherence [21].
The analysis of side effects identified hepatotoxicity, dermatosis, and vomiting as the most frequently reported adverse reactions, showing both consistencies and differences compared to the national Korea Adverse Event Reporting System (KAERS) database [22]. While the KAERS reported nausea as the most common adverse drug reaction (14.6%), followed by hepatic enzyme elevation (14.2%) and rash (11.7%), our Knowledge-iN analysis revealed hepatotoxicity as the predominant concern in public questions. This difference may reflect public awareness and concern about liver-related side effects, rather than clinical incidence.
Medication-specific comparisons revealed strong concordance for certain representative side effects. Visual impairment was the most frequently reported ethambutol-associated adverse effect in both datasets. In our study, 10.4% of ethambutol-related questions mentioned visual impairment, representing the highest proportion among all reported adverse effects. Similarly, in the KAERS database, it accounted for 53.8% of ethambutol-related adverse events, constituting the most prevalent adverse reaction linked to this medication. Additionally, ethambutol demonstrated a markedly higher rate of visual impairment compared to other first-line antitubercular drugs analyzed. In the KAERS data, the incidence of visual impairment was 14.1%, 19.2%, 12.8%, and 53.8% for isoniazid, rifampin, pyrazinamide, and ethambutol, respectively.
For rifampin, both studies identified dermatological manifestations as prominent. Our analysis included dermatosis in 8.6% of rifampin-related questions, while the KAERS reported rash as the leading rifampin-associated adverse event (29.0% of rifampin cases). However, some side effects appeared underrepresented in public queries versus clinical reporting. While the KAERS revealed relatively uniform hepatotoxicity rates across first-line drugs (21.5%–26.5%), our Knowledge-iN analysis suggested disproportionate public concern about isoniazid-related hepatotoxicity (9.3% of isoniazid questions). Interestingly, characteristic rifampin-associated changes in urine color appeared more prominently in public questions (6.7% of rifampin-related queries) but were unreported in the KAERS, as the database generally captures only pathological events, excluding benign pharmacologic side effects. This suggests greater patient concern about visually noticeable yet harmless bodily changes.
This study demonstrated the superior effectiveness and efficiency of LLMs in processing extensive datasets compared to human researchers, keyword matching, and LDA. Keyword List 1 failed to capture 9,125 relevant questions (A-B1, 97% accuracy), as these questions frequently omitted precise medication names (Figure 5A). List 2 was expanded to include variations of the term “antitubercular medication”. Nevertheless, this approach missed 4,242 questions (A-B2, 96% accuracy) due to separated mentions of “tuberculosis” and “medication,” with examples including “I was diagnosed with tuberculosis yesterday and received medication” (Figure 5B). List 3, expanded to include the term “약” (medication), captured irrelevant words containing the syllable “약”, such as “예약” (reservation) and “약속” (appointment), yielding 9,908 irrelevant questions (B3-A, 13% accuracy). This is exemplified by questions like “Do I need to make a doctor’s appointment for a tuberculosis screening?” (Figure 5C). Furthermore, since the Naver Knowledge-iN questions were posted by individuals, grammatical errors, typographical mistakes, and ambiguous expressions complicated keyword matching. Conversely, LLMs demonstrated superior contextual understanding of unstructured and noisy data. The larger volume and higher accuracy of questions in A-B1 and A-B2 compared to B1-A and B2-A further underscore this advantage. However, LLMs missed 148 questions (B2-A, 66% accuracy), focusing on broader topics in which TB treatment was mentioned. Additionally, 201 questions (A-B3, 29% accuracy) were misclassified as mentioning antitubercular medication, often due to ambiguous terms like “TB injection”. While these misclassifications highlight areas for improvement, the small set sizes of these sets limited their impact on the analysis.
LLMs underperformed human researchers in extracting and categorizing questions mentioning antitubercular medication, with lower concordance than inter-rater agreement. However, misclassifications stemmed from ambiguous language by inquirers, which also challenged the researchers. Nevertheless, LLMs significantly enhanced efficiency, processing vast datasets far faster than humans. Fine-tuning further increased cost efficiency, halving expenses compared to reliance on non-fine-tuned models. Crucially, fine-tuning did not compromise performance, as both models achieved similar accuracy rates.
When thematically categorizing questions (Methods-2-2) and identifying subgroups within “Infectivity and Social Implications” (Methods-2-3), initial attempts to use LDA revealed its limitations, leading researchers to adopt LLMs [23].
Researchers first attempted LDA, grouping 100 randomly selected questions into three clusters. Because LDA does not automatically label clusters based on their contents, the researchers reviewed and named each cluster using keywords. However, they could not find meaningful topics for each cluster because LDA failed to clearly classify each question or yield distinctive keywords.
Combined with this inherent limitation, difficulties in processing Korean text—such as handling typographical errors, irregular word spacing, and removing stop words—potentially further hindered LDA in identifying topics. Conversely, LLMs successfully identified three significant topics using the same dataset.
2. Limitations
This study has several limitations concerning the data source and methodological constraints. The Naver Knowledge-iN platform may have introduced sampling bias, potentially underrepresenting digitally marginalized demographics. Additionally, inclusion of outdated questions in the 2002–2024 dataset could misrepresent contemporary public perceptions.
Methodological constraints primarily concerned LLM implementation and performance. Extracting questions that mentioned antitubercular medication (Methods-1) required multi-stage refinement using successive LLMs, ultimately relying on a fine-tuned model (Supplement A). This multistep approach highlights the limitations of LLMs in precise contextual interpretation, necessitating multiple validation steps for accuracy.
The selective use of fine-tuned and non-fine-tuned models across analytical steps posed additional challenges for researchers regarding processing volume and task complexity. While fine-tuning demonstrated economic advantages, it also required considerable effort by the researchers to create and validate training examples.
3. Implications
This exploratory study presents the analysis of spontaneous public questions, revealing aspects overlooked by traditional methodologies. The findings inform future research directions and practical applications. Notably, the relationship between TB treatment outcomes and newly identified concerns warrants confirmatory research. A detailed understanding of social implications could support predictive models for treatment adherence, enabling personalized educational and counseling interventions to improve treatment outcomes in Korea.
The analysis of unstructured social media data is valuable for understanding public health perceptions. While this study utilized Naver Knowledge-iN, the methodology could be extended to platforms such as YouTube. This approach may be particularly useful in studying diseases impacted by social stigma, such as leprosy and mental health disorders, thus improving treatment and quality of life [24].
Furthermore, this study validates the effectiveness of LLMs in analyzing unstructured, noisy content. Applying LLMs to questions with linguistic variations and contextual complexities demonstrates their utility for large-scale healthcare data analysis. The integration of social platform data analysis with LLM-based processing methods provides a systematic framework for future public health research.
XML Download