I. Introduction
Recruiting a sufficient number of participants who meet specified eligibility criteria is essential for every clinical trial. The manual review of clinical data and identification of eligible cases often constitute the most labor-intensive aspect of the trial process [1].
In recent years, the development of electronic health records (EHR) has partially streamlined this process. However, the reliance on unstructured and ambiguous text in EHRs continues to pose significant challenges. To address this issue, researchers have increasingly adopted natural language processing (NLP) techniques [2].
The field of NLP is currently undergoing a revolution driven by rapid advances in LLMs. The application of LLMs to the medical domain has attracted considerable interest [3–5], with use cases ranging from clinical documentation [6,7] and decision support [8,9], to knowledge-based information retrieval and generation [10,11], medical research [12–15], and data processing and analysis [16,17]. However, the use of LLMs for clinical trial matching remains in its early stages [18].
The primary objective of this study is to investigate the effectiveness of prompt-based learning models for cohort selection in clinical trials, utilizing unstructured data from EHRs. We aimed to determine whether prompt-based methods can achieve results comparable to, or better than, those obtained using traditional NLP techniques. Our focus was not limited to benchmarking model performance, but also to assessing whether structured summarization could enhance eligibility classification.
Our focus is on a specific challenge—cohort identification in the 2018 National NLP Clinical Challenges (n2c2)—which provided a dataset of free-text medical records from 288 patients.
II. Methods
1. Dataset
We utilized the dataset from the 2018 n2c2 shared task (track 1), which centered on cohort selection for clinical trials. This dataset includes 288 patient records, each manually labeled by experts to indicate whether the patient met any of 13 eligibility criteria. These criteria cover MAJOR-DIABETES (major diabetes complications), DRUG-ABUSE (history of drug abuse), ABDOMINAL (history of intra-abdominal surgery, intestinal resection, or small bowel obstruction), MI-6MOS (myocardial infarction within the last 6 months), KETO-1YR (diagnosis of ketoacidosis within the past year), ALCOHOL-ABUSE (current alcohol use above recommended limits), MAKES-DECISIONS (ability to make medical decisions independently), ENGLISH (ability to speak English), ASP-FOR-MI (aspirin use for myocardial infarction prevention), ADVANCED-CAD (advanced cardiovascular disease), DIETSUPP-2MOS (dietary supplement use in past 2 months), HBA1C (HbA1c between 6.5% and 9.5%), and CREATININE (serum creatinine above normal). Some criteria, such as MI-6MOS, KETO-1YR, ALCOHOL-ABUSE, and DRUG-ABUSE, have very limited instances in the data, making accurate identification particularly challenging. The dataset was divided into a training set (202 records) and a test set (86 records), with significant class imbalance present.
2. Methodology
Figure 1 provides an overview of the proposed framework’s main stages. Each step is detailed in the following sections.
1) Knowledge graph
Traditionally, ontologies in NLP are used to standardize terminology and features. For this study, we leveraged Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT).
Unlike previous studies that manually curated trigger words for text processing, our method utilizes established ontologies to enable a semi-automatic extractive summarization and relevant sentence extraction process—specifically targeting sentences containing SNOMED CT concepts aligned with each criterion. This involves leveraging concepts encoded in SNOMED CT within the Unified Medical Language System (UMLS) to extract pertinent information from eligibility criteria and patient records, akin to the method described in [19] for evaluating the similarity between criteria and patients. UMLS is the largest comprehensive repository of biomedical concepts and their relationships [20]. For example, the eligibility criterion “ABDOMINAL: History of intra-abdominal surgery, small or large intestine resection, or small bowel obstruction” is translated into SNOMED CT keywords such as “major abdominal surgery,” “small intestine excision,” “large intestine excision,” and “small bowel obstruction.” All descendants of these terms are retrieved automatically with their concept unique identifier (CUI) codes.
To support this process, we employed PyMedTermino2 [21], a Python library built on Owlready2, which facilitates programmatic access to the SNOMED CT ontology via the UMLS 2021AA release. We used the “is-a” relationship to automatically expand broad clinical concepts (e.g., “major abdominal surgery”) into comprehensive lists of descendant terms. For instance, “colectomy” (removal of large bowel) is a descendant of “major abdominal surgery” and was thus included. Retrieved concepts included each term’s label, synonyms, and child terms. We constructed a code list for each criterion comprising the primary SNOMED CT-related keywords; for example, “161617006, major abdominal surgery [Finding]” along with its associated descendants (e.g., colectomy).
2) Extractive summarization
Extractive summarization entails selecting essential sentences from the source text to create a concise summary that respects the input length constraints of the LLM, while preserving the original meaning [22].
In this study, we applied minimal pre-processing to best evaluate the capabilities of LLMs in handling medical narratives. The main pre-processing steps included time normalization and the expansion of certain acronyms and abbreviations, using regular expressions and rule-based approaches.
At this stage, free-text clinical records were automatically annotated by MedCAT [23], an NLP tool extending sciSpa-Cy to recognize named entities linked to clinical concepts. These entities were mapped to SNOMED CT and UMLS terminologies. We used MedCAT v1.0.0rc2, publicly available and pre-trained on the MIMIC-III database and the entirety of SNOMED CT. Patients’ free-text records were annotated by MedCAT, and the extracted concepts were matched against the SNOMED CT descendant code lists from the previous step.
Finally, we extracted temporal information and sentences from the records containing annotated concepts whose CUI codes matched those for each eligibility criterion.
3) Prompt engineering
A prompt is an instruction or question provided to the model to guide its response [24]. In this step, each eligibility criterion was converted into a prompt and presented to the GPT-3.5-turbo model, together with the summarized text from the previous stage. Each criterion was reworded into a question or prompt; Table 1 shows examples for all 13 criteria. For binary classification, our approach required the model to respond with a single word: “yes” or “no.” We mapped “yes” to the class “met” and “no” to “not met.” The model was instructed to determine whether each summarized record met the specified criterion. Evaluation was conducted using all 86 records in the test set. Our approach used a fully zero-shot setting, with no additional fine-tuning on the n2c2 dataset. An illustrative example of zero-shot learning for the “ABDOMINAL” criterion is shown in Figure 2.
4) Evaluation metrics
We followed the evaluation protocol established by the challenge organizers. GPT output was submitted to a predefined evaluation script, which computed precision, recall, F score, specificity, and area under the receiver operating characteristic curve (AUC). In our experiment, precision reflects the proportion of records labeled as “met” that were truly “met.” Recall measures the proportion of actual “met” records correctly identified by our model. Specificity quantifies the proportion of records labeled as “not met” that were correctly classified. The F score, combining both precision and recall, was calculated in both micro and macro variants. Micro F1 aggregates true positives, false positives, and false negatives across all classes before computing the score, whereas macro F1 calculates the F1 for each class independently and averages them, giving equal weight to rare and common classes. In imbalanced datasets, the micro F1 score often exceeds macro F1 because it reflects better performance on dominant classes [25].
III. Results
1. Model
For our experiments, we used OpenAI’s GPT-3.5-turbo model, recognized as the most capable and cost-effective variant of GPT-3.5 at the time of the study [26]. To ensure reproducibility across multiple runs, we set the temperature parameter to 0 for all requests. While a temperature of 0 substantially increases output determinism, a small degree of variability can still occur due to the inherent stochasticity of transformer-based models. The model was instructed to respond with a single word (“yes” or “no”). Although the temperature setting standardized outputs, slight variability may have persisted.
2. Results
To illustrate the effect of the summarization method, we present results for two scenarios. In the first scenario, no summarization was performed; the clinical free-text and prompts were given directly to the model. Due to the model’s text processing limitation of approximately 4,096 tokens, we truncated longer medical narratives and excluded the conclusions from those records. It should be noted that about 11% of the records exceeded 4,000 words. Table 2 provides a summary of the findings from this initial scenario.
After applying our proposed summarization method in the second scenario, we observed a marked improvement in the model’s performance. Table 3 summarizes the results using summarization: our model achieved micro and macro F-measures of 0.9035 and 0.7949, respectively. The criteria ALCOHOL-ABUSE, CREATININE, ENGLISH, and ABDOMINAL achieved the highest F-measures (above 0.9), while KETO-1YR, MI-6MOS, and MAKES-DECISIONS had the lowest F-scores (below 0.7).
Table 4 compares our model’s scores to the best results from other published machine learning (ML) methods on the same n2c2 dataset, both on average and for each criterion. Encouragingly, our model achieved the second-highest micro F-measure and the highest macro F-measure among all compared methods. Moreover, it achieved the top F-score for five of 13 criteria: ABDOMINAL, CREATININE, ASP-FOR-MI, ENGLISH, and ALCOHOL-ABUSE.
We also conducted limited experiments with GPT-4. Unlike GPT-3.5, GPT-4 does not impose strict input length limitations, allowing us to provide unsummarized clinical freetext with prompts. However, we did not observe consistent performance improvements compared to GPT-3.5-turbo. These findings underscore the importance of structured summarization, which enhances the effective use of the model’s capacity. The detailed results for GPT-4 are provided in Supplement A.
IV. Discussion
The application of a prompt-based learning model in this study yielded promising results. To further illustrate the process, we summarized the results for one representative criterion, “ALCOHOL-ABUSE,” in Supplementary Materials. Supplement B provides the list of SNOMED CT terms identified for “ALCOHOL-ABUSE.” Supplement C presents, for each test record, the selected sentences containing one of these terms, the ground truth label, and the model’s prediction for comparison.
Our model achieved overall micro and macro F-scores of 0.9061 and 0.8060, respectively, demonstrating acceptable performance across all criteria. To evaluate performance by criterion, we categorized the criteria based on the NLP methods required, as described by Stubbs et al. [27]:
(1) Concept extraction: Four criteria—ABDOMINAL, MAJOR-DIABETES, CREATININE, and HBA1C—primarily required the extraction of clinical terms. We addressed this by mapping each criterion to relevant SNOMED CT concepts. While standardized ontologies improve consistency, reproducibility, and completeness, challenges remain. Many records in the challenge dataset used rare medical terms, general pathology terms rather than precise category names, or uncommon abbreviations.
(2) Temporal reasoning: Another four criteria—DIET-SUPP-2MOS, MI-6MOS, ADVANCED-CAD, and KETO- 1YR—necessitated temporal processing. Since the data consisted of medical records from different time points, these criteria required time-aware analysis. Relevant sentences and their associated timestamps were extracted for these criteria.
For DIETSUPP-2MOS, a key advantage was leveraging the SNOMED CT ontology to identify supplement names, instead of relying on manually curated dictionaries. The main limitation, however, was incomplete coverage of commercial names and abbreviations.
For MI-6MOS, although our system achieved the second-highest F-score among ML methods, the main limitation was false negatives due to gaps in the SNOMED CT terms used for sentence selection. To address this, we used more general concepts such as “ischemic heart disease” or “disorders of coronary artery” rather than only “myocardial infarction,” but some records still lacked direct references to infarction.
For ADVANCED-CAD, results showed accurate selection of relevant sentences and identification of required time points. However, the exact definitions for required signs and symptoms were not always specified. For example, some antihypertensive drugs may be prescribed to patients with ischemic heart disease absent definitive hypertension, and it is unclear whether such cases meet the criterion.
(3) Inference: Our model outperformed other ML methods for ASP-FOR-MI and ENGLISH, but MAKES-DECISIONS was more challenging. Error analysis revealed that most misclassifications were due to difficulties in concept extraction rather than logical inference. MAKES-DECISIONS encompasses a broad clinical pathology with diverse signs and symptoms, making extraction of all relevant SNOMED CT concepts challenging.
For DRUG-ABUSE and ALCOHOL-ABUSE, our model performed well, with only a few false negatives due to missing rare drug names in the SNOMED CT concept list.
Compared to rule-based systems, ML models offer greater efficiency and adaptability by learning directly from data. Their scalability is evident in their ability to utilize larger, more diverse datasets without the need for rule modifications. Thus, ML techniques are well-suited for handling increasingly complex clinical data [28].
Moreover, the strengths of GPT models for cohort selection become even more apparent with larger datasets. First, in clinical practice, generating labeled training data is often cumbersome; GPT models are advantageous because they do not require task-specific training. Second, GPT models are ideal when a language model with a broad knowledge base is needed, as they are extensively pre-trained on a variety of text sources. Third, pre-processing is typically one of the most time-consuming and expertise-dependent steps in NLP pipelines. Achieving strong results with GPT models without extensive pre-processing can expedite the clinical matching process and reduce reliance on domain experts [29,30].
Additionally, we introduced an extractive summarization method using the SNOMED CT ontology and an annotation tool to capture essential information from source texts. This approach not only addresses the input length limitations of LLMs but also enhances prompt-based model performance. Supplement D provides an example: it displays an unsummarized free-text record alongside its summarized version, the prompt for one eligibility criterion (major abdominal surgery), the true label, the GPT-3.5 prediction using summarized data, and the GPT-4 prediction without summarization. Notably, GPT-4 misclassified the record, while GPT-3.5 with summarization produced the correct label—highlighting the value of summarization for removing irrelevant content and improving classification accuracy.
While newer models such as GPT-4—with larger context windows and improved reasoning—offer promising alternatives, they come with higher computational costs and infrastructure requirements. Our study demonstrates that an ontology-based summarization pipeline can significantly enhance LLM performance, even for models with limited input capacity.
In conclusion, our automated clinical trial matching solution streamlines what is typically a manual, time-consuming recruitment process by leveraging advanced technologies. A key direction for future work is the integration of structured summarization with chain-of-thought (CoT) prompting to further improve clinical eligibility classification. CoT techniques facilitate the decomposition of complex instructions into intermediate reasoning steps, which may enhance accuracy, particularly for tasks involving inference, temporality, or implicit logic.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation
Supplement1
