Constructing preference data for alignment tuning has traditionally required human annotation to label and rank responses; however, this approach is both time-consuming and costly. To address this, we utilize the DAHL score [4], an automated hallucination evaluation metric developed specifically for long-form biomedical text generation, enabling more affordable and efficient annotation. DAHL provides both benchmark questions and a comprehensive pipeline to assess the factual accuracy of model responses. The evaluation relies on two key components: the splitter model and the checker model. The benchmark consists of 8,573 biomedical questions sourced from research papers in PubMed Central (https://pmc.ncbi.nlm.nih.gov/), spanning 29 categories. For each question, the hallucination evaluation pipeline follows three main steps:

The accuracy score for an individual response is its DAHL score, while the average score across all 8,573 responses represents the target model’s DAHL score. In this study, we substitute human-annotated labels with the DAHL score to provide a cost-effective and scalable alternative for constructing preference data.

Using 8,573 biomedical examination questions from DAHL, we generated responses for each question with five different models: Dolly-v2-3B [9], MPT-7B [10], GPT-4o, Qwen-2-7B [11], and Llama-3-8B [12]. We incorporated a variety of models, not solely state-of-the-art ones, to obtain a wider variation of responses. This process produced five distinct responses per question, and the DAHL score was computed for each response. We retained only the responses with the highest and lowest DAHL scores to construct the preference dataset. The response with the highest score was designated as the “chosen” response, while the one with the lowest score was labeled as the “rejected” response. The original question then served as the “prompt.” This construction pipeline is illustrated in Figure 1. To refine the dataset further, we applied two filtering criteria. First, entries were excluded if all five responses exhibited identical DAHL Scores, as this would indicate no discernible difference in factual quality between the “chosen” and “rejected” responses. Second, entries were removed if the “chosen” response repeated the question. Following these filters, the resulting preference dataset comprised 5,551 entries with three columns: “prompt,” “chosen,” and “rejected.” Because the DAHL dataset and evaluation pipeline are tailored for English, the initial preference dataset was in English. We therefore translated all three columns into Korean using GPT-4o, with the translations subsequently verified by human annotators. This process yielded a Korean medical preference dataset (KoMeP) containing 5,551 entries, ready for use in alignment tuning.

The constructed KoMeP dataset contains 5,551 rows, each comprising “prompt,” “chosen,” and “rejected” columns. As shown in Table 1, “chosen” responses are slightly longer than “rejected” responses—their length is approximately 1.2 times greater when measured by character or sentence count. This pattern is consistent with other preference datasets, as reward models in RLHF (reinforcement learning from human feedback) tend to favor longer responses over shorter ones [13].

Additionally, the dataset incorporates 29 medical categories adopted from [14], and Figure 2 visualizes their distribution. Public health, oncology, pharmacology, genetics, microbiology, cardiology, immunology, neurology, and radiology dominate most categories, while each remaining category accounts for less than 3%.

Figure 3 presents an example from KoMeP, showing the Korean text followed by its English translation. In this example, the “chosen” response provides a truthful answer to the “prompt” question, earning a DAHL score of 1, whereas the “rejected” response offers a clinical example related to the question instead of a direct answer. Because the “rejected” response addresses a specific case, its components cannot be judged as entirely accurate, resulting in a DAHL score of 0. Thus, the response that accurately and truthfully addressed the question—and consequently earned a higher DAHL score—was selected as “chosen.”

In most cases, however, the DAHL scores for both “chosen” and “rejected” responses typically fall between 0 and 1 with smaller score differences. In some instances, “rejected” responses received a score of 0 simply by repeating the question instead of answering it, and it was rare for a response to score 0 merely for providing a well-formed, albeit hallucinated, answer.

To investigate the feasibility of transfer learning from Korean to English using DPO training with KoMeP, we evaluated the trained models on two widely used English medical benchmark datasets: MedMCQA [19] and PubMedQA [20]. Model performance was compared before and after DPO training.

Since KoMeP was originally translated from English, we also possess an English version of the medical preference data. After observing that transfer learning from Korean to English was not very effective, we additionally explored transfer learning from English to Korean. Using the English version of our dataset, we applied DPO training to the five models and compared their performance on KorMedMCQA pre- and post-training.

To further assess KoMeP’s utility, we reformatted it into an instruction-tuning dataset. Specifically, the prompt and chosen columns were restructured into a QA task format, adapting the chatting template for each model, as illustrated in Figure 4. For this experiment, we used relatively smaller models (Gemma-2-2B, Qwen-2-1.5B, and Qwen-2-7B). Each model underwent instruction tuning with the restructured dataset for up to three epochs, maintaining the experimental settings described earlier. Their performance on KorMedMCQA was evaluated before and after fine-tuning.

As shown in Figure 6, the results provide no consistent evidence of effective transfer learning from Korean to English in the biomedical domain. For MedMCQA, two models (Gemma-2-9B and Qwen-2-7B) improved, while Gemma-2-2B maintained its performance and Llama-3.1-8B showed reduced performance. On PubMedQA, only Llama-3.1-8B demonstrated improved performance, whereas all other models experienced a decline.

As shown in Figure 7, transfer learning from English to Korean was more effective for most models compared to the Korean-to-English scenario. Performance improvements for Qwen-2 were similar regardless of whether the model was trained on the English or Korean medical preference dataset. However, Gemma-2 demonstrated a significantly greater performance increase when trained on the English dataset.

DPO training led to consistent performance improvements across most models, while ORPO training produced inconsistent results—with some models even exhibiting performance degradation. This outcome suggests that ORPO may not be well-suited for models already fine-tuned with RLHF, rather than pointing to an issue with our dataset. The overall pattern is further supported by the consistent performance gains observed with DPO.

Models pretrained on multilingual data (Qwen-2 and Llama-3.1) performed better with alignment tuning compared to the primarily English-based Gemma-2 models. This divergence can be attributed to the lower Korean proficiency of Gemma-2, likely resulting from a smaller proportion of Korean data in its pre-training corpus. Gemma-2 models, which are officially not multilingual [15], showed lower baseline scores (0.0565 for Gemma-2-2B and 0.0525 for Gemma-2-9B) compared to the multilingual Qwen-2 and Llama-3.1 models [11,12]. This disparity highlights the importance of pre-training data; models exposed to more Korean data during pre-training exhibit better alignment tuning results [21] and underscore the need for comprehensive target-language data to maximize alignment tuning effectiveness [22].

When tested on MedMCQA and PubMedQA, models trained with KoMeP exhibited only limited performance improvements. While some models (e.g., Gemma-2-9B and Qwen-2-7B) showed modest gains on MedMCQA, performance on PubMedQA generally declined. This discrepancy may be due to differences in the pre-training data composition. As noted, Gemma-2 is not an officially multilingual model, whereas Qwen-2 is. Therefore, alignment tuning for Gemma-2—primarily an English-oriented model—had a more pronounced impact on English data.

When the English version of KoMeP was used for training, performance on KorMedMCQA improved significantly across most models, especially for Gemma-2, which benefited more from English data compared to Korean. In contrast, Qwen-2, being inherently multilingual, showed little performance difference between training on English versus Korean data. Unexpectedly, Llama-3.1 experienced a substantial performance decline when trained with English preference data. One possible explanation is that, as the most recent model among those tested, Llama-3.1 may have been pretrained on cleaner and more comprehensive English data than was provided by our English preference dataset.

KoMeP was reformatted into an instruction-tuning dataset by structuring the prompts and responses in a QA format. Three models (Gemma-2-2B, Qwen-2-1.5B, Qwen-2-7B) were trained using this approach. While the Qwen-2 models showed slight improvements in KorMedMCQA performance, Gemma-2-2B experienced a minor decline, possibly due to its limited Korean language proficiency. This outcome may be attributed to both the dataset’s format and its size. Although the data was converted into a QA format suitable for instruction tuning, it remains tailored for long-form QA generation tasks, whereas KorMedMCQA is a multi-class classification benchmark that requires selecting from predefined options.

Furthermore, with only 5,551 entries focused on a single task, the dataset’s impact may be limited when used in isolation. Incorporating KoMeP into a larger, more diverse dataset covering multiple tasks could further enhance its utility—a prospect that merits additional investigation.