Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

On French medical multiple-choice questions, a cheap supervised fine-tune scores within about one accuracy point of a costlier recipe: continual pretraining followed by the same fine-tune, about four times the cost at 7B.

A controlled comparison of continual pretraining and supervised fine-tuning across 12 base models, with paired significance tests, a cost table, and three findings about how medical QA gets scored.

A hospital wants a medical assistant that answers in French. It has an open model, a few gigabytes of French medical text, half a million labeled question-answer pairs, and a GPU budget. This paper measures what each way of spending that budget buys.

Most open language models learn mostly from general-domain English. Adapting one to medicine in another language usually means one or both of two steps. Continual pretraining (CPT) keeps training the model on raw domain text with the same next-token objective it was pretrained with. Supervised fine-tuning (SFT) trains it on labeled examples: an instruction or question in, the correct response out. Both are known to help. How much each helps, and whether stacking them is worth the extra compute, had not been measured in a controlled way for a non-English medical setting.

The paper runs that measurement on French medical question answering. Five results organize this page: SFT gets nearly all of the multiple-choice gain at a fraction of the cost; CPT alone is unreliable; multi-answer exact match hides models that are one option off; overlap metrics and an LLM judge disagree about open-ended answers; and benchmarks machine-translated from English make models look more accurate and more confident than native French benchmarks do. Two of the paper's explanations, both about answer length, do not match its own tables; the open-ended section shows the numbers.

The experiment: 12 models, 3 strategies

The design is a grid. Three model families: Mistral-7B, Gemma-3-4B, and Llama-2 at both 7B and 13B, which gives four model lines. Each line comes in three initializations: the general pretrained base, the instruction-tuned chat release, and a medical variant someone already built (BioMistral-7B, MedGemma-4B, MediTron-7B, MedLlama-13B). That is 4 × 3 = 12 starting models. Each gets three adaptations, CPT, SFT, and CPT followed by SFT, so the study trains 36 models and evaluates 48 once the untouched bases are counted.

Varying the starting point separates the recipe from the starting model. If you adapt a single model and its score rises, you cannot tell how much of the result belongs to the recipe and how much to the model you happened to start from. With three initializations per family, a strategy that helps everywhere looks different from one that helps only an instruction-tuned base. The paper also notes the families differ in language coverage: Mistral and Gemma are described as explicitly multilingual, Llama-2 as primarily English-centric, which shows up in the cross-lingual results.

How continual pretraining and supervised fine-tuning differ

The two strategies train on different data and update different amounts of the model. CPT trains on NACHOS, specifically its small variant: about 4 GB of unlabeled French medical text crawled from the web, 25.3 million sentences sampled from the 7.4 GB full corpus. There are no questions and no answers in it, only domain prose. SFT trains on MedInjection-FR: 543,505 instruction-response pairs from its train and validation splits. About 83% are single-answer multiple-choice questions (MCQU), about 6% are multiple-choice questions with several correct options (MCQ), and about 11% are open-ended questions with a written reference answer (OEQA). Pick a strategy below to see which corpus feeds the model, how much of it trains, and what that cost on a 7B model:

CPT runs the original pretraining objective, predicting each next token, on in-domain text. The model sees French clinical vocabulary, drug names and the phrasing of medical writing, and every weight moves to lower its loss on that text. The paper trains all parameters for three epochs with AdamW (learning rate 2e-5, or 1e-4 for Gemma). This is the decoder-model version of what Gururangan and colleagues called domain-adaptive pretraining in 2020; their experiments used masked-language encoders, but the procedure is the same second pass over a domain corpus.

SFT trains on the instruction-response format: each example is an instruction, a question with its options, and the labeled answer, formatted as one text. The loss is still next-token prediction; in the authors' sft.py the labels are a copy of the input tokens, so the prompt is not masked out and the model is trained on the instruction and the answer alike. What SFT changes is mostly behavior, for example answering with the option letters the grader expects. It runs ten epochs through a parameter-efficient adapter rather than through the full model. The adapter is DoRA (weight-decomposed low-rank adaptation), an extension of LoRA. LoRA freezes a pretrained weight matrix $W_0$ and learns a low-rank update $BA$, with $B$ of shape $d_{\text{out}}\times r$ and $A$ of shape $r\times d_{\text{in}}$. DoRA splits the updated matrix into a direction and a length per column, and learns the length separately:

In this equation $\lVert\cdot\rVert_c$ is the norm of each column, so the fraction is the matrix with every column rescaled to unit length, and $m$ is a trainable vector with one length per column. $W_0$ stays frozen; $B$, $A$ and $m$ train. The paper uses rank $r=16$ on all seven projection matrices of every transformer layer (query, key, value, output, and the MLP's gate, up and down). For Mistral-7B those matrices are 4096 × 4096 (query, output), 1024 × 4096 (key, value), and 14336 × 4096 or 4096 × 14336 (MLP). A rank-16 pair costs $16(d_{\text{in}}+d_{\text{out}})$ weights, which sums to 1,310,720 per layer, plus 43,008 magnitude entries. Over 32 layers that is 43.3 million trainable weights against 7.24 billion in the model, 0.60%. The paper reports 0.602% for DoRA in its Table 4.

The obvious alternative is to fine-tune every weight on the labeled set too. The paper tried it on FrenchMedMCQA before settling on the design (Table 4):

The full fine-tune more than halves the exact-match score. The paper reports that its training loss converged faster and its validation performance was worse, the usual signature of overfitting a narrow labeled set. So the split is deliberate: all weights for CPT, whose job is to absorb a large unlabeled corpus, and a 0.6% adapter for SFT, which also keeps SFT cheap. The whole recipe fits in a few lines:

# one cell of the grid: family x init, e.g. Mistral-7B x Instruct
base = load("Mistral-7B-Instruct-v0.1")

# CPT: next-token loss on unlabeled text, every weight trainable
cpt = train(copy(base), NACHOS_small,          # ~4 GB, 25.3M sentences
            loss=next_token, trainable="all",
            epochs=3, lr=2e-5, opt="AdamW", wd=0.01)

# SFT: next-token loss on the whole formatted example (instruction,
# question, options, answer); base frozen, only a DoRA adapter on
# q,k,v,o,gate,up,down trains
sft     = train(add_dora(copy(base), r=16, alpha=16, dropout=0.05),
                MedInjection_FR, epochs=10, lr=2e-5)
cpt_sft = train(add_dora(copy(cpt), r=16, alpha=16, dropout=0.05),
                MedInjection_FR, epochs=10, lr=2e-5)
# (Gemma uses lr=1e-4 for both stages)

for m in [base, cpt, sft, cpt_sft]:          # 4 models per cell, 48 total
    mcqa = evaluate(m, test_mcqa, decode="greedy",
                    allowed=answer_letters, shuffles=3)   # EM, Jaccard
    oeqa = evaluate(m, test_oeqa, decode="greedy")       # ROUGE-L, judge
    p    = paired_bootstrap(m, base, resamples=10_000)   # vs alpha/12

How the answers are scored

The test split of MedInjection-FR has 14,533 native French examples and 13,293 examples machine-translated from English benchmarks such as MedQA, MedMCQA and MMLU. Its multiple-choice part holds 17,622 single-answer questions and 4,691 multi-answer ones. All evaluation is zero-shot: the model sees the question and no worked examples.

Three evaluation choices keep the multiple-choice scores clean. Decoding is greedy and deterministic. In the main results the output vocabulary is also restricted to the valid option letters (constrained decoding), so a model cannot lose points by rambling or by inventing an option F; unconstrained greedy results are in the appendix. And each question is asked three times with its options shuffled, with the results aggregated, so a model that favors whatever sits in slot A does not get credit for it.

A single-answer question is scored by exact match over the N test questions:

where $y_i$ is the gold answer, $\hat{y}_i$ the prediction, and the indicator is 1 when they match. For a multi-answer question, $y_i$ and $\hat{y}_i$ are sets of option letters and EM requires the two sets to be identical. Picking A and B when the gold set is A, B and C scores zero. So the paper adds a partial-credit score:

Per question this is intersection over union: options both sets share, divided by options either set contains. The paper calls it the "Hamming score", but the formula is the Jaccard index, also known as example-based accuracy (Godbole and Sarawagi, 2004). The classical Hamming score is a different number: it checks every option slot and also credits options correctly left out. Take five options, gold $\{A,B\}$, prediction $\{A,C\}$. One option is shared and three appear in either set, so Jaccard gives $1/3$. Slot by slot, A is right, B is missed, C is an extra, and D and E are correctly omitted, so the Hamming score is $3/5 = 0.6$. This page uses the paper's formula and calls it Jaccard. Toggle the gold and predicted options to compare the three scores:

Two consequences come up later. On a multi-answer question, missing one gold option and adding one wrong option both drop EM to zero while Jaccard stays between 0.5 and 0.67 in the presets. On a single-answer question the three scores agree, since there is only one option to get right. The paper's headline multiple-choice number, "aggregated EM", is the plain average of the MCQ EM and the MCQU EM.

The multiple-choice result: SFT gets most of the gain

Across the 12 starting models, the paper reports one ordering often enough to write it as a pattern:

Take Mistral-7B General. Its aggregated EM is 14.44 untouched, 15.37 after CPT, 19.06 after SFT, and 19.21 after CPT+SFT. SFT adds 4.62 points for $361 of compute. Running CPT first adds another 0.15 points and $1,216, so the combined recipe costs 4.4 times as much as SFT alone for that 0.15. CPT+SFT ranks first in more configurations than any other strategy, but when it wins its margin over SFT rarely exceeds 1.3 points, and in some configurations SFT is ahead: by 3.12 points on Llama-7B Instruct (17.22 against 14.10) and by 1.44 on Mistral-7B Instruct.

The next figure plots each strategy by its training cost and its aggregated EM, using the paper's cost table for the model's size. Hollow points are the ones the paper's significance test could not separate from the base model. Switch the x axis to emissions to see the carbon side:

The cost ratio of CPT+SFT to SFT depends on model size because the two stages ran on different hardware configurations: about 3.2 times at 4B ($2,657 against $832), 4.4 times at 7B ($1,578 against $361), and 5.4 times at 13B ($7,473 against $1,391). The carbon ratio is larger than the cost ratio. At 7B, Table 14 gives 37,779 g CO2e for CPT+SFT against 4,883 g for SFT, about 7.7 times; at 4B and 13B it is about 5.4 times each. These dollar figures exclude the human effort of building half a million labeled pairs, which the paper lists as a limitation.

CPT on its own gives unreliable gains. Its change from the base model is not statistically significant for 8 of the 12 models (MedGemma, all three Llama-13B variants, Llama-7B General and Instruct, Mistral-7B General and Medical), and in some cells it drops below the base: Gemma General goes from 14.68 to 13.28. Reading domain text without any questions in it moves the model's knowledge but does not by itself teach it to answer a multiple-choice question in the expected format; the SFT stage does that.

For multi-answer EM, the best score in each family always comes from an instruction-tuned variant, and the paper calls instruction-tuned models the strongest baseline. For the aggregated score no single initialization wins everywhere, and a medical initialization on its own does not reliably help.

What the significance tests do and do not show

A one-point gap on a test set can come from sampling: a different draw of questions from the same distribution might reverse it. The paper checks each comparison with a paired percentile bootstrap. Both models answer the same questions, so each question yields a pair of outcomes. The test resamples the questions with replacement, 10,000 times, keeping each pair together, and recomputes the accuracy gap on every resample. The spread of those 10,000 gaps estimates how much the gap would move across test sets. If almost all of them fall on one side of zero, the gap is called significant. In the authors' bootstrap.py, with k the number of resampled gaps on the less populated side of zero, the two-sided p-value is:

For the aggregated score the script resamples the multi-answer and single-answer questions separately and averages the two means, mirroring how aggregated EM is defined.

The paper runs 12 strategy comparisons per family, and with many tests some will clear a 5% threshold by luck. The Bonferroni correction divides the threshold by the number of tests: a comparison counts only if its p-value is below $0.05/12 \approx 0.0042$ (and $0.05/9$ for the initialization comparisons). In the figure below, two simulated models answer N questions and disagree on a share of them; the slider sets the true gap. The histogram is the 10,000 bootstrap gaps, the teal band is the plain 95% interval, and the violet band is the wider interval the Bonferroni threshold implies:

A real case from the paper's Appendix J sits between the two bands. For Llama-13B General, CPT against its base has a 95% interval of +0.25 to +2.02 points, entirely above zero, and a p-value of 0.0106. That is below 0.05 and above 0.0042, so after the correction the paper counts it as not significant. The appendix tables are full of p = 0.0002 because that is the formula's floor: with k = 0 it gives 2/10,001.

These tests compare each adapted model with its base model, and CPT+SFT with CPT. Appendix J has no direct test of CPT+SFT against SFT, so the tests cannot say whether CPT+SFT's edge over SFT is real. Against the base, SFT or CPT+SFT fails to reach significance in exactly two configurations: on Gemma-4B Instruct neither does, and on Llama-7B Instruct CPT+SFT does not. The paper's recommendation for SFT therefore rests on cost: CPT+SFT's improvements over SFT are, in the conclusion's words, "typically small and not consistently statistically significant", and they come at 3.2 to 5.4 times the price.

Why multi-answer exact match is so low

Multi-answer EM in Table 1 runs from 0.09% to 7.77%. Read alone, that says the models almost never get these questions right. The Jaccard score from Eq. (2) gives a different picture. The figure shows both for the Mistral family:

For the Mistral rows, Jaccard is between 3.4 times EM (Instruct SFT, 23.42 against 6.80) and 14.6 times EM (General base, 5.40 against 0.37); most rows fall between 4 and 9 times. The untouched general base is the low outlier on both scores. A high Jaccard with a low EM means the model usually has most of the gold options and misses the exact set by one omission or one extra.

The paper's error analysis measures this directly with a near-miss rate: the share of questions where every gold option is ranked among the model's top $k$ options (with $k$ the number of gold options) but the predicted set is still wrong. Across the twelve Mistral variants it is 0.173 to 0.256 on multi-answer questions, roughly one question in five, and adaptation does not lower it. CPT+SFT widens the probability gap between gold and non-gold options, so the ranking gets more confident, but greedy decoding still has to commit to a set of the right size, and the near-misses happen at that step.

Open-ended answers: overlap metrics vs the LLM judge

Open-ended questions have a reference answer but no single right string, so they need a different kind of score. The paper reports two kinds. Overlap metrics compare the generated answer to the reference: ROUGE-L scores the longest common subsequence of words, and BERTScore-F1 matches tokens by the similarity of their contextual embeddings from a BERT-style encoder. The second kind is an LLM judge: MedGemma-27B reads each answer and rates it. The authors chose it because it was the most stable judge in an earlier agreement study against physicians on 500 open-ended items.

The two kinds of score often pick different winners:

Counting from Table 1, ROUGE-L and the judge name different best strategies in 7 of the 12 configurations. The judge's pick is CPT+SFT in 6, the base model in 3, CPT in 3, and SFT in none. CPT is the strategy that most consistently raises ROUGE-L and BERTScore, and SFT often lowers them. Very few of the open-ended differences are significant: against the base, CPT is significant in three cases, SFT in two (all on 4B models), and CPT+SFT in none. When the judge's differences between initializations are significant, the medical initializations are the ones that lose.

The paper explains the split with answer length (Sec. 7.2): CPT models write longer answers, and recall-oriented overlap metrics reward longer answers. Its own Table 11 lists the mean and median answer length of every model, so the explanation can be checked. Plot length against each score:

The tables do not support the verbosity explanation for CPT. A CPT model writes shorter answers than its base in 9 of the 12 configurations. The Llama-13B General case from the previous figure is the extreme: the CPT model averages 26 words with a median of 9, against 168 for the base, and it has the highest ROUGE-L in the block. Across all 48 models, the correlation between mean length and ROUGE-L is −0.83, with BERTScore −0.66, and with the judge 0.04. The length claim holds in a narrower form: CPT+SFT writes longer answers than SFT in 10 of 12 configurations. The paper's second claim, that concise answers align with higher judge scores, also fails on the extreme cases. The shortest answers in the table come from Mistral-7B, Llama-7B and Llama-13B Instruct after SFT, 17 to 20 words, and in each of those three blocks SFT gets the lowest judge score of the four strategies. Short answers scoring well on ROUGE-L would fit a reference set of short answers, since ROUGE-L's F-score penalizes extra words through precision, but the paper does not report reference lengths, so the reason for CPT's overlap scores remains open.

Neither kind of score substitutes for a clinician. Overlap metrics penalize a correct answer that is phrased differently from the reference. LLM judges have documented biases toward answer position, length, and answers that resemble their own output (Zheng et al., 2023). This page adds a caveat the paper does not raise: the judge is a Gemma-family model, and the models it grades include Gemma-3-4B and MedGemma-4B. The authors base their recommendations on the multiple-choice results and treat the open-ended trends cautiously, partly because open-ended questions are only about 11% of the SFT data.

French training, English gains

Several of the test benchmarks exist in both languages: the English original and its French machine translation. That allows a direct check of what French-only adaptation does in English. Adapting on French medical data improves the native English single-answer benchmarks too, and sometimes by more than it improves their French translations, which the paper interprets as cross-lingual transfer of medical knowledge.

Which language scores higher depends on the family. The bars show EM on the French translation and on the English original; under each pair is the English lead (English minus French) at the base model and after the selected strategy:

Mistral scores higher on the French translations than on English before and after every adaptation, with the differences statistically significant. Gemma and Llama start ahead in English and stay ahead, and for them the adaptation gains are often larger in English, so the English lead grows. Llama-13B Instruct is the clearest case: 21.68 French and 23.60 English at base, a 1.9-point lead, then 30.10 French and 46.99 English after SFT, a 16.9-point lead. Llama-7B is the exception at the base level: all three of its bases score slightly higher in French (General 25.48 against 25.17, Instruct 24.34 against 23.44, Medical 24.19 against 23.68), and the paper reports the difference as not significant (Table 13 marks it so for the General and Medical bases). After SFT, English overtakes for all three; after CPT the Instruct and Medical models still score slightly higher in French, as does the Instruct model after CPT+SFT. Several cells sit around 23 to 25% in both languages, and some of those models score 0.00 in both languages under unconstrained greedy decoding (Llama-7B and Llama-13B Instruct at base), so their constrained scores mostly reflect forced picks among the options; a "lead" there carries little information. The paper reads the pattern as Mistral encoding French better while Gemma and Llama have stronger English representations.

Translated benchmarks inflate accuracy and confidence

Many "French" medical benchmarks are English benchmarks run through machine translation. The paper tests whether that substitution is neutral by comparing two single-answer sets of similar size: MediQAl, written in French (4,343 questions), and MedMCQA, translated from English (4,183 questions). The questions differ, so the comparison is between benchmarks, but every model scores higher on the translated set, and most also become more confident. The paper publishes these results only as scatter plots; the figure below re-draws its points:

Every model scores higher on the translated benchmark, by 2.9 to 16.2 points. By our reading of the plots, entropy over the answer options falls for 45 of the 48 models, so their predictions are more peaked, and for 39 of 48 the probability on a wrong chosen answer rises. Higher accuracy alone could mean the translated test is easier. Higher confidence on the questions the model still gets wrong points to a calibration change as well, in the overconfident direction. A model validated only on translated benchmarks gets an over-optimistic estimate of how it will do on French written by French clinicians, for accuracy and for how much its confidence can be trusted. A plausible mechanism, which the paper does not test, is that translated questions keep the structure and phrasing of their English source, which is closer to the English-heavy text the models were pretrained on.

Which adaptation strategy to use

The paper's guidelines are organized by the data you have:

• Only unlabeled domain text. CPT gives small and unstable multiple-choice gains (not significant for 8 of 12 models) and should not be used on its own. Its gains show up mainly on the overlap metrics for open-ended answers, where the paper's length explanation does not match its Table 11.
• Labeled question-answer data. SFT with a parameter-efficient adapter is the default: it gives the best performance for the compute across all families and frequently matches or beats CPT+SFT.
• Both. CPT+SFT most often scores highest, by margins that rarely exceed 1.3 points (the paper never tests them against SFT directly), at 3.2 to 5.4 times the cost and 5.4 to 7.7 times the emissions of SFT. Use it when the last point is worth that.
• Starting model. An instruction-tuned base is the strongest starting point for French medical multiple choice. A model already adapted to medicine is not a reliable head start.

Three evaluation findings hold whichever strategy you pick. Multi-answer exact match understates models that are usually one option off. Overlap metrics and an LLM judge rank open-ended answers differently, and neither tracks answer length the way the paper says. Translated benchmarks inflate both accuracy and confidence. The study is limited to medicine and French, to zero-shot evaluation, and to CPT and SFT (no preference optimization or reinforcement learning), and its check of whether NACHOS was already in the models' pretraining data was inconclusive.