Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

A cheap fine-tune usually matches the recipe that costs four times more.

A controlled study of French medical question answering, across three model families and three starting points. Supervised fine-tuning gives the best accuracy per dollar, continual pretraining alone is unstable, and translated benchmarks inflate both accuracy and confidence.

A hospital wants a medical assistant that works in French. It has a general-purpose language model, a pile of French medical text, and a fixed GPU budget, and it has to decide how to spend that budget.

Almost every strong language model is trained mostly on general-domain English. To use one for medicine, and in a language that is not English, you adapt it. In practice adaptation means two things, often stacked: keep pretraining the model on a pile of domain text, and fine-tune it on labeled question-and-answer examples. The first is called continual pretraining (CPT), the second supervised fine-tuning (SFT). Both are known to help. What nobody had pinned down is when each is worth its cost, especially outside English, where domain data and compute are both scarce.

This paper runs the comparison as a controlled experiment. It holds the task fixed (French medical QA) and varies three things on purpose: the base model, the adaptation strategy, and the way the answer is scored. The reason for the care is a confound that trips up most prior work: if you only ever adapt one model, you cannot tell whether a gain came from the adaptation or from the model you happened to start with. Vary the starting point too, and the two effects come apart. The experiment yields a set of guidelines a practitioner can act on, and a couple of warnings about the benchmarks everyone reports.

The choice: which knob to turn

The experiment is a grid. Three model families (Mistral-7B, Gemma-4B, and Llama-2 at 7B and 13B), each in three starting states: a plain general-purpose base, an instruction-tuned chat variant, and a version already pushed toward medicine by its authors (BioMistral, MedGemma, MediTron, MedLlama). On each of those, three adaptation strategies: CPT alone, SFT alone, and CPT followed by SFT. The paper sweeps that grid, reading one axis at a time.

The two adaptation strategies feed on different data. CPT reads NACHOS, about 4 GB of unlabeled French medical text scraped from health websites: no questions, no answers, just the raw language of the domain. SFT reads MedInjection-FR, 543,505 labeled instruction-and-response pairs, most of them multiple-choice questions with their correct answers. Pick a strategy below and watch which data source feeds the model, how much of the model it trains, and what that costs:

That footprint difference, the whole model versus a sliver of it, is why the two strategies sit at opposite ends of the budget. On a 7B model the paper measures CPT at about $1,216 of compute and SFT at about $361, so CPT+SFT lands near $1,577, more than four times the cost of SFT and with roughly four times the carbon. Every claim that follows is really a claim about whether the extra spend buys anything.

CPT and SFT do different jobs

The word "fine-tuning" hides two different operations, and the results only make sense once you tell them apart. CPT keeps playing the original pretraining game, predict the next token, but now on in-domain text. It is exposure, not instruction: the model reads French medical prose and its whole internal representation drifts toward the vocabulary, the facts, and the phrasing of the domain. Because that drift has to reach across every layer, CPT updates all the weights. (This is the decoder version of what Gururangan and colleagues called domain-adaptive pretraining in 2020; their version used the masked-language objective on an encoder, but the idea is the same second pass over a domain corpus.)

SFT plays a supervised game instead: given an instruction, produce the labeled response. It teaches behavior and format, how to answer a medical multiple-choice question, not new raw knowledge. The supervised signal is narrow and the labeled set can be small, so training every weight on it risks memorizing the answer key and overwriting what the model already knew. To avoid that, SFT here trains a parameter-efficient adapter rather than the full model. It uses DoRA, a close cousin of LoRA: it splits each weight column into a length and a direction, freezes the base, and learns a small low-rank update to the direction plus a per-column rescaling. The trainable part is about 0.6% of the model. The recipe, all three strategies and the scoring, is short enough to read as pseudocode:

# the study grid: 3 families x 3 starts x 3 strategies
base = load(family, init)              # init: General / Instruct / Medical

# CPT: full-parameter, self-supervised next-token on domain text
cpt = continue_pretrain(base, NACHOS_4GB, epochs=3, full=True)

# SFT: a small DoRA adapter, supervised on labeled Q-A pairs
sft = finetune(base, MedInjection_FR, epochs=10, dora=True)  # ~0.6% params
cpt_sft = finetune(cpt, MedInjection_FR, epochs=10, dora=True)

# score: greedy + constrained decoding, then a paired bootstrap test
for m in [base, cpt, sft, cpt_sft]:
    acc = evaluate(m, test_fr, constrain_to_options=True)
    p   = paired_bootstrap(m, base, resamples=10_000)   # Bonferroni-corrected

Splitting the model this way is a design choice, and the paper checks it rather than asserting it. The obvious alternative is to fine-tune every weight on the labeled data. On a small medical multiple-choice set that did worse: the full fine-tune scored 0.1121 exact match while the DoRA adapter scored 0.2435. Training every weight on a small labeled set let the model overfit it, and the low-rank cap acted as a regularizer that forced a more general update. So the division is deliberate, full-parameter where the goal is broad knowledge, a small adapter where the goal is a narrow behavior on little data, which is why SFT stays cheap.

The multiple-choice verdict

Multiple-choice medical QA anchors the paper, because it can be scored without argument. Two flavors appear. Single-answer questions (MCQU) ask for exactly one option; multi-answer questions (MCQ) can have several correct options at once. Single-answer questions are scored by exact match, the fraction of questions answered exactly right:

For multi-answer questions, exact match is unforgiving: you get a point only if your set of chosen options is exactly the gold set, no partial credit. So the paper adds a partial-credit score, the overlap between the predicted set and the gold set:

The paper labels equation (2) the "Hamming score," but that is not what a multi-label textbook means by the name. The quantity written above, intersection over union, is the Jaccard index (the example-based accuracy of Godbole and Sarawagi, 2004). The classical Hamming score is a different number: it averages correctness over all option slots and also credits every option you correctly left out, so on a five-option question with two gold answers it drifts higher. Take gold $\{A,B\}$ and a prediction $\{A,C\}$: the Jaccard score is $1/3 \approx 0.33$, while the true Hamming score is $3/5 = 0.6$. The formula the paper prints is a perfectly standard partial-credit metric, just under a name the field usually reserves for something else, so read those numbers as Jaccard overlap, not Hamming. On single-answer questions the distinction vanishes, since both collapse to plain accuracy.

With the scoring pinned down, one ordering shows up again and again across families and starting points:

Adapting at all beats the untouched base by a wide margin. Among the strategies, CPT+SFT most often takes the top score, SFT is close behind, and CPT alone trails. So far this reads like a clean win for spending the most. Put cost on the other axis, though, and that reading falls apart. Each point below is one strategy, placed by its training cost and its aggregated multiple-choice accuracy; pick a family and a starting point and look at where the money goes:

When CPT+SFT does win, the margin is small: across the grid it beats SFT by more than 1.3 points in almost no case. And a bootstrap test (resample the test set 10,000 times, keep the strategies paired on each resample, then shrink the significance threshold with a Bonferroni correction for all the comparisons made) shows those small gains often sit inside the noise. For several models, neither CPT nor CPT+SFT clears the bar of being reliably better than the base. That non-significance does not mean the strategies are equal; the average still tilts toward CPT+SFT. It means the gap is small enough that this test set cannot tell it apart from sampling luck, and paying four times the cost for a difference you cannot measure is a poor trade. Hence the paper's headline: with labeled data in hand, SFT is the default, and CPT+SFT is justified only when you truly need the last fraction of a point and can afford it.

CPT alone behaves the least predictably. It sometimes helps, sometimes drops below the base, and for eight of the models its change over the base is not statistically significant at all. Reading a domain corpus with no labeled questions nudges the model's knowledge but does little for its multiple-choice accuracy on its own; the task supervision converts that knowledge into answers. Domain text is most useful as the first half of CPT+SFT, not as a standalone fix.

Why multi-answer scores look low

The tables carry an alarming-looking number: exact match on multi-answer questions is often in the low single digits, a few percent. Taken at face value that says the models are nearly useless at multi-answer medicine. The partial-credit score says otherwise, and the gap between the two explains why. Compare them for the Mistral family below, the exact-set match against the Jaccard overlap:

The partial score runs five to ten times the exact one. That means the model usually identifies most of the right options and misses the exact set by a label or two, which exact match throws away entirely. The paper's error analysis confirms it with a near-miss rate: on roughly a fifth of multi-answer questions the model ranks every gold option among its top choices yet still submits the wrong set, an omission here or an extra option there. That rate barely moves as you adapt the model. Adaptation makes the rankings more confident, but confidence in a ranking is not the same as committing to the exact set, so the hardest part of multi-answer questions is the one adaptation leaves mostly untouched. When you read the tiny multi-answer exact-match numbers, read them as a strict metric punishing near-misses, not as a model that has no idea.

Free-form answers: metrics disagree

Multiple choice is safe to score but narrow. Real medical questions want a written answer, so the paper also evaluates open-ended QA, and here the ground gets soft, because there is no single right string to match against. Two kinds of automatic score step in, and they do not agree. The first kind measures surface overlap with a reference answer: ROUGE-L counts the longest shared word sequence, BERTScore compares answers through contextual embeddings. The second kind hands each answer to a strong model, MedGemma-27B, and asks it to judge quality, the increasingly common "LLM-as-a-judge" setup. Watch the two disagree:

The disagreement has a mechanical cause, and it lives in the metrics, not in the judge. Overlap scores like ROUGE-L and BERTScore reward recall: a longer, term-dense answer covers more of the reference and ticks more boxes, and the score does not charge much for the padding around them. CPT, which floods the model with medical prose, tends to produce exactly those longer, jargon-heavy answers, so it climbs the overlap metrics. The judge, meanwhile, prefers the shorter, cleaner answers that instruction tuning and SFT produce. So the two families of score are measuring different things, surface coverage versus a proxy for quality, and a strategy can win one while losing the other. Notably this is the reverse of the usual worry about LLM judges, which is that they are fooled into preferring longer answers; here the length reward sits on the overlap metrics, and the judge leans the other way.

None of these open-ended scores is authoritative, and the paper treats them accordingly. Overlap metrics penalize a correct answer phrased differently from the reference. The judge is only a proxy for a clinician, with its own known biases (it can be swayed by answer order, by length, and by answers written in a style like its own), and here it is a Gemma-family model grading, among others, Gemma-family models, a self-preference confound. That is why the authors ground their conclusions in multiple choice and read the open-ended trends cautiously. So CPT writes answers that look thorough to overlap metrics, SFT writes answers a judge tends to prefer, and neither the metrics nor the judge should be mistaken for a doctor.

French training helps English too

The adaptation data is entirely French, so you would expect it to help French and do nothing for English. It helps English. Adapting on French medical text lifts the models' scores on the original English medical benchmarks as well, sometimes by more than it lifts French. The medical knowledge the model picks up (drug interactions, anatomy, the shape of a clinical question) does not live in one language; it rides on representations the model shares across languages, so improving it in French improves it in English too. This is transfer running in the less usual direction, from a lower-resource language back to English.

Which language a model reads better depends on what it was pretrained on. Move the slider through the strategies and watch both bars climb together, while the gap between them stays put:

Mistral, which was pretrained with real multilingual data, does at least as well on the French benchmarks as on the English ones. English-centric Llama-2, and even multilingual Gemma, keep an English edge even after all their new medical training is in French. The takeaway for a non-English deployment: a base model with genuine multilingual pretraining gives you more of your adaptation back in the target language, and the medical gains transfer across languages either way.

A benchmark trap: translated tests inflate the score

That cross-lingual result comes with a warning, because many "French" medical benchmarks are not native French; they are English benchmarks run through machine translation. The paper compares a genuinely French benchmark against a translated one and finds the translated test is not a neutral substitute. Every model scores higher on the translated benchmark, and it does so while getting more confident: its answer distributions grow peakier (lower entropy), and its confidence rises even on the questions it gets wrong.

Higher accuracy paired with higher confidence on wrong answers points to an easier test, not a smarter model. Translated questions carry the syntactic fingerprints of their English source, which can make the right option easier to pick out, so the model looks both better and surer than it is. The practical consequence is direct: a medical model validated only on translated benchmarks will read as more accurate and better calibrated than it will be in the clinic, where the French is real. Native-language evaluation is not a nicety here; it is the difference between an honest number and a flattering one.

What to actually do

The study lands on advice you can follow from what you have on hand. If all you have is unlabeled domain text, CPT alone buys little on multiple choice and can even hurt, so it is not worth doing on its own. If you have labeled question-answer data, reach for SFT: a parameter-efficient fine-tune that captures most of the available gain for a small fraction of the compute and carbon. If you have both kinds of data and genuinely need the top of the leaderboard, CPT+SFT will usually get you there, but expect a gain often too small to pass a significance test, at several times the cost. For a starting point, an instruction-tuned model is the strongest base for French medical multiple choice; a model that was already pushed toward medicine by its authors is not, on its own, a reliable head start.

And read the evaluation as carefully as the model. On multiple choice, tiny multi-answer exact-match scores hide models that are usually a label away from right. On open-ended answers, overlap metrics and an LLM judge measure different things and disagree, so neither settles quality alone. And a benchmark translated from English will tell you the model is better and more confident than it is. The cheapest adaptation is usually the right call; the real effort goes into reading the scores for what they actually measure.