Which local LLM translates best? A reproducible eval

How good are local LLMs at translation, and do you actually need the cloud? A reproducible benchmark of 24 on-device, self-hosted, and cloud models translating into English, with the low-resource case (Afrikaans) front and centre. The headline: on Afrikaans→English a local 18 GB model lands in a statistical tie with frontier cloud. Same blinded Tatoeba sentences, same prompt, greedy decoding, scored multi-reference with COMET (meaning) and chrF++ (surface). Built to pick a translation model for Lector.

Open and reproducible: harness, every model's raw outputs, and the seeded test sets: github.com/heuwels/llm-lang-eval

Leaderboard

All 24 models across three languages on one zoomed COMET axis: each row a model (ranked by Afrikaans), one dot per language, the connector its cross-language spread. The zoom makes the tight differences legible; gaps under ~1.5 COMET are sampling noise (see Significance).

Deployment tier:on-device (laptop) self-hosted box (18 GB) cloud (OpenRouter)

The numbers

COMET (meaning, ×100) over chrF++ (surface), per language. chrF++ rewards character overlap with the reference, so it docks valid paraphrases ("scenery is magnificent" vs "landscape is breathtaking"); COMET scores meaning and credits them. Green = leading band (within ~1.5 COMET, a statistical tie, not a single winner). Rows share the chart's order (ranked by Afrikaans COMET), so near-ties can sit a place apart despite equal rounded scores. n=200 per language.

Cost, and what you can actually run

Cloud models fill the top of that board, but this is a study of local models, and most of the cloud field can't run on the box at all. The frontier APIs (GPT-5, Claude, Gemini) are closed weights, so self-hosting them was never an option. The open models I could reach through OpenRouter are mostly too big for the hardware: Llama 3.3 is 70B, Mistral Large is larger again, and even the 24B and 27B open models (Mistral Small, Gemma 2 and 3 at 27B) sit at or past the ceiling of an 18 GB Mac once the OS and the KV cache take their share. What genuinely fits is the on-device and self-hosted-box tiers, so here is that field on its own.

The strongest model that actually fits, gemma-4-12b-qat at 7.5 GB, is the same one sitting in the frontier band up top. apfel-foundation is Apple's built-in Foundation model, the one that ships with macOS, run through the Apfel harness. It scores respectably on what it answers, but it refused or errored on 26% of the Afrikaans sentences (52 of 200), against 8% in Spanish, and Apple doesn't list Afrikaans among its supported languages, which is why it sits near the bottom.

Cost: use what you've got

Cost barely enters into it. A translation is tiny, roughly 80 tokens, so the entire cloud sweep (24 models across three languages, plus the holdout and the cloze probe) came to $13.62 on OpenRouter, well under a cent per translation even on the frontier models. Per-token pricing still spans an order or two of magnitude, the frontier APIs against the cheap tiers like Gemini Flash or GPT-4o-mini, but at this token count the absolute bill is small whichever way you go.

What moves the decision is what you already own. A spare Mac is a sunk cost, so a local model is free per lookup beyond the electricity. An existing Claude plan is free at the margin too, within its limits, which is why I reach for the Anthropic OAuth route first. OpenRouter is the only one of the three that adds a real per-token bill, and it earns its place when you need a specific model you can't self-host or don't have a plan for. So the honest answer is usually to use what you've got: a spare box runs a local model, an existing plan already covers the lookups, and with neither, the cheap cloud tier is pennies per thousand. Since a 12B you can run at home already ties the frontier on Afrikaans, paying frontier rates per token buys very little for this particular job.

Contamination check: does it survive on unseen data?

Caveat on the caveat: exact training-cutoff dates aren't published for every model, and recently-added sentences may differ subtly in style or difficulty, so read a small Δ as "holds up", not as a precise measurement of contamination.

Parroting probe: memorisation, measured directly

Recovery = exact match of the blanked word. A large positive gap = memorisation; near-zero = genuine context prediction. n≈150 per cell, so gaps within ~±10 are noise.

Side-by-side generations

The numbers only say so much. Here are the actual translations where models disagree most. Green marks the highest per-sentence chrF++ for that sentence.

Afrikaans: where the models split

Sentences where models split into clear camps, several agreeing on one wording, several on another (one-off wordings collapsed to a tail). Count × tier-dots per camp; green = within ~6 chrF of the closest-to-reference camp.

German: where the models split

Sentences where models split into clear camps, several agreeing on one wording, several on another (one-off wordings collapsed to a tail). Count × tier-dots per camp; green = within ~6 chrF of the closest-to-reference camp.

Spanish: where the models split

Sentences where models split into clear camps, several agreeing on one wording, several on another (one-off wordings collapsed to a tail). Count × tier-dots per camp; green = within ~6 chrF of the closest-to-reference camp.

Methodology

Caveats