LLM token cost by language: we measured 14 of them

How much more do other languages cost in tokens?

Non-English text costs between 7% and 54% more tokens than English on OpenAI's current tokenizer, and up to 4.4 times more on the previous one. We measured this on real sentence pairs rather than quoting a rule of thumb. The bottom line: a German or French bill runs roughly 20% above the English equivalent, and Hindi runs about 54% above.

A tokenizer is the component that splits text into the numeric pieces a language model reads and bills for. Every API prices input and output per million of those pieces. If your language splits into more pieces, the same message costs more and fills the context window faster.

What did we measure, and how?

We counted tokens for the same sentences in English and in each target language, then divided. The methodology is simple enough to repeat, which is the point.

OPUS-100 is a public English-centric parallel corpus of about one million sentence pairs, published on Hugging Face by Helsinki-NLP. We pulled up to 300 test-split pairs per language and kept only pairs where the English side had at least 40 characters and the translation differed from the English. That left between 163 pairs (Turkish) and 300 pairs (French, Chinese).

We counted tokens with OpenAI's tiktoken library on two encodings. cl100k_base is the encoding used by GPT-4 and GPT-3.5 Turbo, according to the OpenAI cookbook on counting tokens. o200k_base is the larger-vocabulary encoding used by GPT-4o and GPT-4o mini, per the same cookbook page. For each language we summed the tokens of all translations and divided by the summed tokens of their English originals.

One limit matters. We did not run Claude's or Gemini's tokenizers. Counting Claude tokens needs an API call, and we did not test Gemini at all. The ratios below describe OpenAI encodings only.

Which languages cost the most?

The table shows tokens needed for a translation divided by tokens for its English original, so 1.00 means no penalty. Each figure comes from our run of tiktoken on the OPUS-100 test split.

Language	Pairs	cl100k_base (GPT-4)	o200k_base (GPT-4o)
Chinese	300	1.61	1.07
Portuguese	264	1.28	1.08
Dutch	290	1.38	1.12
Korean	204	1.75	1.14
Spanish	224	1.33	1.15
German	274	1.38	1.18
Japanese	222	1.53	1.18
Turkish	163	1.50	1.20
French	300	1.39	1.21
Italian	257	1.39	1.25
Arabic	236	2.74	1.28
Polish	228	1.48	1.30
Russian	292	2.23	1.32
Hindi	225	4.36	1.54

Hindi is the most expensive language we tested on both encodings. Polish is the most expensive European Latin-script language, at 30% above English on the newer tokenizer.

What did the newer tokenizer change?

Moving from cl100k_base to o200k_base cut the penalty most sharply for non-Latin scripts. Hindi fell from 4.36 to 1.54, Arabic from 2.74 to 1.28 and Russian from 2.23 to 1.32. For example, Chinese dropped from 1.61 to 1.07, which makes it nearly free of any surcharge.

European Latin-script languages improved less. German moved from 1.38 to 1.18, and Italian from 1.39 to 1.25. A 15% to 30% surcharge is the new floor for these languages on OpenAI's tokenizer.

This also corrects a common claim. Several explainers say Chinese and Japanese cost two to four times more tokens than English. On cl100k_base our Chinese ratio was 1.61, and on o200k_base it was 1.07. The Tokenizer Tax study of Indian languages reports an average of 8.0 times the English token count on cl100k_base, according to the arXiv paper 2607.24276. Earlier work, Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages, also compared tokenizers across Indian languages. Our Hindi figure of 4.36 is lower because we compared short everyday sentences from a different corpus, not that study's FLORES-200 test set.

What does this mean for an API bill?

Multiply your English-equivalent bill by the ratio for your language. A team spending $1,000 a month on English prompts and replies would expect roughly $1,180 for the same German content, $1,210 for French and $1,300 for Polish on o200k_base. The same Hindi workload would run about $1,540.

Context windows shrink by the same factor. Consider a support bot that keeps the last 50,000 tokens of chat history. In Polish that window holds about 23% less conversation than in English, because each message uses 30% more tokens.

Output tokens follow the same rule. The model's German reply tokenizes like German text, so both sides of the bill grow. Price per million tokens stays the same, which is why the surcharge is invisible on the pricing page.

Does the same pattern hold for Claude?

Anthropic does not publish per-language ratios, but its pricing page says Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text, and that the exact increase depends on the content. Claude Sonnet 4.6 and earlier models use the previous tokenizer.

The Claude Sonnet 5 overview gives a second data point. It says 1M tokens is roughly 555,000 words on the current tokenizer, while earlier models fit about 750,000 words in 1M tokens. That implies about 35% more tokens per English word, by our arithmetic (750 divided by 555).

Claude Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens, according to the pricing page. The per-token price did not change, but the same text now uses more tokens, so a Claude bill can rise even when the price list does not. The Claude token counting documentation is the reliable way to measure your own prompts, since our OpenAI ratios do not transfer to Claude.

Why do some languages need more tokens?

Tokenizers learn their vocabulary from training text, and training text is mostly English. Common English words get a single token, while rarer scripts get chopped into smaller pieces. The Tokenizer Tax paper on arXiv links the penalty to failed merges that leave text as single-byte tokens, and it reports a correlation of 0.89 between merge failure and the tax.

OpenAI's own cookbook shows the effect on one phrase. The Japanese string "お誕生日おめでとう" takes 14 tokens with r50k_base, 9 with cl100k_base and 8 with o200k_base, according to the tiktoken cookbook. Each new vocabulary absorbed more of the script, so the same greeting got cheaper.

That is why the gap shrank so much for Hindi, Arabic and Russian. The larger vocabulary of o200k_base gave those scripts whole-word tokens that cl100k_base had to spell out piece by piece.

How much text fits in a context window, by language?

Less than the headline number suggests. Anthropic's model overview says a 1M-token window holds roughly 555,000 English words on the current Claude tokenizer, and 200,000 tokens is roughly 150,000 words. Dividing by our o200k_base ratios gives a rough capacity in English-equivalent words for other languages.

Language	Ratio vs English	Words of content in 1M tokens	In 200,000 tokens
English	1.00	555,000	150,000
Spanish	1.15	about 483,000	about 130,000
German	1.18	about 470,000	about 127,000
French	1.21	about 459,000	about 124,000
Italian	1.25	about 444,000	about 120,000
Polish	1.30	about 427,000	about 115,000
Russian	1.32	about 420,000	about 114,000
Hindi	1.54	about 360,000	about 97,000

This is an estimate, because it applies an OpenAI ratio to a Claude count. It still shows the direction: a Hindi support transcript uses about a third more of the window than the same conversation in English. For long documents, that decides whether a book fits in one prompt or needs splitting.

How can you run this test on your own text?

Count tokens for pairs of real messages and divide, using the same library we used. Install it with pip install tiktoken, then run this:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def ratio(pairs):
    english = sum(len(enc.encode(en)) for en, _ in pairs)
    other = sum(len(enc.encode(xx)) for _, xx in pairs)
    return other / english

# pairs = [("English sentence", "Translated sentence"), ...]
print(round(ratio(pairs), 2))

Use at least 100 pairs of your real traffic, not a single sample. Swap the encoding name to cl100k_base to compare with GPT-4. For Claude, use Anthropic's free token counting endpoint instead, since tiktoken does not match its tokenizer.

What are the limits of this measurement?

Four limits apply, and they all push in known directions. Treat the table as a guide to ordering, not as a quote for your invoice.

• One vendor. We measured two OpenAI encodings only. Claude, Gemini and open models will rank languages differently.
• Noisy corpus. OPUS-100 contains short, subtitle-like sentences. Technical or legal prose can tokenize differently.
• Sample size. Each language used between 163 and 300 pairs, so small differences such as 1.07 versus 1.08 are noise.
• Token ratio is not quality. A language that costs fewer tokens is not necessarily answered better. We did not test answer quality.

What other tokenizer and pricing news should you know about?

Three recent changes matter if you budget for multilingual AI. All come from Anthropic's pricing page, which is the primary source for each.

1. Sonnet 5's price rise was cancelled. The $2 and $10 per million token price announced as introductory is now standard, and the scheduled increase to $3 and $15 on September 1, 2026 will not occur.
2. Opus 5.5 is cheaper than Opus 5. It lists $4 and $20 per million tokens against $5 and $25.
3. Haiku 5.5 has a long-prompt tier. Prompts up to 100,000 tokens cost $0.10 and $0.50, and longer prompts cost $0.50 and $2.50.

We worked through what the new Claude tokenizer does to a bill in a separate post, "Claude's 30% token inflation: does your bill actually rise?" For a full cost example with a target language, see "Translating a 100,000-word book with Claude: what it costs."

Should you change anything?

Measure your own traffic before you change anything. Take 200 real messages in your target language, count them with the exact model's tokenizer and compare to the English equivalents. If the ratio is below 1.3, a model switch will not save much. If you serve Hindi, Arabic or Russian on an older GPT-4-era model, upgrading the tokenizer is worth more than any prompt trimming.

Three practical rules follow from the table:

1. Budget 1.2 to 1.3 times the English token count for French, Italian, Polish and Russian.
2. Budget about 1.5 times for Hindi even on the newer tokenizer.
3. Never reuse token counts from another model family when estimating cost.

Readers who compare model prices can pair this with our price per task comparison of AI coding models, and teams building agents can see what each design costs in our guide to agentic AI architecture patterns.