The $18 Question
Imagine two people using ChatGPT to process the same information. Same content, same meaning, same value. But one person pays $29, and the other pays $47.
The difference? One writes in English. The other writes in Hindi.
This isn’t a pricing tier. It’s not a regional markup. It’s a consequence of how language models count words—or more precisely, how they count “tokens.” And it affects 544 million Indian users of ChatGPT, along with hundreds of millions more who don’t speak English as their primary language.
I spent the last few weeks analyzing OpenAI’s tokenizers across 12 languages, and what I found reveals a systematic “language premium” built into the architecture of modern AI.
What’s a Token, Anyway?
If you’ve used ChatGPT’s API, you know you pay per “token.” OpenAI describes this as “roughly 4 characters” or “about ¾ of a word” in English. Seems straightforward.
But here’s what they don’t tell you: that fraction changes dramatically depending on your language.
Let me show you what I mean. Take these three words:
English: “hello” → 1 token
Arabic: “مرحبا” (hello) → 2 tokens
Hindi: “नमस्ते” (hello) → 4 tokens
Same semantic content. Wildly different token counts. And since APIs charge per million tokens ($2.50 per million of input tokens for GPT-4o), this difference isn’t academic—it’s expensive.
The Numbers Are Worse Than You Think
I analyzed the top 50,000 most frequent words in 12 languages using OpenAI’s latest tokenizer (o200k_base, released with GPT-4o in 2024 and use on every OpenAI models ever since). Here’s what actual text looks like when you tokenize it:
English: 1.16 tokens per word
Chinese: 1.32 tokens per word
French: 1.52 tokens per word (+31%)
German: 1.52 tokens per word (+31%)
Japanese: 1.55 tokens per word (+33%)
Spanish: 1.56 tokens per word (+34%)
Portuguese: 1.60 tokens per word (+38%)
Italian: 1.66 tokens per word (+43%)
Indonesian: 1.84 tokens per word (+58%)
Hindi: 1.89 tokens per word (+63%)
Russian: 1.96 tokens per word (+69%)
Arabic: 1.97 tokens per word (+70%)
For every dollar an English speaker spends on ChatGPT, a Hindi speaker pays $1.63 for the same information.
This isn’t a small edge case. India is the second-largest ChatGPT market with 544 million monthly visits (vs. 883 million in the US). Indonesia ranks 5th with 216 million visits. We’re talking about hundreds of millions of people systematically paying more for the same service.
But Wait, Didn’t OpenAI Fix This?
Yes and no. The evolution is actually fascinating—and shows both progress and persistent problems.
I compared three generations of OpenAI’s tokenizers, from 2021 to 2024:
Hindi’s journey: 2021 (p50k_base): 6.55 tokens/word 2023 (cl100k_base): 4.19 tokens/word 2024 (o200k_base): 1.89 tokens/word Improvement: -71%
Russian’s journey: 2021: 5.16 tokens/word 2023: 2.74 tokens/word 2024: 1.96 tokens/word Improvement: -62%
English’s journey: 2021: 1.19 tokens/word 2023: 1.17 tokens/word 2024: 1.16 tokens/word Improvement: -2.4%
The good news: OpenAI made dramatic improvements for non-Latin scripts. The 2021 tokenizer was brutally unfair—Hindi speakers paid 5.5× more than English speakers.
The bad news: Even after a 71% improvement, Hindi speakers still pay 63% more than English speakers. And English barely changed at all, suggesting it was already optimized from day one.
Three Dimensions of Inequality
This tokenization disparity creates problems across three dimensions:
Economic Inequality
Direct cost disparity. Using the GPT-4o pricing of $2.50 per million input tokens:
Language Cost per 1M words Premium vs English
English $2.90 —
French $3.81 +31%
Indonesian $4.60 +58%
Hindi $4.73 +63%
Russian $4.90 +69%
Arabic $4.93 +70%
At scale, this adds up fast. An Indian company processing 100 million words per month pays $473 vs $290 for an equivalent US company—a $2196 annual “language tax.”
Technical Barriers
Context window limitations. GPT-4o has a 128,000 token context window. That sounds like a lot, but:
English speakers can fit ~110,000 words
French speakers can fit ~84,000 words (24% less)
Hindi speakers can fit ~68,000 words (38% less)
You’re not just paying more—you’re getting less utility from the same technical infrastructure.
Performance Gaps
Processing time scales with tokens. More tokens means:
Longer inference times
Higher latency
Faster rate limiting
If you’re building an AI product in Hindi, your users experience measurably slower performance than English users, even with identical hardware.
Why Does This Happen?
Three root causes explain these disparities:
1. Training Data Imbalance
English dominates the internet. Even “multilingual” datasets like mC4 contain 30%+ English content. When you train a Byte-Pair Encoding (BPE) tokenizer on this data, it naturally allocates more vocabulary slots to frequent English patterns.
The tokenizer learns: “Oh, ‘the’ appears 50 billion times? Better make that a single token.” But a Hindi word appearing 50 million times? That might still get fragmented.
2. UTF-8 Encoding Bias
This is subtle but important. OpenAI uses byte-level BPE, which operates on UTF-8 bytes rather than characters.
UTF-8 represents ASCII (English-compatible characters) with 1 byte each. But:
Latin letters with accents: 2 bytes
Cyrillic, Arabic, Devanagari: 2-3 bytes
Chinese/Japanese: 3-4 bytes
Even if BPE were completely script-agnostic (it’s not), the UTF-8 encoding makes English patterns “shorter” in byte sequences. This makes them more likely to merge early in the BPE training process, receiving dedicated tokens.
3. Morphological Complexity
Languages with rich morphology (lots of verb conjugations, case markings, agglutination) generate more unique word forms from smaller root vocabularies.
French has ~100 conjugated forms per verb. Turkish can generate thousands of forms per root through agglutination. English has ~5 forms per verb.
More unique forms = vocabulary spread thinner = fewer full-word tokens = more fragmentation.
Can This Be Fixed?
The fascinating part is that there are actually two separate problems here, requiring different solutions:
Problem 1: Economic Fairness (Solvable Today)
The issue: Users pay the same per-token rate but tokens represent different amounts of information across languages.
The solution: Change the pricing model.
OpenAI (and other API providers) could switch to:
Per-word pricing (like Google Cloud Translation does)
Per-character pricing (normalizes for text length)
Language-adjusted pricing (apply multipliers to achieve parity)
This requires zero technical changes. It’s a business decision. You don’t need to retrain any models—just change the billing logic.
Problem 2: Technical Fairness (Requires Retraining)
The issue: Context windows are measured in tokens, so non-English speakers get less utility.
The solution: Train new tokenizers with explicit parity constraints.
This is harder. Some approaches:
Monolingual tokenizer ensembles — Train separate BPE tokenizers for each language, then merge vocabularies while maintaining parity. Allocate vocabulary budget proportionally to speaker populations, not corpus frequency.
Character-level models — Models like ByT5 and CANINE eliminate learned tokenization entirely, achieving perfect parity by definition. But UTF-8 still introduces bias (1 byte for ‘A’ vs 3 for ‘中’). True parity requires UTF-32 encoding, which quadruples sequence length for English.
Parity-aware BPE — Modify the BPE training algorithm to optimize for cross-lingual parity rather than total corpus compression. Use multi-objective optimization that balances efficiency across languages.
What Actually Happened (And What It Means)
Let me show you something interesting about OpenAI’s o200k_base tokenizer that suggests they are thinking about this problem:
Vocabulary allocation (estimated):
The tokenizer went from 50,000 tokens (p50k) to 200,000 tokens (o200k). That’s 150,000 new tokens.
Where did they go?
Non-Latin scripts saw 50-71% efficiency gains
Latin-script languages saw 13-23% gains
English saw 2.4% gain
This suggests OpenAI allocated the majority of those new tokens to improving non-Latin script coverage. They didn’t just scale everything proportionally—they deliberately addressed the worst disparities.
But they stopped short of parity.
Why? Probably because:
Legacy compatibility — Changing token-to-text mappings breaks existing fine-tuned models
English-first incentives — English users (their largest market) don’t want degraded performance
Technical constraints — Achieving true parity might require radical changes (character-level models, UTF-32 encoding)
Still, the trajectory is clear: each tokenizer generation reduces disparity. The question is whether we’ll accept gradual improvement or demand systematic solutions.
Why This Matters Beyond API Costs
You might think: “Okay, so some people pay a bit more. Is that really a big deal?”
Yes. Here’s why:
1. It Affects Who Can Build AI Products
If you’re building an AI startup in Indonesia, your unit economics are 58% worse than a competitor in the US—before considering any other factors. This creates a structural disadvantage for non-English AI entrepreneurship.
2. It Shapes Language Use Patterns
Bilingual users face a choice: pay more to use their native language, or switch to English for cheaper service. Over time, this creates incentives to abandon native languages, accelerating linguistic homogenization.
3. It Reflects Broader Inequality
This isn’t just about tokenization. It’s about who gets considered when building foundational technology. English optimization was baked in from day one. Other languages? Added later, incompletely, as afterthoughts.
The same pattern appears in:
Speech recognition (higher error rates for non-native accents)
Image generation (cultural bias in visual concepts)
Knowledge bases (English Wikipedia has 6.8M articles, Hindi has 150K)
Tokenization is just one visible manifestation of a deeper problem: AI systems are built primarily for English speakers, then adapted for everyone else.
What You Can Do
If you work with AI systems:
As a user:
Track your token consumption by language
Calculate your actual per-word costs
Advocate for transparent, language-normalized pricing
As a developer:
Use local tokenizers (like tiktoken) to calculate costs accurately before API calls
Implement caching strategies to reduce redundant tokenization
As a business:
Factor language overhead into your unit economics
Test APIs from multiple providers (OpenAI, Anthropic, Google) to find best pricing for your language
Push vendors for language-adjusted pricing tiers
As a researcher:
Publish tokenization benchmarks for your language
Advocate for parity-aware tokenizer designs
Contribute to open-source multilingual tokenization projects
The Bottom Line
Here’s what we’ve learned:
OpenAI has made real progress — 50-71% efficiency improvements for non-Latin scripts
Significant disparities remain — Hindi users still pay 63% more than English users
This affects hundreds of millions of people — India alone has 544M ChatGPT users Economic fairness is solvable today — Change pricing models without retraining
Technical fairness requires innovation — New tokenizer designs or character-level models
The most important finding: This isn’t inevitable. These disparities are design choices, not natural laws. They can be changed.
The question is whether AI companies will prioritize multilingual equity, or continue optimizing primarily for English while offering gradual improvements to everyone else.
As AI becomes global infrastructure—as essential as electricity or internet access—we need to demand that it serves all languages equitably. Not as an afterthought. Not with a 30-70% premium. But with genuine parity.
Because language shouldn’t determine the price of intelligence.
Methodology Note
This analysis used:
Data: Top 50,000 most frequent words per language from OpenSubtitles 2018 corpus
Tokenizers: OpenAI’s p50k_base (2021), cl100k_base (2023), and o200k_base (2024)
Metric: Frequency-weighted mean tokens per word (weights by actual usage patterns)
Languages: English, French, Spanish, German, Portuguese, Italian, Indonesian, Chinese, Japanese, Russian, Hindi, Arabic
Full methodology and data available in the accompanying research paper.
If you found this analysis valuable, please share it with others working on multilingual AI systems. The more visibility these disparities get, the faster we can push for equitable solutions.
Subscribe for more deep dives into the hidden biases and design choices shaping AI systems.
Related Reading:
Ahia et al. (2023): “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models”
Petrov et al. (2023): “Language Model Tokenizers Introduce Unfairness Between Languages”
Rust et al. (2021): “How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models”