What Is a Token? How LLMs Split Text and Why Your API Bill Depends on It
Byte-pair encoding in plain language: why "strawberry" is one token but "strawberrys" is three, how different models tokenize the same text, and how to estimate cost before you call the API.
Tokens are not words
Language models never see words or characters — they see integer IDs from a fixed vocabulary of subword pieces. The tokenizer greedily merges frequent byte sequences (byte-pair encoding), so common English words become single tokens while rare words, typos, and non-English text shatter into several. As a rule of thumb, English prose runs about 4 characters or 0.75 words per token — but the rule breaks exactly where it matters.
- Code tokenizes worse than prose: indentation, brackets and camelCase identifiers all fragment.
- Non-Latin scripts can cost 2–4× more tokens per sentence than English.
- JSON keys, quotes and braces add real overhead — a verbose schema can double a prompt’s cost.
- Numbers split unpredictably:
2026may be one token while20261is two.
Why every model counts differently
Each model family ships its own vocabulary: GPT-4o uses the o200k encoding, older GPT-4 used cl100k, and Claude and Gemini use their own tokenizers. The same paragraph can differ 10–20% in token count between models, which means a prompt that fits one model’s context window can overflow another’s. Never reuse a count from one provider as an estimate for another when you are near a limit.
Context windows and the cost asymmetry
A context window is the model’s total budget for input plus output combined. Two properties drive real-world bills: input tokens are usually several times cheaper than output tokens, and in a chat, the entire history is re-sent — and re-billed — on every turn. A long system prompt is a recurring cost, not a one-time one; a conversation’s cost grows quadratically with its length unless you summarize or truncate history.
Practical prompt economics
- Prefer terse field names in JSON payloads the model must read or write (
qtyvsquantityOrdered) when schemas are large. - Strip boilerplate (headers, footers, navigation) from retrieved documents before stuffing them into context.
- Cap output with explicit length instructions and
max_tokens— output is the expensive direction. - For repeated system prompts, use provider prompt-caching where available; cached input is billed at a fraction.