What Is a Token? How LLMs Split Text and Why Your API Bill Depends on It

Tokens are not words

Language models never see words or characters — they see integer IDs from a fixed vocabulary of subword pieces. The tokenizer greedily merges frequent byte sequences (byte-pair encoding), so common English words become single tokens while rare words, typos, and non-English text shatter into several. As a rule of thumb, English prose runs about 4 characters or 0.75 words per token — but the rule breaks exactly where it matters.

• Code tokenizes worse than prose: indentation, brackets and camelCase identifiers all fragment.
• Non-Latin scripts can cost 2–4× more tokens per sentence than English.
• JSON keys, quotes and braces add real overhead — a verbose schema can double a prompt’s cost.
• Numbers split unpredictably: 2026 may be one token while 20261 is two.

Why every model counts differently

Each model family ships its own vocabulary: GPT-4o uses the o200k encoding, older GPT-4 used cl100k, and Claude and Gemini use their own tokenizers. The same paragraph can differ 10–20% in token count between models, which means a prompt that fits one model’s context window can overflow another’s. Never reuse a count from one provider as an estimate for another when you are near a limit.

Context windows and the cost asymmetry

A context window is the model’s total budget for input plus output combined. Two properties drive real-world bills: input tokens are usually several times cheaper than output tokens, and in a chat, the entire history is re-sent — and re-billed — on every turn. A long system prompt is a recurring cost, not a one-time one; a conversation’s cost grows quadratically with its length unless you summarize or truncate history.

Estimating tokens when you cannot run the tokenizer

When you need a quick estimate in a spreadsheet or a back-of-envelope budget, use these ratios and then add a safety margin: English prose ≈ 1 token per 4 characters or 0.75 tokens per word; source code ≈ 1 token per 3 characters (punctuation and short identifiers fragment more); JSON ≈ 1 token per 2.5–3 characters (every quote, colon and brace costs); Chinese, Japanese and Korean ≈ 1–1.5 tokens per character; other non-Latin scripts often 2–3× the English rate. For a document of N words of English, budget about 1.3·N tokens and you will rarely be surprised.

The estimate breaks down exactly where cost matters: long repeated boilerplate (cheap to cache, expensive to resend), base64 blobs (about 1 token per 3 characters of pure noise — never put images in the prompt as text), and tables (whitespace-aligned columns are token-expensive; CSV or Markdown pipes are cheaper).

Context window arithmetic for real applications

• RAG pipelines: the retrieved chunks, the system prompt, the question *and* the answer must all fit. Reserve the answer budget first; a 128k window with 120k of retrieved context leaves no room to reply.
• Chat history: summarize or truncate old turns once the conversation passes a threshold; the alternative is quadratic cost growth and eventual overflow errors.
• Structured output: ask for the tersest schema that works; a verbose JSON response with long keys can double output tokens — the expensive direction.
• Long documents: chunk at semantic boundaries (sections, paragraphs) rather than fixed token counts, and overlap chunks by 10–15% so answers that straddle a boundary are not lost.

Practical prompt economics

• Prefer terse field names in JSON payloads the model must read or write (qty vs quantityOrdered) when schemas are large.
• Strip boilerplate (headers, footers, navigation) from retrieved documents before stuffing them into context.
• Cap output with explicit length instructions and max_tokens — output is the expensive direction.
• For repeated system prompts, use provider prompt-caching where available; cached input is billed at a fraction.