What is Token?

How it works

Before a model sees your prompt, a tokenizer splits it into pieces from a fixed vocabulary, often tens of thousands to a few hundred thousand entries. Common words are usually one token. Rare words, names and code are broken into sub-word pieces, so "tokenization" might become "token" plus "ization". Each token maps to an ID number, and the model works only with those IDs.

Most tokenizers use a sub-word method such as byte-pair encoding (BPE), which builds the vocabulary by repeatedly merging the most frequent character pairs in training text. That gives a compact vocabulary that can still spell any word.

Google's Gemini documentation gives a useful rule of thumb: a token is about 4 characters, and 100 tokens is about 60 to 80 English words. Other languages, code and numbers often use more tokens per word, and every model family has its own tokenizer, so the same text can produce different counts.

Why it matters

• Limits. A model's context window and maximum output are measured in tokens, not words or pages.
• Cost. API pricing is per million input and output tokens, with output usually priced higher.
• Speed. Generation speed is quoted in tokens per second, because the model produces one token per step during inference.
• Odd mistakes. Because models see tokens rather than letters, tasks like counting the letters in a word can trip them up.

Example

A 1,500-word English article is roughly 2,000 tokens. If you paste it into a prompt and ask for a 300-word summary, you pay for about 2,000 input tokens plus around 400 output tokens. Most vendors offer a token-counting tool or API so you can check before you send.