What are tokens in AI? Explained with word-game tiles
Tokens are the small pieces of text an AI model reads and writes, and the unit used to measure limits and cost. Explained with a word game whose tiles carry chunks of words.
- Author
Umair Mateen- Published
- Updated
- Reading time
- 5 min
- Category
- AI
In one sentence: tokens are the small pieces of text, often parts of words, that an AI model reads and writes, and they are the unit AI services use to measure how much text fits in a request and how much it costs.
The everyday picture
Think of a word game played with tiles. Here is the twist: the tiles do not carry single letters. Each tile carries a common chunk of text. One tile might say "the". Another might say "ing", or " house" with a space in front.
Before the game starts, a tile cutter turns your sentence into a row of tiles. A common word fits on one tile. A rare or long word needs several tiles joined together. The player then reads the row and answers by laying down new tiles, one at a time.
The AI model is the player. The tiles are tokens. The tile cutter is called a tokenizer.
Your text is cut into tokens, like tiles carrying chunks of words. The model reads the row of tiles and answers by adding new tiles, one at a time.
How it works, step by step
- Your text (the sentence you bring to the table). Your question, instructions and any pasted documents all start as plain text.
- The tokenizer (the tile cutter) splits that text into tokens. OpenAI describes tokens as pieces of words, which can include spaces and parts of words. Its rule of thumb for English is about four characters per token, or roughly 100 tokens for every 75 words.
- The model reads the tokens (the player studies the row of tiles). The model never sees your text as you typed it. It only sees the row of tokens.
- The model writes in tokens (the player lays down new tiles). The answer is built from tokens too, one piece after another, until the reply is complete.
- Everything is counted (the scorekeeper tallies the tiles). Services count the tokens going in and coming out. Limits on how much a model can handle at once are measured in tokens, and so is usage on your bill.
Try it · Myth or fact
Tokens: myth or fact?
Decide for yourself first, then tap each statement to check it.
0 of 3 checked
Where the picture stops
- Every player has their own tile set. In a real game, every set is the same. In AI, each model family can cut text differently. Anthropic notes that its newer Claude models use a new tokenizer that turns the same text into about 30 percent more tokens than earlier models.
- Tiles are not syllables. The pieces are not chosen by grammar or sound. They come from how often chunks of text appear, so the splits can look odd to a person.
- It is not only words. Real tiles hold letters. AI models can also take in pictures and documents, and these are counted in tokens as well. Anthropic's token counter accepts images and PDFs.
Why it matters for your business
- Cost. AI services measure usage in tokens. Anthropic states that usage and billing reflect token counts. Long documents sent with every request add up quickly.
- Languages differ. OpenAI points out that text in some languages breaks into more tokens per character than English, which can make it more expensive to process.
- Switching models changes the count. The same prompt can be a different number of tokens on another model. Check costs again before you move a workflow to a new model.
- You can check before you send. Providers offer ways to count tokens in advance. Anthropic's token counting tool is free to use, though the count is an estimate that can differ slightly from the final figure.
An example
Imagine a 15-person travel agency that uses an AI assistant to draft replies to customers in English and Spanish. The manager notices the monthly bill is higher than expected.
She looks at the token counts. Every request carries the agency's full 40-page policy document, and the Spanish replies use more tokens than English ones of the same length. The team trims the policy text to the two pages the assistant really needs. They also budget a little more for Spanish work. The bill becomes easier to predict, because they now know what the tiles cost.
Frequently asked questions
How many words is a token?
In English, a token is about three quarters of a word on average, so 100 tokens is roughly 75 words. Other languages and code can differ a lot.
Is a token the same as a word?
Not always. A short common word is often one token, but a long or unusual word may be split into several. Spaces and punctuation can be part of tokens too.
Why do AI services charge by the token?
Tokens are what the model actually reads and writes, so they are a natural way to measure each request. Both the text you send and the reply it writes are counted.
Do images use tokens?
Yes. Many models turn images and documents into tokens too, so they count towards limits and usage. The exact count depends on the model and the file.
Related explainers
- What is a context window? Explained with a clerk's desk
- What is an LLM? Explained with a new hire who read the whole library
- What is a prompt? Explained with briefing a new contractor
The DevDrift helps teams build AI features with costs they can predict: see how.
Sources
Could you explain this to a colleague now?
Mark it and posts that mention it will show a tick. Saved in this browser only.