Token

What is a token in AI?

Outside AI, a token can mean a coin-like object, a symbol of something, or a digital access key, and in crypto a token is a digital asset. In AI, a token is purely a unit of text (or other data) that a model processes. Anthropic's documentation describes tokens as the smallest individual units of a language model, which can correspond to words, subwords, characters, or even bytes.

How tokens work

The analogy: models do not read letters or whole words the way you do. They read Lego bricks of text. Common words are often one brick; rare or long words are built from several.

The precise version: a tokenizer splits text into tokens from a fixed vocabulary, and each token maps to a number the model works with. A word like "learning" might be one token or two, depending on the tokenizer. The ratio of characters to tokens varies by model and language. Anthropic's documentation, for example, puts it at roughly 3.5 English characters per token on its earlier models and notes that newer Claude models produce about 30 percent more tokens for the same text. For exact counts, providers offer token-counting tools.

A real example of tokens

You paste a 2,000-word report into a chatbot and ask for a summary. The report, your question, and any standing instructions are all converted into tokens and must fit within the model's context window. The summary it writes back is counted in tokens too. If you use the model through an API, both the tokens you sent and the tokens it wrote show up on the bill, usually at different rates.

Why tokens matter

• Cost: API usage is billed per token, so long prompts and long answers cost more.
• Limits: a model's context window, how much it can consider at once, is measured in tokens.
• Speed: more output tokens means a longer wait for the full answer, which affects latency.
• Instructions: because models count in tokens, Anthropic notes that asking for a set number of sentences or paragraphs works better than asking for an exact word count.

Related terms

• Context window: the token budget for a single request.
• Inference: running the model, measured and billed in tokens.
• Streaming: delivering a reply token by token as it is generated.
• Embedding: a different idea; an embedding represents meaning as numbers, while a token is a unit of text.

Common misconceptions about tokens

• One token equals one word. Often it is less than a word, and the ratio varies by model and language.
• Token counts are the same across models. Different tokenizers split the same text differently, so the same document can cost a different number of tokens on different models.
• AI tokens are crypto tokens. They are unrelated; an AI token is just a piece of text.

Token in practice

Tokens are the currency of every LLM. If you are planning a project, our tutorial on choosing an AI model covers how context size and cost shape the choice, and the context window entry explains the limit tokens fill up.