Token

Understand it deeply

What is a token, really?

A small unit a model uses to read and generate text; it is not always one character or one word.

People read a sentence as a whole. A language model first splits it into tokens, turns them into numbers and computes with those numbers. It also generates its response one token at a time.

Think of building blocks for machine-readable text

A token is a text building block prepared for a machine. A common English word may be one block, while a long or uncommon word may be several. Chinese is not automatically one character per token either. Each model's tokenizer makes those splits, so the same sentence can use different token counts in different models.

How text becomes an answer

1. Split the text

   The prompt, documents and chat history are split into tokens. The model receives a sequence of units, not a complete sentence in the human sense.

2. Represent and calculate

   Each token becomes a numeric representation. The model uses the surrounding context to calculate what is likely to come next.

3. Generate one by one

   The chosen next token is appended to the existing text, then the model predicts again until an answer is formed.

What does it directly affect?

Tokens are not just vocabulary. They determine how much a request can contain, as well as much of its cost and latency.

• 01 — Context window

  Prompts, files, chat history and generated output all use tokens. Content beyond the window cannot be considered together in one request.

• 02 — Cost

  Many APIs charge for input and output tokens. Long documents, long chats and long output increase usage.

• 03 — Generation speed

  Models generate tokens sequentially, so longer output normally takes longer.

Where do tokens show up in a product?

When an AI summarises a long report, the report, your instruction, earlier messages and the summary itself must all fit in the context window. If they do not, split the document, summarise first, or retrieve only the relevant passages with RAG.

Common misunderstandings

Is one Chinese character always one token?

No. Tokens are chosen by a model's tokenizer. One token may be a character, several characters, part of a word, or include spaces and punctuation.

Are fewer tokens always better?

No. Removing irrelevant material saves cost and window space, but over-compressing can remove essential context. Keep what the task actually needs.

Tokens are the building blocks a model uses for language: they shape how much it can see, how quickly it responds and roughly what a call costs.