Token

Data
About 1 min read

The basic unit through which an LLM recognizes and generates text, created by breaking sentences into semantic pieces smaller than words or characters. It is a key metric that determines the computational cost of the AI model, response speed, and the amount of information it can remember at once (context window).

Also known as
TokenTokenization

Detailed explanation

A token is the 'smallest particle' by which AI converts language into data. A sentence passes through a tokenizer to be converted into numerical tokens, allowing the model to grasp the context. In the past, Korean token efficiency was lower compared to English, incurring 2 to 3 times the cost for the same content. However, modern models such as GPT-4o's 'o200k_base' and Claude 3 have improved tokenizer efficiency, significantly optimizing the cost and speed of processing Korean. Beyond a simple unit of text, tokens serve as the standard metric for measuring API billing and a model's 'memory capacity.'

Why it matters in tool selection

Tokens directly impact the operational economics and performance of AI services. Even for the same document, the billed amount varies based on the model's tokenizer efficiency, which also determines how much information can fit within the context window. Especially in Korean language environments, checking the Korean token compression rate for each model is key to cost optimization.

What to look for when choosing a tool

  • Does the model improve Korean efficiency by using a modern tokenizer (e.g., GPT-4o's o200k)?
  • Is the price difference between input and output tokens within your budget?
  • Is the token limit (context window) sufficient for your tasks when processing long contexts?

Examples of Korean token processing by model

While the English word 'Apple' is usually 1 token, the Korean word for apple ('사과') varies greatly depending on the model. In the older GPT-4 (cl100k_base), a single Korean character consumed about 2 to 3 tokens, but the latest GPT-4o (o200k_base) has improved compression efficiency by more than 1.5 times, processing the same Korean sentence with fewer tokens.

Comparison of text measurement units

Character

The number of individual characters, including spaces. The amount of text as perceived by humans.

Word

A unit based on spacing. The standard primarily used in English-speaking regions.

Token

The unit of internal AI processing. The actual standard for billing and model memory usage.

Related terms

Context Window