BERT

AI concepts
About 1 min read

A pre-trained language model based on a bidirectional Transformer encoder, released by Google in 2018, which excels at understanding tasks such as search and classification.

Also known as
Bidirectional Encoder Representations from Transformers

Detailed explanation

BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model released by Google in 2018. It uses only the encoder part of the Transformer and reads sentences in both left and right directions simultaneously to grasp the context of words. It is pre-trained using Masked Language Modeling, where certain words are masked and predicted, and Next Sentence Prediction, which learns the relationship between two sentences. It excels at tasks that classify and interpret text, such as search query understanding, sentiment analysis, and question answering, and has led to variant models like RoBERTa, ALBERT, and DistilBERT. Unlike the GPT family that generates new text, its defining characteristic is that it specializes in understanding and encoding.

Why It Matters in Tool Selection

For tasks where 'understanding' is key—such as classification, search, and semantic matching—BERT-family encoder models are often lighter and faster than generative LLMs. When choosing document classifiers, search ranking systems, or embedding generation tools, knowing which family of models is used helps you more accurately evaluate cost, response latency, and task suitability.

What to Check When Choosing a Tool

  • Is it an understanding task like classification and search, or a text generation task? (Criteria for choosing the model family)
  • Does it use a model trained on multilingual data, including Korean?
  • Does it expose embeddings or classification results so they can be used directly?
  • Have you compared whether the inference cost and speed are advantageous compared to generative LLMs?

Real-world Example

For tasks like automatically classifying customer inquiries into categories such as 'Refunds, Shipping, or Technical Support,' using a BERT-family classification model is often more than enough, rather than calling a heavy generative LLM. It is suitable for high-volume processing as it achieves the same level of accuracy with lower costs and faster response times.

Related terms

TransformerNLPPre-training