Pre-training

AI concepts
About 1 min read

The initial stage preceding fine-tuning, where a model is first trained on large-scale data to acquire foundational language and knowledge capabilities.

Also known as
Pre-trainingPre-training

Detailed explanation

Pre-training is the process of teaching a model language structure, factual knowledge, and basic reasoning capabilities using massive unsupervised datasets (such as internet text). In this stage, the model gains general-purpose understanding and generation capabilities not bound to specific tasks, which are later adjusted for specific use cases through fine-tuning. Because it is a high-cost stage requiring billions of parameters, vast datasets, and thousands of GPUs over weeks or months, most services start by utilizing pre-trained foundation models. The characteristics of an AI tool are heavily dictated by which foundation model is used as the pre-trained base.

Why it matters for tool selection

The default behavior and scope of knowledge of an AI tool stem from the underlying pre-trained model. Knowing which foundation model is used and its knowledge cutoff date helps you predict its ability to reflect up-to-date information and the quality of specific languages. While you will rarely pre-train a model yourself, choosing a foundation model is, in essence, choosing a pre-training result.

What to check when choosing tools

  • Which foundation model is it built on?
  • Is the model's knowledge cutoff date sufficiently up-to-date for your use case?
  • Was the required language (such as Korean) sufficiently represented in the pre-training data?
  • If up-to-date information is needed, is it supplemented with Retrieval-Augmented Generation (RAG)?

Real-world example

Even for chatbots with the same interface, the naturalness of language expression and awareness of recent events differ depending on the underlying foundation model. A model trained only up to last year might not know about products released this year, so it is safer to choose tools integrated with search if real-time information is critical.

Related terms

Fine-tuningLLMTransfer LearningParameter