Key Takeaway

In short, the difference between pretraining, fine-tuning, and RLHF comes down to this general sequence: an LLM is typically built through ① pretraining (next-token prediction → Base model), ② fine-tuning/SFT (instruction–response examples → instruction-following model), and ③ RLHF (incorporating human preferences → conversational model). This post explores that overarching flow.

This post is part of my AI study notes series for the SK hynix AI Hackathon preliminary round (Oct 17–18). While my Day1–4 topics, such as Day1’s Self-Attention and Day2’s Transformer, focused on inference and model usage, today we shift to the “training” axis—how models are actually built.

Breaking down the entire flow, we have three main stages:

  1. ① Pretraining: Repeatedly predicting the next token across massive text datasets to create a Base model.
  2. ② Fine-tuning (SFT): Conducting additional training on instruction–response data to teach the model how to follow specific commands.
  3. ③ RLHF: Creating a reward model based on human preference rankings, then using reinforcement learning to develop a fully conversational (Chat) model. This outlines the general approach described in the InstructGPT paper; it does not imply that every commercial model follows this exact training procedure.

The general three stages of building an LLM: large text goes through ① pretraining (next-token prediction) to a Base model, the Base model plus instruction–response data goes through ② SFT to an instruction model, and the instruction model plus human preference comparisons goes through ③ RLHF (reward model + reinforcement learning) to a Chat model

What Is Pretraining? What Does Next-Token Prediction Teach a Model?

One-line answer: It’s the phase where a model repeatedly guesses the next token across massive text datasets, building a Base model that has absorbed language and knowledge patterns.

  1. Pretraining: The model repeats “guess the next token” over a large amount of text, picking up language and knowledge patterns along the way. Because the text itself provides the correct answers, human labeling isn’t required (this is known as self-supervised learning).
  2. Base model traits: While it excels at smoothly continuing a sentence, it isn’t designed to answer questions or follow instructions like a chatbot. For example, if you feed it a question, it might simply generate more similar questions instead of providing an answer.
  3. Training vs. inference: Training involves updating the model’s weights, whereas inference generates tokens using those fixed weights. We covered this distinction in detail in the Day3 post.

Fine-Tuning (SFT) vs. Prompting: Does It Change the Model or the Input?

One-line answer: Fine-tuning fundamentally alters the model by updating its weights with instruction–response data, whereas prompting simply optimizes the input while keeping the model’s weights exactly the same.

  1. SFT (supervised fine-tuning): This process further trains the model’s weights using human-crafted instruction–response pairs, teaching the AI to properly follow commands.
  2. Prompting: This approach leaves the model’s internal weights untouched and focuses purely on crafting effective inputs. Prompt design techniques are covered in the Day4 prompt engineering post.
  3. LoRA: This technique freezes the pretrained weights and only trains a small set of low-rank matrices (adapters) added to each layer. This cuts down the number of trainable parameters and the associated memory burden. (Hu et al., 2021)
  4. The key differences among these approaches are summarized in the table below.
ApproachWhat ChangesWhat’s NeededWhen
PromptingInput (weights unchanged)Well-designed instructions/examplesWhen trying things quickly without changing the model
SFT (Full Fine-Tuning)All model weightsInstruction–response example data, training resourcesWhen teaching the model a specific format/domain behavior
LoRASmall adapter weights (original weights frozen)Example data, relatively fewer training resourcesWhen fine-tuning under limited resources

What Is RLHF? How Does It Reflect Human Preferences in a Model?

One-line answer: First, a reward model is trained on human-ranked answers, and then reinforcement learning steers the language model to generate outputs that align with those human preferences.

  1. The flow (based on the InstructGPT approach): The model generates several different answers to a single prompt, and humans rank them from best to worst. A reward model learns from this preference data, and reinforcement learning (specifically PPO) is then used to update the LLM’s policy to maximize that reward. (Ouyang et al., 2022)

The RLHF loop (general flow as in InstructGPT): ① generate answers A and B → ② a human ranks A > B → ③ train a reward model on the comparison data → ④ update the policy (LLM) with reinforcement learning (PPO) using the reward score → generate again and repeat

  1. The goal: This process aligns the model with what people prefer, encouraging answers that are helpful, harmless, and honest—a concept known as alignment.
  2. An alternative: Methods like DPO skip the complex reward model and reinforcement learning phases entirely, opting to train the model directly on human preference data.
  3. A three-stage summary: Pretraining establishes language and foundational knowledge, SFT introduces instruction-following capabilities, and RLHF fine-tunes the model to align with human preferences.

[Exam one-liner] It’s worth remembering this general sequence: an LLM progresses from pretraining (Base model) → fine-tuning/SFT (instruction-following model) → RLHF (conversational model aligned with human preferences).

Shall We Test What You’ve Learned With a Quiz?

Q1. During pretraining, do humans need to manually provide the correct answers for every example?

A1. No. Since the raw text itself acts as the correct answer for the “guess the next token” task, the process is self-supervised and doesn’t require manual human labeling.

Q2. What is the fundamental difference between fine-tuning and prompting?

A2. Fine-tuning actively modifies the model by updating its internal weights through additional training, while prompting simply changes the input query without touching the model’s weights.

Q3. How does LoRA save on training resources?

A3. It freezes the model’s original pretrained weights and only trains small, newly added low-rank adapters in each layer.

Q4. What specific role do humans play in the RLHF process?

A4. Humans evaluate and rank the different answers generated by the model, providing the crucial comparison data that the reward model needs to understand human preferences.