Alignment & the ChatGPT Moment
RLHF, InstructGPT, and the ChatGPT phenomenon.
InstructGPT and RLHF
OpenAI’s InstructGPT demonstrated that a 1.3B parameter model aligned with human preferences via reinforcement learning from human feedback could be preferred over the 175B GPT-3, proving that alignment technique matters as much as raw scale.
ChatGPT
Released on November 30, 2022, ChatGPT was a conversationally fine-tuned GPT-3.5 model that reached 100 million users in two months, transforming large language models from research curiosities into the fastest-growing consumer product in history.
Constitutional AI
Anthropic’s Constitutional AI replaced the need for extensive human labeling of harmful content by having the model critique and revise its own outputs according to a written set of principles, then training a preference model using AI-generated judgments (RLAIF).
Direct Preference Optimization (DPO)
Rafailov et al. showed that the RLHF objective could be mathematically reformulated as a simple classification loss on preference pairs, eliminating the need for a separate reward model and the instability of RL training while matching or exceeding PPO’s quality.
Instruction Tuning and FLAN
Google’s FLAN showed that fine-tuning language models on diverse NLP tasks phrased as natural-language instructions dramatically improves zero-shot generalization, and scaling to 1,800 tasks produced some of the largest gains in model capability per dollar ever observed.
Synthetic Data for Training
The practice of using language models to generate training data for other (or the same) models became a defining technique of the LLM era, enabling everything from Stanford Alpaca’s $600 chatbot to DeepSeek-R1’s reasoning breakthroughs.
GPT-4
OpenAI’s GPT-4, released in March 2023, was the first multimodal frontier model to accept both text and image inputs, and it achieved a massive leap in reasoning, coding, and factuality that set a new ceiling for AI capabilities — while revealing almost nothing about how it was built.
The AI Arms Race Begins
ChatGPT’s explosive success in late 2022 triggered a global technology arms race, with Google declaring “code red,” Microsoft investing 10B+ in OpenAI, and annual AI investment surpassing 100 billion as every major tech company scrambled to compete.