Chapter 4: Training LLMs from Scratch

In this chapter, we shifted from exploring the anatomy of large language models to understanding how they are brought to life through training. While architecture defines what a model could do, the training process determines what it actually learns. We saw that training an LLM is as much about engineering pipelines and infrastructure choices as it is about clever algorithms.

We began with the foundation: data collection, cleaning, deduplication, and filtering. The raw material of an LLM often comes from diverse sources — books, code, web scrapes, and specialized corpora. But raw text is messy. To avoid “garbage in, garbage out,” we need systematic cleaning (removing HTML, standardizing text), deduplication (eliminating repeats with MinHash or SimHash), and filtering (excluding spam, low-quality, or harmful content). These steps ensure the dataset is broad yet usable, setting the stage for effective learning.

Next, we examined curriculum learning, mixture datasets, and synthetic data. Just as humans learn in structured progressions, models can benefit from being exposed first to clean and simple data, then gradually to noisier or domain-specific material. Mixture datasets combine different sources — Wikipedia, books, code — with carefully tuned weights to balance capabilities. And when data is scarce, synthetic data generated by other models or back-translation techniques can fill the gap, providing rare examples or domain-specific training pairs. Together, these strategies make the dataset not only bigger, but smarter.

From there, we turned to infrastructure — the hardware and distributed systems that make large-scale training possible. We saw how GPUs, TPUs, and newer accelerators differ in cost, ecosystem, and performance. We explored data parallelism, model parallelism, and pipeline parallelism, as well as hybrid approaches used in practice. Practical techniques like PyTorch Distributed Data Parallel (DDP) demonstrate how training can be spread across devices. These strategies are the reason models with hundreds of billions of parameters can even exist.

Finally, we addressed the pressing issues of cost optimization and sustainability. Training is expensive in both money and energy. Methods like mixed-precision training, gradient checkpointing, efficient optimizers (ZeRO, Shampoo), and spot-instance scheduling reduce costs dramatically. Beyond economics, sustainability now plays a central role: choosing renewable-powered data centers, measuring carbon impact, and reporting energy use are becoming standard practice. Responsible AI development means pushing performance while also minimizing environmental costs.

The key lesson from this chapter is that training an LLM is a whole-system problem. It is not just about model architecture, but about aligning data quality, training strategies, distributed infrastructure, and sustainability into a coherent pipeline. The smartest design in the world will fail if trained on noisy data or inefficient hardware. Conversely, thoughtful data engineering and infrastructure choices can turn a modest design into a robust, useful model.

In this chapter, we shifted from exploring the anatomy of large language models to understanding how they are brought to life through training. While architecture defines what a model could do, the training process determines what it actually learns. We saw that training an LLM is as much about engineering pipelines and infrastructure choices as it is about clever algorithms.

We began with the foundation: data collection, cleaning, deduplication, and filtering. The raw material of an LLM often comes from diverse sources — books, code, web scrapes, and specialized corpora. But raw text is messy. To avoid “garbage in, garbage out,” we need systematic cleaning (removing HTML, standardizing text), deduplication (eliminating repeats with MinHash or SimHash), and filtering (excluding spam, low-quality, or harmful content). These steps ensure the dataset is broad yet usable, setting the stage for effective learning.

Next, we examined curriculum learning, mixture datasets, and synthetic data. Just as humans learn in structured progressions, models can benefit from being exposed first to clean and simple data, then gradually to noisier or domain-specific material. Mixture datasets combine different sources — Wikipedia, books, code — with carefully tuned weights to balance capabilities. And when data is scarce, synthetic data generated by other models or back-translation techniques can fill the gap, providing rare examples or domain-specific training pairs. Together, these strategies make the dataset not only bigger, but smarter.

From there, we turned to infrastructure — the hardware and distributed systems that make large-scale training possible. We saw how GPUs, TPUs, and newer accelerators differ in cost, ecosystem, and performance. We explored data parallelism, model parallelism, and pipeline parallelism, as well as hybrid approaches used in practice. Practical techniques like PyTorch Distributed Data Parallel (DDP) demonstrate how training can be spread across devices. These strategies are the reason models with hundreds of billions of parameters can even exist.

Finally, we addressed the pressing issues of cost optimization and sustainability. Training is expensive in both money and energy. Methods like mixed-precision training, gradient checkpointing, efficient optimizers (ZeRO, Shampoo), and spot-instance scheduling reduce costs dramatically. Beyond economics, sustainability now plays a central role: choosing renewable-powered data centers, measuring carbon impact, and reporting energy use are becoming standard practice. Responsible AI development means pushing performance while also minimizing environmental costs.

The key lesson from this chapter is that training an LLM is a whole-system problem. It is not just about model architecture, but about aligning data quality, training strategies, distributed infrastructure, and sustainability into a coherent pipeline. The smartest design in the world will fail if trained on noisy data or inefficient hardware. Conversely, thoughtful data engineering and infrastructure choices can turn a modest design into a robust, useful model.

In this chapter, we shifted from exploring the anatomy of large language models to understanding how they are brought to life through training. While architecture defines what a model could do, the training process determines what it actually learns. We saw that training an LLM is as much about engineering pipelines and infrastructure choices as it is about clever algorithms.

We began with the foundation: data collection, cleaning, deduplication, and filtering. The raw material of an LLM often comes from diverse sources — books, code, web scrapes, and specialized corpora. But raw text is messy. To avoid “garbage in, garbage out,” we need systematic cleaning (removing HTML, standardizing text), deduplication (eliminating repeats with MinHash or SimHash), and filtering (excluding spam, low-quality, or harmful content). These steps ensure the dataset is broad yet usable, setting the stage for effective learning.

Next, we examined curriculum learning, mixture datasets, and synthetic data. Just as humans learn in structured progressions, models can benefit from being exposed first to clean and simple data, then gradually to noisier or domain-specific material. Mixture datasets combine different sources — Wikipedia, books, code — with carefully tuned weights to balance capabilities. And when data is scarce, synthetic data generated by other models or back-translation techniques can fill the gap, providing rare examples or domain-specific training pairs. Together, these strategies make the dataset not only bigger, but smarter.

From there, we turned to infrastructure — the hardware and distributed systems that make large-scale training possible. We saw how GPUs, TPUs, and newer accelerators differ in cost, ecosystem, and performance. We explored data parallelism, model parallelism, and pipeline parallelism, as well as hybrid approaches used in practice. Practical techniques like PyTorch Distributed Data Parallel (DDP) demonstrate how training can be spread across devices. These strategies are the reason models with hundreds of billions of parameters can even exist.

Finally, we addressed the pressing issues of cost optimization and sustainability. Training is expensive in both money and energy. Methods like mixed-precision training, gradient checkpointing, efficient optimizers (ZeRO, Shampoo), and spot-instance scheduling reduce costs dramatically. Beyond economics, sustainability now plays a central role: choosing renewable-powered data centers, measuring carbon impact, and reporting energy use are becoming standard practice. Responsible AI development means pushing performance while also minimizing environmental costs.

The key lesson from this chapter is that training an LLM is a whole-system problem. It is not just about model architecture, but about aligning data quality, training strategies, distributed infrastructure, and sustainability into a coherent pipeline. The smartest design in the world will fail if trained on noisy data or inefficient hardware. Conversely, thoughtful data engineering and infrastructure choices can turn a modest design into a robust, useful model.

In this chapter, we shifted from exploring the anatomy of large language models to understanding how they are brought to life through training. While architecture defines what a model could do, the training process determines what it actually learns. We saw that training an LLM is as much about engineering pipelines and infrastructure choices as it is about clever algorithms.

We began with the foundation: data collection, cleaning, deduplication, and filtering. The raw material of an LLM often comes from diverse sources — books, code, web scrapes, and specialized corpora. But raw text is messy. To avoid “garbage in, garbage out,” we need systematic cleaning (removing HTML, standardizing text), deduplication (eliminating repeats with MinHash or SimHash), and filtering (excluding spam, low-quality, or harmful content). These steps ensure the dataset is broad yet usable, setting the stage for effective learning.

Next, we examined curriculum learning, mixture datasets, and synthetic data. Just as humans learn in structured progressions, models can benefit from being exposed first to clean and simple data, then gradually to noisier or domain-specific material. Mixture datasets combine different sources — Wikipedia, books, code — with carefully tuned weights to balance capabilities. And when data is scarce, synthetic data generated by other models or back-translation techniques can fill the gap, providing rare examples or domain-specific training pairs. Together, these strategies make the dataset not only bigger, but smarter.

From there, we turned to infrastructure — the hardware and distributed systems that make large-scale training possible. We saw how GPUs, TPUs, and newer accelerators differ in cost, ecosystem, and performance. We explored data parallelism, model parallelism, and pipeline parallelism, as well as hybrid approaches used in practice. Practical techniques like PyTorch Distributed Data Parallel (DDP) demonstrate how training can be spread across devices. These strategies are the reason models with hundreds of billions of parameters can even exist.

Finally, we addressed the pressing issues of cost optimization and sustainability. Training is expensive in both money and energy. Methods like mixed-precision training, gradient checkpointing, efficient optimizers (ZeRO, Shampoo), and spot-instance scheduling reduce costs dramatically. Beyond economics, sustainability now plays a central role: choosing renewable-powered data centers, measuring carbon impact, and reporting energy use are becoming standard practice. Responsible AI development means pushing performance while also minimizing environmental costs.

The key lesson from this chapter is that training an LLM is a whole-system problem. It is not just about model architecture, but about aligning data quality, training strategies, distributed infrastructure, and sustainability into a coherent pipeline. The smartest design in the world will fail if trained on noisy data or inefficient hardware. Conversely, thoughtful data engineering and infrastructure choices can turn a modest design into a robust, useful model.