Chapter 3: Anatomy of an LLM

In this chapter, we opened up the “black box” of large language models to examine their inner workings. While tokenization and embeddings (from Chapter 2) give models their vocabulary, it is the transformer block that makes them powerful. By exploring attention, positional encodings, normalization strategies, and architectural refinements, we gained insight into how modern LLMs achieve their impressive abilities.

We began with multi-head self-attention, the mechanism that lets each token attend to every other token in a sequence. Instead of processing words in isolation, the model builds context dynamically, discovering relationships like “it” referring back to “the mat.” Multiple heads allow the model to capture different kinds of relationships simultaneously — local patterns, long-range dependencies, and structural cues. This ability to look at many things at once is the foundation of LLM reasoning.

Next, we looked at rotary embeddings (RoPE), a clever way of giving the transformer a sense of sequence order. Unlike fixed sinusoidal encodings, RoPE rotates query and key vectors in embedding space, making relative positions more natural to encode. This method scales gracefully to longer contexts, which is critical as modern LLMs process documents tens of thousands of tokens long.

Training such deep models would not be possible without normalization strategies. We explored LayerNorm, RMSNorm, and the pre-norm approach, which stabilize activations and ensure gradients flow correctly through dozens or even hundreds of stacked layers. These seemingly small choices make the difference between a model that converges and one that collapses.

From there, we discussed the trade-offs of depth versus width. Deeper models (more layers) capture hierarchical features, while wider models (larger hidden states and attention heads) can encode richer information per step. The right balance is guided by scaling laws and practical compute constraints.

We then turned to positional encoding tricks, comparing RoPE with ALiBi. While RoPE encodes relative positions via rotations, ALiBi biases attention scores directly based on token distance. Both allow models to generalize to longer contexts, though in slightly different ways.

Finally, we explored advanced architectural refinements:

• SwiGLU activations, which improve feedforward expressiveness.
• Grouped Query Attention (GQA), which reduces memory use by letting multiple query heads share key/value projections.
• Sparse attention patterns, which let models scale to very long contexts by focusing only on nearby or important tokens.

Together, these innovations reveal that the transformer block is not a static design but a constantly evolving system. Every choice — from activations to attention patterns — influences efficiency, stability, and capability.

The key lesson from this chapter is that modern LLMs are a product of careful engineering. Their strength comes not just from more data or more parameters, but from architectural innovations that make scaling possible. As we continue to the next chapter, we’ll shift from structure to process: how these models are trained from scratch, including data curation, deduplication, and the infrastructure needed to bring Titans of AI to life.

In this chapter, we opened up the “black box” of large language models to examine their inner workings. While tokenization and embeddings (from Chapter 2) give models their vocabulary, it is the transformer block that makes them powerful. By exploring attention, positional encodings, normalization strategies, and architectural refinements, we gained insight into how modern LLMs achieve their impressive abilities.

We began with multi-head self-attention, the mechanism that lets each token attend to every other token in a sequence. Instead of processing words in isolation, the model builds context dynamically, discovering relationships like “it” referring back to “the mat.” Multiple heads allow the model to capture different kinds of relationships simultaneously — local patterns, long-range dependencies, and structural cues. This ability to look at many things at once is the foundation of LLM reasoning.

Next, we looked at rotary embeddings (RoPE), a clever way of giving the transformer a sense of sequence order. Unlike fixed sinusoidal encodings, RoPE rotates query and key vectors in embedding space, making relative positions more natural to encode. This method scales gracefully to longer contexts, which is critical as modern LLMs process documents tens of thousands of tokens long.

Training such deep models would not be possible without normalization strategies. We explored LayerNorm, RMSNorm, and the pre-norm approach, which stabilize activations and ensure gradients flow correctly through dozens or even hundreds of stacked layers. These seemingly small choices make the difference between a model that converges and one that collapses.

From there, we discussed the trade-offs of depth versus width. Deeper models (more layers) capture hierarchical features, while wider models (larger hidden states and attention heads) can encode richer information per step. The right balance is guided by scaling laws and practical compute constraints.

We then turned to positional encoding tricks, comparing RoPE with ALiBi. While RoPE encodes relative positions via rotations, ALiBi biases attention scores directly based on token distance. Both allow models to generalize to longer contexts, though in slightly different ways.

Finally, we explored advanced architectural refinements:

• SwiGLU activations, which improve feedforward expressiveness.
• Grouped Query Attention (GQA), which reduces memory use by letting multiple query heads share key/value projections.
• Sparse attention patterns, which let models scale to very long contexts by focusing only on nearby or important tokens.

Together, these innovations reveal that the transformer block is not a static design but a constantly evolving system. Every choice — from activations to attention patterns — influences efficiency, stability, and capability.

The key lesson from this chapter is that modern LLMs are a product of careful engineering. Their strength comes not just from more data or more parameters, but from architectural innovations that make scaling possible. As we continue to the next chapter, we’ll shift from structure to process: how these models are trained from scratch, including data curation, deduplication, and the infrastructure needed to bring Titans of AI to life.

In this chapter, we opened up the “black box” of large language models to examine their inner workings. While tokenization and embeddings (from Chapter 2) give models their vocabulary, it is the transformer block that makes them powerful. By exploring attention, positional encodings, normalization strategies, and architectural refinements, we gained insight into how modern LLMs achieve their impressive abilities.

We began with multi-head self-attention, the mechanism that lets each token attend to every other token in a sequence. Instead of processing words in isolation, the model builds context dynamically, discovering relationships like “it” referring back to “the mat.” Multiple heads allow the model to capture different kinds of relationships simultaneously — local patterns, long-range dependencies, and structural cues. This ability to look at many things at once is the foundation of LLM reasoning.

Next, we looked at rotary embeddings (RoPE), a clever way of giving the transformer a sense of sequence order. Unlike fixed sinusoidal encodings, RoPE rotates query and key vectors in embedding space, making relative positions more natural to encode. This method scales gracefully to longer contexts, which is critical as modern LLMs process documents tens of thousands of tokens long.

Training such deep models would not be possible without normalization strategies. We explored LayerNorm, RMSNorm, and the pre-norm approach, which stabilize activations and ensure gradients flow correctly through dozens or even hundreds of stacked layers. These seemingly small choices make the difference between a model that converges and one that collapses.

From there, we discussed the trade-offs of depth versus width. Deeper models (more layers) capture hierarchical features, while wider models (larger hidden states and attention heads) can encode richer information per step. The right balance is guided by scaling laws and practical compute constraints.

We then turned to positional encoding tricks, comparing RoPE with ALiBi. While RoPE encodes relative positions via rotations, ALiBi biases attention scores directly based on token distance. Both allow models to generalize to longer contexts, though in slightly different ways.

Finally, we explored advanced architectural refinements:

• SwiGLU activations, which improve feedforward expressiveness.
• Grouped Query Attention (GQA), which reduces memory use by letting multiple query heads share key/value projections.
• Sparse attention patterns, which let models scale to very long contexts by focusing only on nearby or important tokens.

Together, these innovations reveal that the transformer block is not a static design but a constantly evolving system. Every choice — from activations to attention patterns — influences efficiency, stability, and capability.

The key lesson from this chapter is that modern LLMs are a product of careful engineering. Their strength comes not just from more data or more parameters, but from architectural innovations that make scaling possible. As we continue to the next chapter, we’ll shift from structure to process: how these models are trained from scratch, including data curation, deduplication, and the infrastructure needed to bring Titans of AI to life.

In this chapter, we opened up the “black box” of large language models to examine their inner workings. While tokenization and embeddings (from Chapter 2) give models their vocabulary, it is the transformer block that makes them powerful. By exploring attention, positional encodings, normalization strategies, and architectural refinements, we gained insight into how modern LLMs achieve their impressive abilities.

We began with multi-head self-attention, the mechanism that lets each token attend to every other token in a sequence. Instead of processing words in isolation, the model builds context dynamically, discovering relationships like “it” referring back to “the mat.” Multiple heads allow the model to capture different kinds of relationships simultaneously — local patterns, long-range dependencies, and structural cues. This ability to look at many things at once is the foundation of LLM reasoning.

Next, we looked at rotary embeddings (RoPE), a clever way of giving the transformer a sense of sequence order. Unlike fixed sinusoidal encodings, RoPE rotates query and key vectors in embedding space, making relative positions more natural to encode. This method scales gracefully to longer contexts, which is critical as modern LLMs process documents tens of thousands of tokens long.

Training such deep models would not be possible without normalization strategies. We explored LayerNorm, RMSNorm, and the pre-norm approach, which stabilize activations and ensure gradients flow correctly through dozens or even hundreds of stacked layers. These seemingly small choices make the difference between a model that converges and one that collapses.

From there, we discussed the trade-offs of depth versus width. Deeper models (more layers) capture hierarchical features, while wider models (larger hidden states and attention heads) can encode richer information per step. The right balance is guided by scaling laws and practical compute constraints.

We then turned to positional encoding tricks, comparing RoPE with ALiBi. While RoPE encodes relative positions via rotations, ALiBi biases attention scores directly based on token distance. Both allow models to generalize to longer contexts, though in slightly different ways.

Finally, we explored advanced architectural refinements:

• SwiGLU activations, which improve feedforward expressiveness.
• Grouped Query Attention (GQA), which reduces memory use by letting multiple query heads share key/value projections.
• Sparse attention patterns, which let models scale to very long contexts by focusing only on nearby or important tokens.

Together, these innovations reveal that the transformer block is not a static design but a constantly evolving system. Every choice — from activations to attention patterns — influences efficiency, stability, and capability.

The key lesson from this chapter is that modern LLMs are a product of careful engineering. Their strength comes not just from more data or more parameters, but from architectural innovations that make scaling possible. As we continue to the next chapter, we’ll shift from structure to process: how these models are trained from scratch, including data curation, deduplication, and the infrastructure needed to bring Titans of AI to life.