THE MATHEMATICS OF TRANSFORMERS AND LLMs: Attention, Token Probabilities, Scaling Laws, and Language Model Inference

THE MATHEMATICS OF TRANSFORMERS AND LLMs: Attention, Token Probabilities, Scaling Laws, and Language Model Inference

Author: Simon Calder

Description: Ask a large language model to explain the theory of relativity, and it will generate a paragraph that reads like it understands physics. Yet every word it produces is the result of multiplying matrices and applying softmax—operations that are purely mechanical. How does a stack of linear algebra and probability calculations give rise to behavior that appears intelligent? This question lies at the heart of modern AI, and The Mathematics of Transformers and LLMs provides a complete answer by taking you through the precise mathematical pipeline that turns raw text into token probabilities. Each chapter builds on the last, until you can trace every tensor shape and probability distribution from input to output. The Mathematics of Transformers and LLMs: Attention, Token Probabilities, Scaling Laws, and Language Model Inference is a rigorous yet intuition-first mathematical treatment of the architecture that powers most modern language models. Author Simon Calder begins with the observation that a language model is fundamentally a probability distribution over token sequences, and systematically builds the machinery to compute that distribution. The book covers tokenization (BPE, WordPiece, Unigram), token embeddings, positional encodings (sinusoidal, ALiBi, RoPE), scaled dot-product attention, multi-head attention, feed-forward networks, gating mechanisms (GELU, SwiGLU), residual connections, layer normalization, and the complete transformer block. It extends to the language modeling head, training objectives (maximum likelihood, cross-entropy, perplexity), optimization (AdamW, warmup, cosine decay, gradient clipping), scaling laws (Kaplan 2020, Chinchilla 2022), in-context learning theories, autoregressive inference, KV caching, decoding strategies (greedy, beam, top-k, top-p), attention complexity, and efficient architectures (MQA, GQA, MoE). With 28 chapters organized into 9 parts, the book mirrors the exact data flow of an LLM, from raw text to generated token. It is designed for readers who want to understand not just what a model does, but why it works mathematically. The book stands alone with a dedicated mathematics review chapter, and also fits as the third volume in a four-volume series on the mathematics of deep learning. The book focuses on three core mathematical engines that underpin every large language model: • Scaled dot-product attention and multi-head attention, derived from first principles with complete tensor shape tracking. A step-by-step numerical example with 2x2 matrices illustrates every calculation. • Scaling laws that quantify how model size, data, and compute interact, including a derivation of the Kaplan and Chinchilla relationships and their implications for compute-optimal training. The book shows how to predict loss and allocate resources. • Autoregressive inference, including prefill and decode phases, KV caching, and a comparison of decoding strategies (greedy, beam search, top-k, top-p). Memory costs and latency trade-offs are analyzed quantitatively. Additionally, the book covers efficient architectures like Multi-Query Attention and Mixture of Experts, and discusses fundamental limitations such as hallucination and effective context. Every mathematical section follows a consistent framework: conceptual intuition, formal equation, component-by-component interpretation, tensor shape analysis, and computational or memory cost. Numerical examples with small matrices make abstract operations concrete. The book is heavily illustrated with tensor shape grids and data flow diagrams—over 150 images in total—to break up dense math and reinforce understanding. All notation is consistent throughout, with a global notation table to prevent symbol drift. The book also includes explicit parameter counts and FLOP calculations for every transformer component, enabling you to reason about model efficiency. The representation space is built from the ground up: tokenization converts text to IDs, embeddings map IDs to continuous vectors, and positional encodings add order information. The attention mechanism is derived step by step before being extended to multiple heads and different masking patterns. The transformer block is assembled with attention, feed-forward networks, residual connections, and normalization, and a complete chapter calculates its parameter count and computational cost. The language modeling part derives the probability chain rule, logits, softmax, cross-entropy, and perplexity. Optimization chapters cover AdamW, learning rate schedules, and gradient clipping. Scaling laws are derived from empirical power-law fits and interpreted for training decisions. Inference is broken into prefill and decode, with KV caching and memory analysis. Decoding strategies are compared for diversity and coherence. Efficient architectures like MQA, GQA, and MoE are explained mathematically. The final chapter integrates everything into a single end-to-end mathematical description and discusses fundamental limitations, including hallucination and effective context length. This book is intended for AI practitioners, machine learning engineers, and graduate students who have a solid foundation in linear algebra and calculus and want to truly understand how LLMs work from the inside out. It avoids MLOps, RLHF, and framework-specific code, focusing instead on the durable mathematics that will remain relevant as architectures evolve. A dedicated chapter reviews the specific mathematical tools needed, making the book self-contained. The book is ideal for self-study or as a reference for researchers and engineers working with Transformer-based models. It is also suitable for advanced undergraduates seeking a deep understanding of the mathematical foundations of modern AI. By the end, you will be able to trace the complete mathematical data flow of an LLM from raw text to generated token, understand why attention is scaled, how KV caching reduces latency, and what scaling laws imply for training decisions. This is the foundation you need to design, analyze, or deploy large language models with confidence. Whether you are a researcher, engineer, or student, this book will give you the mathematical lens to see through the black box and understand the true nature of language models.

AI summary: This book provides a rigorous mathematical treatment of Transformers and Large Language Models, covering the entire pipeline from tokenization to autoregressive inference. It derives scaled dot-product attention, multi-head attention, feed-forward networks, scaling laws, and training objectives with tensor shape tracking and numerical examples. Designed for readers with basic linear algebra and calculus, it offers an intuition-first approach to understanding how LLMs compute token probabilities.

Best for

AI practitioners, machine learning engineers, and graduate students with foundational linear algebra and calculus who want a rigorous mathematical understanding of Transformers and LLMs.

Reader persona

A machine learning engineer or graduate student with a solid foundation in linear algebra and calculus, seeking a rigorous mathematical understanding of how Transformers and LLMs work from text input to token probability output.

Search intent

To understand the mathematical foundations and derivations behind Transformer architectures and large language models, including attention, scaling laws, and inference.

Unique angle

This book offers a rigorous, intuition-first mathematical derivation of every component in the Transformer pipeline, from tokenization to inference, with consistent tensor shape tracking and numerical examples, distinguishing it from high-level overviews or pure implementation guides.

Content type

knowledge book

Quick summary

• This book covers the complete mathematics behind Transformers and LLMs, from tokenization and embeddings to attention mechanisms and scaling laws.
• It provides intuition-first derivations of scaled dot-product attention, multi-head attention, and feed-forward networks with tensor shape tracking.
• The book explains training objectives like cross-entropy and perplexity, optimization with AdamW, and scaling laws for compute-optimal training.
• Autoregressive inference, KV caching, and decoding strategies are analyzed with computational and memory costs.
• Target readers are ML engineers, AI practitioners, and graduate students with foundational linear algebra and calculus.

Key topics: Transformer architecture, Attention mechanisms, Scaled dot-product attention, Multi-head attention, Tokenization and vocabulary, Positional encoding, Feed-forward networks, Language modeling, Scaling laws, Autoregressive inference, KV caching

Entities: Transformer, Large Language Model, attention mechanism, scaling law, token probability, autoregressive model, positional encoding, feed-forward network, layer normalization, KV cache, BPE, WordPiece

Needs addressed

• Understanding the mathematical structure of Transformers
• Deriving attention and multi-head attention from scratch
• Applying scaling laws to optimize model size and training data
• Implementing efficient autoregressive inference with KV caching
• Choosing decoding strategies for text generation

Read if

• Machine learning engineers
• Deep learning engineers
• AI researchers
• Graduate students in computer science or AI
• Software engineers working with LLMs
• Instructors teaching Transformer-based models

May not fit if

• Beginners without linear algebra or calculus background
• Readers looking for hands-on coding tutorials
• Practitioners only interested in high-level API usage

Table of contents

1. Introduction (introduction)
2. An Overview of Transformers and Large Language Models (part)
3. Language Models from a Mathematical Perspective (chapter)
4. Language as a Sequence of Tokens (section)
5. What Does a Language Model Actually Learn? (section)
6. From n-grams to Neural Language Models (section)
7. Training and Inference (section)
8. Why Transformers Became the Dominant Architecture (section)
9. The Overall Structure of a Transformer (chapter)
10. Transformer Inputs and Outputs (section)
11. Components of a Transformer Block (section)
12. Data Flow Through a Transformer Block (section)
13. Stacking Transformer Blocks (section)
14. Encoders, Decoders, and Encoder–Decoder Models (section)
15. The Overall Structure of a Large Language Model (chapter)
16. Tokenizer and Vocabulary (section)
17. Embeddings and Positional Information (section)
18. The Transformer Backbone (section)
19. The Language Modeling Head (section)
20. Complete Training and Inference Flows (section)
21. Mathematical Tools and Notation (chapter)
22. Vectors, Matrices, and Tensors (section)
23. Shapes, Dimensions, and Axes (section)
24. Dot Products and Matrix Multiplication (section)
25. Softmax, Logarithms, and Cross-Entropy (section)
26. Computational and Memory Complexity (section)
27. From Text to Representation Spaces (part)
28. Tokenization and Vocabulary Spaces (chapter)
29. Why Language Models Need Tokenization (section)
30. Character, Word, and Subword Tokenization (section)
31. Byte Pair Encoding (section)
32. WordPiece and the Unigram Language Model (section)
33. Vocabulary Size and Model Cost (section)
34. Token Embeddings and Semantic Spaces (chapter)
35. From Token IDs to Vectors (section)
36. The Embedding Matrix as Learned Parameters (section)
37. Distance, Dot Products, and Similarity (section)
38. Static and Contextual Embeddings (section)
39. Weight Tying (section)
40. The Mathematics of Positional Representation (chapter)
41. Why Self-Attention Does Not Understand Order Automatically (section)
42. Sinusoidal Positional Encoding (section)
43. Learned Positional Embeddings (section)
44. Relative Position Bias and ALiBi (section)
45. Rotary Position Embedding (section)
46. The Core Mathematics of Attention (part)
47. Queries, Keys, Values, and Attention (chapter)
48. Contextual Information Selection (section)
49. Queries, Keys, and Values (section)
50. Linear Projections of Queries, Keys, and Values (section)
51. Dot Products as Relevance Scores (section)
52. Weighted Sums of Values (section)
53. Scaled Dot-Product Attention (chapter)
54. The Complete Attention Equation (section)
55. Why Divide by the Square Root of the Key Dimension? (section)
56. The Attention Matrix (section)
57. Attention in Tensor Form (section)
58. A Step-by-Step Numerical Example (section)
59. Masking and Causal Attention (chapter)
60. Padding Masks (section)
61. Causal Masks (section)
62. Applying Masks Before Softmax (section)
63. Bidirectional and Autoregressive Attention (section)
64. Causal Masks During Training and Inference (section)
65. Multi-Head Attention (chapter)
66. Why One Attention Head Is Not Enough (section)
67. Per-Head Projections (section)
68. Parallel Head Computation (section)
69. Concatenation and Output Projection (section)
70. Head Count, Head Dimension, and Model Dimension (section)
71. Alternative Views of Attention (chapter)
72. Attention as Retrieval (section)
73. Attention and Associative Memory (section)
74. Attention and Kernel Methods (section)
75. Attention as Conditional Smoothing (section)
76. Attention Weights and Interpretability (section)
77. The Mathematical Structure of Transformers (part)
78. Feed-Forward Networks and Gating (chapter)
79. Position-Wise Transformations (section)
80. Expanding and Contracting the Hidden Dimension (section)

Frequently asked questions