MATHEMATICS FOR DEEP LEARNING: From Matrix Calculus and Optimization to Statistical Learning and Neural Network Training

MATHEMATICS FOR DEEP LEARNING: From Matrix Calculus and Optimization to Statistical Learning and Neural Network Training

Author: Simon Calder

Description: Why do some learning rates cause training to diverge while others converge slowly? Why does one optimizer work for image classification but fail for language models? The answers lie in the mathematics that governs every gradient update, every loss landscape, and every parameter initialization. This book is your guide to that mathematics. MATHEMATICS FOR MACHINE LEARNING AND DEEP LEARNING is a rigorous, derivation-focused guide that bridges high-level framework usage with the underlying mathematical principles. It covers the entire training workflow: from matrix calculus and backpropagation through optimization, probabilistic inference, statistical learning, and the mathematics of core architectures like linear models, neural networks, and CNNs. Every formula is connected to a concrete model or training behavior. • Derive gradients for any layer using Jacobians and the chain rule. • Analyze loss curvature to understand optimizer behavior and convergence. • Connect probabilistic assumptions to loss functions like cross-entropy and MSE. Written for ML/DL practitioners, data scientists, and graduate students, this book assumes basic calculus and linear algebra but provides all necessary notation and foundation in the first part. It goes beyond heuristics, showing you how to derive, debug, and design models from first principles. With over 120,000 words of detailed derivations, intuition, and model connections, this book equips you to move beyond guesswork and make principled decisions about model architecture, training stability, and generalization.

AI summary: This book provides a rigorous, derivation-focused treatment of the mathematics underlying machine learning and deep learning training. It covers matrix calculus, backpropagation, optimization (including gradient descent and modern optimizers), statistical learning theory, and the mathematics of neural network architectures. Aimed at ML/DL practitioners, data scientists, and graduate students, it connects every formula to concrete model training behaviors and modern frameworks like PyTorch and TensorFlow.

Best for

ML/DL practitioners, data scientists, engineers, and graduate students seeking rigorous mathematical understanding of model training.

Reader persona

A professional or student who wants to move beyond high-level API usage and master the mathematical derivations behind model training, from gradients to optimization and generalization.

Search intent

To learn the mathematical foundations of machine learning and deep learning training, including matrix calculus, optimization, and statistical learning, in order to understand, debug, and design models from first principles.

Unique angle

This book provides rigorous step-by-step derivations of every formula, connecting each mathematical concept to concrete model training behaviors, and bridging theory with modern frameworks like PyTorch and TensorFlow.

Content type

educational textbook

Quick summary

• This book covers the core mathematics of machine learning and deep learning training, including matrix calculus, optimization, statistical learning, and neural network mathematics.
• It is written for ML/DL practitioners, data scientists, and graduate students who want to understand the formulas behind model training.
• The book derives gradients using Jacobians and the chain rule, analyzes loss curvature, and connects probabilistic assumptions to loss functions like cross-entropy and MSE.
• It goes beyond heuristics, showing how to derive, debug, and design models from first principles.

Key topics: Matrix Calculus, Backpropagation, Gradient Descent and Optimizers, Statistical Learning Theory, Loss Functions, Regularization, Probabilistic Inference, Generalization, Neural Network Mathematics, Weight Initialization

Entities: Jacobian, Hessian, Automatic Differentiation, Maximum Likelihood Estimation, Cross-Entropy Loss, Adam Optimizer, Batch Normalization, VC Dimension, Double Descent, Lagrange Multipliers, Convolutional Neural Networks, InfoNCE Loss

Needs addressed

• Understanding why certain learning rates cause convergence issues
• Deriving gradients for complex neural network layers
• Choosing the right loss function for different tasks
• Analyzing the convergence properties of optimization algorithms
• Connecting probabilistic assumptions to model objectives via MLE and MAP
• Debugging training instability and generalization failures in deep networks

Read if

• ML engineers and data scientists
• Graduate students in AI/ML/DL
• Software developers transitioning into ML
• Self-directed learners with basic ML knowledge
• Instructors teaching ML mathematics
• Researchers seeking deeper theoretical understanding

May not fit if

• Complete beginners with no calculus or linear algebra background
• Readers looking for a purely practical coding guide without derivations
• Those seeking a quick reference without detailed explanations
• Individuals who only want high-level conceptual overviews without mathematics

Table of contents

1. Introduction (introduction)
2. The Mathematical Foundations of Model Training (part)
3. Mathematics in Machine Learning and Deep Learning (chapter)
4. Machine Learning from a Mathematical Perspective (section)
5. The Mathematical Layers of a Learning System (section)
6. From Data to Models, Loss Functions, and Parameters (section)
7. The Mathematics of Inference and Training (section)
8. The Mathematical Roadmap of This Book (section)
9. The Mathematical Language of Machine Learning (chapter)
10. Scalars, Vectors, Matrices, and Tensors (section)
11. Shapes, Dimensions, and Data Axes (section)
12. Scalar-, Vector-, and Matrix-Valued Functions (section)
13. Vectorization and Elementwise Operations (section)
14. From Mathematical Notation to Code (section)
15. Core Transformations in Learning Models (chapter)
16. Linear Transformations (section)
17. Affine Transformations (section)
18. Nonlinear Transformations and Activation Functions (section)
19. Compositions of Functions in Deep Models (section)
20. Prediction as a Mathematical Mapping (section)
21. From Predictions to Optimization Problems (chapter)
22. Parameters, Hyperparameters, and Model State (section)
23. Loss Functions and Objective Functions (section)
24. Gradients and Parameter Updates (section)
25. Training, Validation, and Test Data (section)
26. The Complete Training Loop (section)
27. Matrix Calculus and Backpropagation (part)
28. Derivatives of Multivariable Functions (chapter)
29. Partial Derivatives and Parameter Sensitivity (section)
30. Gradients of Scalar-Valued Functions (section)
31. Directional Derivatives (section)
32. Taylor Approximations (section)
33. Gradients in Machine Learning Models (section)
34. Jacobians and Vector Derivatives (chapter)
35. Vector-Valued Functions and the Need for Jacobians (section)
36. The Jacobian Matrix (section)
37. Jacobian–Vector Products (section)
38. Vector–Jacobian Products (section)
39. Jacobians in Neural Networks (section)
40. Differentiation with Respect to Matrices (chapter)
41. Loss Functions Dependent on Weight Matrices (section)
42. Matrix Gradients (section)
43. Derivatives of Matrix Products (section)
44. Derivatives of Vector and Matrix Norms (section)
45. Matrix Calculus for a Neural Network Layer (section)
46. The Chain Rule, Computational Graphs, and Backpropagation (chapter)
47. The Chain Rule in Multilayer Models (section)
48. Computational Graphs (section)
49. Forward Propagation (section)
50. Backward Propagation (section)
51. Automatic Differentiation (section)
52. Curvature and Optimization (part)
53. Hessians and the Geometry of Loss Functions (chapter)
54. Second Derivatives (section)
55. The Hessian Matrix (section)
56. Eigenvalues of the Hessian (section)
57. Local Minima, Maxima, and Saddle Points (section)
58. Flat and Sharp Regions (section)
59. Foundations of Optimization (chapter)
60. Parameter Spaces and Objective Functions (section)
61. Local and Global Optimization (section)
62. Convex Sets and Convex Functions (section)
63. First- and Second-Order Optimality Conditions (section)
64. Nonconvex Optimization in Deep Learning (section)
65. Gradient Descent and Convergence (chapter)
66. The Gradient Descent Update Rule (section)
67. Learning Rates (section)
68. Smoothness and Lipschitz Conditions (section)
69. Analysis on Quadratic Functions (section)
70. Learning Rate Schedules (section)
71. Stochastic Gradient Descent and Modern Optimizers (chapter)
72. Full-Batch, Stochastic, and Mini-batch Gradients (section)
73. Batch Size and Gradient Variance (section)
74. Momentum and Nesterov Momentum (section)
75. AdaGrad, RMSProp, and Adaptive Learning Rates (section)
76. Adam and AdamW (section)
77. Second-Order and Constrained Optimization (chapter)
78. Newton’s Method (section)
79. BFGS and L-BFGS (section)
80. Hessian–Vector Products (section)

Frequently asked questions