The course
A graduate probability course built out of tools and examples rather than a slow development of the foundations.
Twenty-two sessions, in two modules. Each one is organized around a technique worth owning — Lindeberg swapping, Stein’s method, Wick’s theorem, the resolvent, Gaussian integration by parts — and around an example that shows what the technique is for. A substantial part of that second half is modern machine learning: wide networks at initialization, kernel regression, the spectrum of a Gaussian random matrix, the maxima of Gaussian processes, stochastic gradient descent, and diffusion models.
No measure theory. Nothing is assumed beyond linear algebra, multivariable calculus and an undergraduate probability course. Where a standard fact from analysis is needed — Fourier inversion, the spectral theorem — it is quoted honestly and marked as such in the notes.
How it works
Homework, assigned every sessionnot graded
Weekly quiz, 15 min, in class20%
Midterm, in class30%
Final50%
The lowest quiz is dropped. Both examinations are closed book.
Grades
A90–100
B80–89
C70–79
D60–69
F0–59
Module I · Gaussians
12 sessions · Sep 2 – Oct 14
Typed notes and solutions are linked session by session below. The handwritten notes from lecture, in one running file updated after each class: hand-notes.pdf.
1Sep 2Gaussians: definition and linear structure
- Definition by the Fourier transform, and why not the density
- Linear images, existence, rotational invariance
- Uncorrelated and jointly Gaussian implies independent
- Maximum entropy at fixed covariance
notes·solutions
2Sep 9Gaussians: conditioning as projection
- Conditional expectation as orthogonal projection
- The Schur complement, derived
- Tower property and total variance, read off the picture
- Bayesian posteriors and ridge regression; Tweedie's formula
notes·solutions
3Sep 14CLT: Lindeberg's method
- Maximum entropy, recalled; why a Gaussian at all
- What convergence in distribution means, and why smooth test functions
- Lindeberg's swapping argument, proved
- Lindeberg's and Feller's conditions; the CLT for triangular arrays
- Records: why random search improves only log n times, and why the Gaussian/Poisson dial is s_n
notes·solutions
4Sep 16Stein's method
- The Gaussian characterized by an identity, not a limit
- Stein's equation; the CLT with a rate
- Chen–Stein for Poisson; dependence
notes·solutions
5Sep 21Wick's theorem, and a Gaussian random matrix
- Every moment of a Gaussian vector, as a sum over pair partitions
- Proved from the characteristic function — no density, no invertibility
- The Gaussian Wigner matrix
- ‖Mu‖² for a fixed direction: its mean and variance, by Wick
- Cumulants, in the reading
notes·solutions
6Sep 23The semicircle law by counting paths
- The theorem, and the strategy: moments are traces
- E tr M² and E tr M⁴ by hand
- Matrix powers as sums over paths; from a path to a pair partition
- Only tree paths survive; non-crossing pairings and the Catalan numbers
- The fluctuations are small
notes·solutions
7Sep 28The Stieltjes transform
- The resolvent and mN(z) = (1/N) tr (M − zI)−1
- Eigenvalues are the poles: counting them in an interval by a contour integral
- What to compute: limε→0 limN→∞ (1/π) Im mN(x + iε), and why the limits come in that order
- Near z = ∞ the transform is the generating function of the traces
notes·solutions
8Sep 30The semicircle law by the Stieltjes transform
- Schur complement: one diagonal entry of a resolvent through a smaller resolvent
- The self-consistent equation m² + zm + 1 = 0, with explicit error bounds
- The trace concentrates: a Poincaré inequality and a cancellation of N's
- Stability of the equation, and the density read off the unit circle
notes·solutions
9Oct 5Gaussian processes: definition, feature maps, and maxima
- A Gaussian process is a Gaussian vector indexed by an arbitrary set; mean μt and kernel Σs,t
- Feature maps: every kernel is an inner product; Mu on the sphere, Brownian motion, the wide network
- The canonical metric
- The union bound E max ≤ σ√(2 log n), with or without independence; the Gaussian width as homework
notes·solutions
10Oct 7Gaussian processes: suprema and chaining
- Independent Gaussians reach √(2 log n)
- Covering numbers; Dudley's inequality stated, applied to the sphere, and proved by chaining
- Sudakov's lower bound and Talagrand's majorizing measure theorem, stated
notes·solutions
11Oct 12Gaussian processes: comparison
- Sudakov–Fernique and Slepian by Gaussian interpolation
- The operator norm of a Gaussian matrix: E‖M‖ ≤ √N + √d
- Sudakov's lower bound, proved; what the midterm covers
notes·solutions
Midterm
Wednesday Oct 14 · in class · 80 minutes · closed book
Three problems: a pre-registered theorem stated, proved and applied; a second pre-registered theorem stated and proved; definitions, theorem statements and a computation. Everything that can be asked is on the guide. The practice midterm has the same shape and length as the real one — try it under time before reading the solutions.
What to know — definitions, theorem statements, and the pre-registered theorems to proveguide
Practice midterm, solutionspdf
Module II · Dynamics and Diffusion
10 sessions · Oct 26 – Dec 2
Notes for each block are linked as the block begins; solutions are posted once the block is finished. The handwritten notes continue in hand-notes.pdf.
1Oct 26Martingales: conditional expectation and definitions
- L² and orthogonal projection, recalled; conditional expectation and its rules; the freezing lemma
- Filtrations, martingales, and orthogonal increments
- Examples: sums, Doob martingales, likelihood ratios, Pólya's urn, stochastic gradient descent
notes·solutions
2Oct 28Martingales: stopping
- The Doob decomposition and quadratic variation; martingale transforms
- Stopping times and optional stopping; gambler's ruin, the biased walk, Wald's identities
- Doob's upcrossing inequality; Ville's inequality
notes·solutions
3Nov 2Martingales: limit theorems and the CLT
- Doob's maximal inequality; convergence in L²; Bayesian learning of a coin
- The martingale CLT: comparison with the Gaussian-increment martingale
- Adaptively collected data: bandits and autoregressions
notes·solutions
4Nov 4Martingales: concentration
- Azuma–Hoeffding; McDiarmid and the generalization gap
- Ville's inequality
- Freedman's inequality: variance-adaptive and uniform in time
notes·solutions
5Nov 9SGD: stochastic approximation
- The one-step recursion; constant steps and the noise floor
- The Robbins–Monro rate 1/n; least mean squares
- The Robbins–Monro conditions and the ODE picture
notes·solutions
6Nov 11SGD: fluctuations and averaging
- Asymptotic normality of the iterates by the martingale CLT
- Polyak–Ruppert averaging attains the efficient variance
- High-probability bounds along the trajectory by Freedman
notes·solutions
7Nov 16Langevin dynamics: sampling by noisy gradient descent
- The unadjusted Langevin algorithm; integration by parts against e−U and why e−U is the equilibrium
- The Gaussian target exactly; SGD as sampling from a Gibbs measure
- Contraction by coupling; convergence, and the discretization bias
- What fails without strong convexity, and how smoothing fixes it
notes
8Nov 18Diffusion models: the forward process and Tweedie's formula
- Noising as Langevin dynamics for a quadratic potential; a Gaussian channel applied repeatedly
- Tweedie's formula from integration by parts
- Denoising as score estimation
notes
9Nov 30Diffusion models: reversal and score matching
- The reverse of a Gaussian step, and the score as its drift
- Denoising score matching; the DDPM objective
- Ancestral sampling
notes
10Dec 2Diffusion models: guarantees, the variational bound, and the close
- The chain rule for relative entropy and the error of the sampler
- The ELBO
- The heat equation and the Gaussian as its kernel; relative entropy across the course
notes