ORF 526 · Princeton · Fall 2026

Probability for
Modern Machine Learning

The Gaussian The semicircle
MW 10:40–12:00 22 sessions Sep 2 – Dec 2 No measure theory

The course

A graduate probability course built out of tools and examples rather than a slow development of the foundations.

Twenty-two sessions, in two modules. Each one is organized around a technique worth owning — Lindeberg swapping, Stein’s method, Wick’s theorem, the resolvent, Gaussian integration by parts — and around an example that shows what the technique is for. A substantial part of that second half is modern machine learning: wide networks at initialization, kernel regression, the spectrum of a Gaussian random matrix, the maxima of Gaussian processes, stochastic gradient descent, and diffusion models.

No measure theory. Nothing is assumed beyond linear algebra, multivariable calculus and an undergraduate probability course. Where a standard fact from analysis is needed — Fourier inversion, the spectral theorem — it is quoted honestly and marked as such in the notes.

How it works

Homework, assigned every sessionnot graded
Weekly quiz, 15 min, in class20%
Midterm, in class30%
Final50%

The lowest quiz is dropped. Both examinations are closed book.

Grades

A90–100
B80–89
C70–79
D60–69
F0–59

On the homework and the quizzes. The exercises live inside the lecture notes, typed A, B and C: A checks that nothing was missed, B asks whether the argument was understood, and C withholds exactly one idea. They are not collected and not graded — use whatever help you like, including an AI assistant, and if you are short of time do the C problems. Full solutions are posted beside the notes once a block of lectures is finished; read them after you have tried the problems, not instead. The weekly quiz is where that work is cashed in: closed book, by hand, drawn from the week’s assigned exercises. The homework is the exploration layer; the quiz is the verification layer, and it is the only thing that needs policing.

Module I · Gaussians 12 sessions · Sep 2 – Oct 14

Typed notes and solutions are linked session by session below. The handwritten notes from lecture, in one running file updated after each class: hand-notes.pdf.

1Sep 2
Gaussians: definition and linear structure
  • Definition by the Fourier transform, and why not the density
  • Linear images, existence, rotational invariance
  • Uncorrelated and jointly Gaussian implies independent
  • Maximum entropy at fixed covariance
notes·solutions
2Sep 9
Gaussians: conditioning as projection
  • Conditional expectation as orthogonal projection
  • The Schur complement, derived
  • Tower property and total variance, read off the picture
  • Bayesian posteriors and ridge regression; Tweedie's formula
notes·solutions
3Sep 14
CLT: Lindeberg's method
  • Maximum entropy, recalled; why a Gaussian at all
  • What convergence in distribution means, and why smooth test functions
  • Lindeberg's swapping argument, proved
  • Lindeberg's and Feller's conditions; the CLT for triangular arrays
  • Records: why random search improves only log n times, and why the Gaussian/Poisson dial is s_n
notes·solutions
4Sep 16
Stein's method
  • The Gaussian characterized by an identity, not a limit
  • Stein's equation; the CLT with a rate
  • Chen–Stein for Poisson; dependence
notes·solutions
5Sep 21
Wick's theorem, and a Gaussian random matrix
  • Every moment of a Gaussian vector, as a sum over pair partitions
  • Proved from the characteristic function — no density, no invertibility
  • The Gaussian Wigner matrix
  • ‖Mu‖² for a fixed direction: its mean and variance, by Wick
  • Cumulants, in the reading
notes·solutions
6Sep 23
The semicircle law by counting paths
  • The theorem, and the strategy: moments are traces
  • E tr M² and E tr M⁴ by hand
  • Matrix powers as sums over paths; from a path to a pair partition
  • Only tree paths survive; non-crossing pairings and the Catalan numbers
  • The fluctuations are small
notes·solutions
7Sep 28
The Stieltjes transform
  • The resolvent and mN(z) = (1/N) tr (M − zI)−1
  • Eigenvalues are the poles: counting them in an interval by a contour integral
  • What to compute: limε→0 limN→∞ (1/π) Im mN(x + iε), and why the limits come in that order
  • Near z = ∞ the transform is the generating function of the traces
notes·solutions
8Sep 30
The semicircle law by the Stieltjes transform
  • Schur complement: one diagonal entry of a resolvent through a smaller resolvent
  • The self-consistent equation m² + zm + 1 = 0, with explicit error bounds
  • The trace concentrates: a Poincaré inequality and a cancellation of N's
  • Stability of the equation, and the density read off the unit circle
notes·solutions
9Oct 5
Gaussian processes: definition, feature maps, and maxima
  • A Gaussian process is a Gaussian vector indexed by an arbitrary set; mean μt and kernel Σs,t
  • Feature maps: every kernel is an inner product; Mu on the sphere, Brownian motion, the wide network
  • The canonical metric
  • The union bound E max ≤ σ√(2 log n), with or without independence; the Gaussian width as homework
notes·solutions
10Oct 7
Gaussian processes: suprema and chaining
  • Independent Gaussians reach √(2 log n)
  • Covering numbers; Dudley's inequality stated, applied to the sphere, and proved by chaining
  • Sudakov's lower bound and Talagrand's majorizing measure theorem, stated
notes·solutions
11Oct 12
Gaussian processes: comparison
  • Sudakov–Fernique and Slepian by Gaussian interpolation
  • The operator norm of a Gaussian matrix: E‖M‖ ≤ √N + √d
  • Sudakov's lower bound, proved; what the midterm covers
notes·solutions
12Oct 14
Midterm examination
  • Sessions 1–11, closed book, in class
guide·practice·solutions
Midterm Wednesday Oct 14 · in class · 80 minutes · closed book

Three problems: a pre-registered theorem stated, proved and applied; a second pre-registered theorem stated and proved; definitions, theorem statements and a computation. Everything that can be asked is on the guide. The practice midterm has the same shape and length as the real one — try it under time before reading the solutions.

What to know — definitions, theorem statements, and the pre-registered theorems to proveguide
Practice midtermpdf
Practice midterm, solutionspdf
Module II · Dynamics and Diffusion 10 sessions · Oct 26 – Dec 2

Notes for each block are linked as the block begins; solutions are posted once the block is finished. The handwritten notes continue in hand-notes.pdf.

1Oct 26
Martingales: conditional expectation and definitions
  • L² and orthogonal projection, recalled; conditional expectation and its rules; the freezing lemma
  • Filtrations, martingales, and orthogonal increments
  • Examples: sums, Doob martingales, likelihood ratios, Pólya's urn, stochastic gradient descent
notes·solutions
2Oct 28
Martingales: stopping
  • The Doob decomposition and quadratic variation; martingale transforms
  • Stopping times and optional stopping; gambler's ruin, the biased walk, Wald's identities
  • Doob's upcrossing inequality; Ville's inequality
notes·solutions
3Nov 2
Martingales: limit theorems and the CLT
  • Doob's maximal inequality; convergence in L²; Bayesian learning of a coin
  • The martingale CLT: comparison with the Gaussian-increment martingale
  • Adaptively collected data: bandits and autoregressions
notes·solutions
4Nov 4
Martingales: concentration
  • Azuma–Hoeffding; McDiarmid and the generalization gap
  • Ville's inequality
  • Freedman's inequality: variance-adaptive and uniform in time
notes·solutions
5Nov 9
SGD: stochastic approximation
  • The one-step recursion; constant steps and the noise floor
  • The Robbins–Monro rate 1/n; least mean squares
  • The Robbins–Monro conditions and the ODE picture
notes·solutions
6Nov 11
SGD: fluctuations and averaging
  • Asymptotic normality of the iterates by the martingale CLT
  • Polyak–Ruppert averaging attains the efficient variance
  • High-probability bounds along the trajectory by Freedman
notes·solutions
7Nov 16
Langevin dynamics: sampling by noisy gradient descent
  • The unadjusted Langevin algorithm; integration by parts against e−U and why e−U is the equilibrium
  • The Gaussian target exactly; SGD as sampling from a Gibbs measure
  • Contraction by coupling; convergence, and the discretization bias
  • What fails without strong convexity, and how smoothing fixes it
notes
8Nov 18
Diffusion models: the forward process and Tweedie's formula
  • Noising as Langevin dynamics for a quadratic potential; a Gaussian channel applied repeatedly
  • Tweedie's formula from integration by parts
  • Denoising as score estimation
notes
–Nov 23
No class
  • Thanksgiving week
9Nov 30
Diffusion models: reversal and score matching
  • The reverse of a Gaussian step, and the score as its drift
  • Denoising score matching; the DDPM objective
  • Ancestral sampling
notes
10Dec 2
Diffusion models: guarantees, the variational bound, and the close
  • The chain rule for relative entropy and the error of the sampler
  • The ELBO
  • The heat equation and the Gaussian as its kernel; relative entropy across the course
notes