Date | Topic | Reading | Notes |
---|---|---|---|
Jan 12 | Introduction | Ch. 1-2 of Givens+Hoeting | See Sun and Yuan (2006) for further details on convergence analysis. Berland's notes on automatic differentiation. Mahsereci and Hennig (2016) on Bayesian linesearch. HW: derive the convergence rate of the secant method. |
Jan 19 | Choosing search directions: Newton, generalized linear models, inexact Newton, quasi-Newton, Fisher scoring, BFGS. Exploiting special structure to solve Newton linear equations more efficiently: banded, sparse, low-rank, block-structured (etc.) matrices | See Vandenberghe's notes for some further background | |
Jan 26 | Conjugate gradients. Preconditioning. Toeplitz, circulant, and Kronecker matrices. Application: Gaussian processes | Shewchuk (1994); see Chan and Ng (1996) on PCG for Toeplitz systems. Gardner et al '19, Loper et al '20 for fast GP inference. | See Rasmussen and Williams (2006) for more background on GP regression. Also notes by John Cunningham. Rahimi+Recht '07 on random features; Drineas + Mahoney '16 on randomized linear algebra. HW: code up a GP regression. |