Date | Topic | Reading | Notes |
---|---|---|---|
Jan 21 | Introduction; linesearch | Ch. 1-2 of Givens+Hoeting | See Sun and Yuan (2006) for further details on convergence analysis |
Jan 23 | Choosing search directions: Newton, generalized linear models, inexact Newton, quasi-Newton, Fisher scoring, BFGS | See Vandenberghe's notes for some further background | |
Jan 28 | Exploiting special structure to solve Newton linear equations more efficiently: banded, sparse, low-rank (etc.) matrices | ||
Jan 30 | Conjugate gradients | Shewchuk (1994) | |
Feb 4 | Preconditioning. Toeplitz and circulant matrices | See Chan and Ng (1996) on PCG for Toeplitz systems. | |
Feb 6 | Expectation maximization | See Dempster et al (1977) and Neal and Hinton (1999) for further reading. | |
Feb 11 | Gaussian process regression | Guest lecture by John Cunningham. See notes here, and Rasmussen and Williams (2006) for more background on GP regression | |
Feb 13 | Snow day - class cancelled | ||
Feb 18 | Constrained and non-smooth optimization: convex functions; interior point methods | Boyd and Vandenberghe, ch. 3-4 | |
Feb 20 | Linear, quadratic, and semidefinite programs | ||
Feb 25 | LASSO methods | Efron et al (2004), Zou et al (2007), Friedman et al (2010), Bradley et al (2011) | |
Feb 27 | Convex duality, KKT conditions. Support vector methods | Boyd and Vandenberghe, ch. 5 | |
Mar 4 | A brief tour of some advanced topics: proximal methods, dual decomposition, and convex relaxation | Background: Bach et al (2011), Boyd et al (2011), Luo et al (2010) | |
Mar 6, 11 | Graphical models; dynamic programming; message passing; LP relaxations | Rabiner tutorial, Jordan (2004) | Background: Wainwright and Jordan (2008), Smith et al (2012), MP and AMP notes by A. Maleki, LP relaxation notes by Y.-W. Teh |
Mar 13 | Monte Carlo basics. Rejection and importance sampling; adaptive rejection sampling | Ch. 1-7 of Robert and Casella | Background: Devroye (1986), Doucet (2010). |
Mar 18-20 | No class | Spring break | |
Mar 25 | Short project presentations | ||
Mar 27 | Guest lecture by Rahul Mazumder on integer programming via branch-and-bound | notes | |
Apr 1-3, 10 | Metropolis-Hastings. Gibbs sampling: slice sampling, Bayesian lasso, spike-and-slab, hit and run. MCMC diagnostics. Rao-Blackwellization. Adaptive simulated tempering. | Background: Park and Casella (2008), Neal (2003), Doucet (2010), Mohamed et al. (2011), Salakhutdinov (2010) | |
Apr 8 | Hamiltonian Monte Carlo | Neal (2010) |
Guest lecture by Ari Pakman. Further background: Hoffman and Gelman (2012), Pakman and Paninski (2013) |
Apr 15 | Sequential Monte Carlo | Doucet and Johansen (2011), Pitt and Shephard (1999) | Further reading collected by A. Doucet |
Apr 17 | Nonparametric Bayes methods | Guest lecture by Peter Orbanz | |
Apr 22, 24 | Deterministic approximations for posteriors: variational Bayes, expectation propagation, Gaussian quadrature | Ormerod and Wand (2010), Sudderth (2002) | Additional reading: Hoffman et al (2013) |
Apr 29 | Stochastic approximation methods | Ch. 4-5 in Spall (2003) | Guest lecture by Lauren Hannah; notes |
May 1 | Project presentations | Send me your report as a .pdf by May 9. |