6.S088 Modern Machine Learning: Simple Methods that Work

Welcome to 6.S088!


Over the past decade, interest in machine learning research has spiked drastically, with advancements in deep learning being a significant driving force. Indeed, deep learning has transformed many areas in computer science including computer vision, natural language processing, and reinforcement learning. Unfortunately, given the rapid pace of progress in deep learning, a newcomer looking for a simple set of guiding principles for building machine learning applications can be easily overwhelmed by the nuances of training deep networks. Thus, motivated by recent developments in machine learning, we present a simple class of machine learning methods that are easy to implement and which achieve competitive performance in practice. In particular, our methods rely on the recently established equivalence between kernel regression and infinite width neural networks given by the neural tangent kernel (NTK). In addition to being a theoretical tool for understanding neural networks, we demonstrate that the NTK is a simple method for achieving competitive results in a variety of machine learning applications including regression, classification, and matrix/image completion. We provide problem sets containing both theoretical and coding exercises with the aim of (1) providing newcomers to machine learning a simple toolkit for building effective machine learning models in practice and (2) preparing interested students for research in the area.


Units: 1-0-5

Prereqs: Knowledge of linear algebra (level of 18.06 or 18.700), analysis (level of 18.100), and probability (level of 6.041 or 18.600) is generally assumed. Familiarity with Python (in particular, NumPy) is also assumed. Knowledge of Fourier analysis (18.103), functional analysis (18.102), random matrix theory (18.338), and complex analysis (18.112) is suggested for students who want to pursue research in this area.

Schedule: Monday – Friday, January 18 – January 28, 1-2:30pm, room 32-141

Instructor: Adityanarayanan Radhakrishnan, aradha@mit.edu

Faculty Advisor: Prof Caroline Uhler

TAs: Max Luyten, George Stefanakis, Cathy Cai


Lecture Date Topics Problem Set Notes Videos
1 Tue, Jan. 18 Lecture 1: Course Overview and Preliminaries Problem Set 1: Review of Linear Algebra, Analysis, and Probability notes Course Overview
Math Review
2 Wed, Jan. 19 Lecture 2: Linear Regression Problem Set 2: Linear Regression and Kernel Regression notes Linear Regression
3 Thurs, Jan. 20 Lecture 3: Kernel Regression notes Kernel Regression
4 Fri, Jan. 21 Lecture 4: NNGP, Dual Activations, and Over-parameterization Problem Set 3: Random Fourier Features and NNGP Derivations notes NNGP
5 Mon, Jan. 24 Lecture 5: NTK Origin and Derivation Problem Set 4: NTK, Neural Tangents Library + Project Proposals notes NTK Introduction
6 Tue, Jan. 25 Lecture 6: NTK for Deep Networks and the Convolutional NTK (CNTK) notes NTK for Deep Networks
7 Wed, Jan. 26 Lecture 7: NTK Applications - Matrix Completion and Image Inpainting notes
8 Thurs, Jan. 27 Lecture 8: Additional Office Hours
9 Fri, Jan. 28 Project Proposal / Paper Review Presentations notes