6.7920

Note: The schedule is subject to minor changes, and will be updated periodically with lecture slides and readings.
Note: All material will be posted in Canvas.

Date	Tuesday	Date	Thursday	Friday
PART 1: Dynamic Programming
		09/10	L01. What is sequential decision making? Introduction, finite-horizon problem formulation, Markov decision processes, course overview Readings: N1 §3, N2 §1-3, N3 §1; DPOC 1.1-1.3, 2.1; SB Ch1 (skim) [Lec] HW0 assigned (due 9/15, not graded)	R01. Inventory control. Readings: N3 §3; DPOC 3.2
09/15	L02. Dynamic programming - What makes sequential decision-making hard? Finite horizon dynamic programming algorithm, sequential decision making as shortest path Readings: DPOC 3.3-3.4 HW0 due (not graded), HW1 assigned	09/17	L03. Special structures - What makes some sequential decision-making problems easy? DP arguments, inventory problem, optimal stopping Readings: N3 §2; DPOC 3.1	R02. Combinatorial optimization as DP. Readings: DPOC App-B
09/22	L04. Special Structures Linear quadratic regulator HW1 due, HW2 assigned	09/24	L05. Non-discounted Infinite Horizon Problem LQR Readings: DPOC 1.4, 4.1-4.2	No recitation (student holiday)
09/29	L06. Discounted infinite horizon problems Bellman equations, value iteration Readings: DPOC2 1.1-1.2, 1.5, 2.1-2.3 HW2 due, HW3 assigned	10/01	L07. Discounted infinite horizon problems (Part 2) Policy iteration, geometric interpretation Readings: DPOC2 1.1-1.2, 1.5, 2.1-2.3	R03. Linear quadratic regulator
PART 2: Reinforcement Learning
10/06	L08. Model-free methods - From DP to RL Monte Carlo, policy evaluation, stochastic approximation of a mean, temporal differences Readings: NDP 5.1-5.3, SB 12.1-12.2 HW3 due, HW4 assigned	10/08	L09. Value-based reinforcement learning - Policy learning without knowing how the world works State-action value function, Q-learning Readings: NDP 5.6, 4.1-4.3	R04. Convergence of stochastic value iteration
10/13	NO CLASS (Monday schedule)	10/15	L10. Value-based reinforcement learning - Policy learning without knowing how the world works (Part 2) Stochastic approximation of a fixed point Readings: NDP 4.1-4.3, 6.1-6.2; DPOC2 6.3	R05. Approximate value iteration
10/20	L11. Approximate value-based RL - How to approximately solve an RL problem Function approximation, approximate policy evaluation Readings: DPOC2 2.5.3; NDP 3.1-3.2, SB 16.5 HW4 due, HW5 assigned	10/22	L12. Approximate value-based RL - How to approximately solve an RL problem (Part 2) Approximate VI, fitted Q iteration, DQN, DDQN and friends Readings: DPOC2 2.5.3; NDP 3.1-3.2, SB 16.5	R06. Performance difference lemma
10/27	L13. Policy gradient - Simplicity at the cost of variance Approximate policy iteration, policy gradient, variance reduction Readings: NDP 6.1; SB Ch13.1-13.4 HW5 due, HW6 assigned	10/29	L14. Actor-critic Methods - Bringing together value-based and policy-based RL Compatible function approximation, A2C, A3C, DDPG, SAC Readings: SB Ch13.5-13.8	R07. Review, part 1 Project proposals due
11/03	L15. Advanced policy gradient methods - Managing exploration vs exploitation Conservative policy iteration, NPG, TRPO, PPO Readings: RLTA 11.1-11.2, Ch12 HW6 due, HW7 assigned	11/05	L16. Multi-arm bandits - The prototypical exploration-exploitation dilemma Readings: Bandits Ch 1, Ch 8	R08. Review, part 2
PART 3: Special topics
11/10	L17. Evaluation in RL - Is the RL method working? Sensitivity analysis of RL, benchmarking, statistical methods, overfitting to benchmarks, model-based transfer learning Readings: Lecture Appendices A-C HW7 due, HW8 assigned	11/12	L18. Quiz	R09. Concentration inequalities and statistical analysis
11/17	L19. Monte Carlo tree search Online planning, MCTS, AlphaGo, AlphaGoZero, MuZero Readings: SB 16.6	11/19	L20. Guest Lecture HW8 due
11/24	L21. Guest Lecture	11/26	NO CLASS (Thanksgiving)	No class (Thanksgiving)
12/01	L22. Guest Lecture	12/03	L23. Guest Lecture
12/08	L24. Guest Lecture / Final Project Presentations	12/10	L25. Guest Lecture / Final Project Presentations