Machine Learning Theory

CS 7545 -- Georgia Tech

View on GitHub

Course Information

Course Description

This course will study theoretical aspects of prediction and decision-making probelms, and to explore the mathematical underpinnings of machine learning. We hope to bring students to the frontiers of research and to develop tools that can be used to contribute to emerging literature. The course will cover, among other things, concentration inequalities, uniform deviation bounds, Vapnik-Chervonenkis Theory, Rademacher Complexity, margin bounds, boosting, some theoretical aspects of deep learning, online learning theory, regret minimization, multi-armed bandit algorithms, and connections to convex optimization. Additional topics may be covered if time permits, including reinforcement learning theory, differential privacy, sampling, and and other areas of interest.

Prerequisites: Familiarity with the analysis of algorithms, probabilistic analysis, and several similar topics. CS7641 (Machine Learning) may be helpful but not strictly necessary. The material will be about 90% “theory” and thus students must have a strong mathematical background. We shall rely heavily on techniques from calculus, probability, and convex analysis, but many tools will be reviewed in lecture.

Coursework: There will be 4 problem sets throughout the semester.

Grade Breakdown:

Note: The exam will be held on Thursday March 30 during classtime.

Lecture notes

Every lecture will have two student scribes, and this pair of students will be asked to take detailed notes in class on what was covered. The quality of these notes will be graded!

Grade Rubric:

Here is the course wiki.

Note: all students are allowed and encouraged to contribute to this wiki! Once the scribes have submitted their notes, other students are welcome to add any additional comments, results, material, etc. to the lecture notes. It would be great if the wiki provides a broad set of resources for students, beyond what was covered in lecture.

References:

Roughly half of the course will follow material from the following text:

Much of the material in online learning is specific to this course. For students that want more in-depth reading material on this topic, however, there are several surveys released in the last several years that explore several many areas we shall cover. These include:

The Latex template for HW submissions is available here.

Previous offerings of the course:

Homeworks

Homework Due Date Solution
Homework 1 January 31, 2023, 11:59 pm HW1 solution
Homework 2 February 28, 2023, 11:59 pm HW2 solution
Homework 3 March 19, 2023, 11:59 pm HW3 solution
Homework 4 April 25, 2023, 11:59 pm HW4 solution