Statistical Machine Learning (Summer term 2020)

Due to Covid-19, this course is going to take place in a different form than usual. The short version is this:

Lectures and tutorials are going to start in the week of April 20.

You need to register for the course and for the tutorials: to register for the course, please proceed to this Ilias page: XXXXX (link will be provided soon). Ideally, please register by Apil 15, the latest possibility is April 23.

You also need to register for the tutorials and indicate your preferred time slots. Please proceed to this page: XXXXX (link will be provided soon). Ideally, please register by Apil 15, the latest possibility is April 23.

Lectures will be video-recorded, so everybody can watch them at home. We will provide the link to the first lectures in the first week of the term (April 20).

Tutorials will take place via zoom. Links will be provided.

On tuesday, April 21 at 8:30 there is going to be a first online lecture using zoom. In this lecture, Ulrike von Luxburg will explain the whole setup of the course. From the second week on, a Question and Answer session will take place every thursday at 8:30 via zoom. The link to these zoom meetings willl be provided here by April 15.

In the first week we will provide a tutorial for those students who have never used python before (if you know python, you do not need to attend). The tutorial is going to take place via zoom, link will be provided here. XXXX (soon).

Economics students: this year, the lecture will be open to a small set of economics students. If you would like to particpate, please get in touch with Prof. Joachim Grammig. He is then going to check whether you have the required background knowledge.

Who, when, where

The lecture is intended for Master students in computer science or mathematics, but might also be interesting for students of other disciplines like physics, linguistics, economics, etc. If in doubt, please simply attend the first lecture and talk to me in the end.

Lectures (by Prof. U. von Luxburg). In case we will need them later this term, the time slots and booked rooms are Tuesday 8:15 - 10:00, Hörsaal 036, Neuphilologicum, Wilhelmstraße 50; Thursday 8:15-10:00, Hörsaal 22, Kupferbau, Hölderlinstraße 5.
Tutorials: In the first week of of April 20, there will be only one tutorial session. It will take place on Wednesday April 22 from 16:00-18:00. This tutorial session will give an introduction to Python (You should attend this tutorial if you are not already familiar with python and numpy). From the week of April 27, there will be multiple tutorial sessions every week. We have five tutorial groups, you need to register for these groups (the link will be provided during the next days). You can choose between Tuesday 14-16 (group 1), Tuesday 16-18 (group 2), Wednesday 14-16 (group 3), Wednesday 16-18 (group 4) and Thursday 16-18 (group 5). Once completed, the allocation to the tutorials will be made available here on the website.
Please join the ILIAS room so that we can give relevant information to you. There will be an email with further information on the class to all registered students with on Friday, April 17.
Teaching assistants and their contacts:

Until further notice, tutorials will take place via zoom. As usual, you will hand in a weekly assignment, both code and math. You will send everything to us via email and will get your correxted assignments, both code and math, returned to you via email. Math and coding exercises will then be discussed in the tutorials. You will also be allowed to work in groups of two (that is, two students hand in an assignment). Therefore, tutorial registration will allow you to state the name of a partner with whom you would like to be assigned to the same tutorial. Of course it will also be possible to find a partner during the first weeks of the semester.

Course material

Will be updated when the semester starts.
Recap material (aldready available):

linear algebra
probability theory

Python Tutorial (will be made available in the first week of the semester)

Tutorial.ipynb

Slides:

Version of last year (some parts will be removed/updated/added): pdf
New version of the slides, updated 2020-04-13:

Weekly assignments:

Will arrive here once the semester starts!

Contents of the lecture

The focus of the lecture is on both algorithmic and theoretic aspects of machine learning. We will cover many of the standard algorithms and learn about the general principles and theoretic results for building good machine learning algorithms. Topics range from well-established results to very recent results.
The basics:

Bayesian decision theory, no free lunch theorem.
Supervised learning problems (regression, classification): Simple baselines (nearest neibhbor, random forests); linear methods; regularization; non-linear kernel methods and the theory behind it
Unsupervised learning problems: Dimension reduction from PCA to manifold methods; Clustering from k-means to spectral clustering and spectral graph theory
Statistical Learning Theory: consistency and generalization bounds
How to model machine learning problems

Advanced topics:

Low rank matrix completion, compressed sensing
Ranking
Fairness in machine learning
Online learning and its theory

The following topics will NOT be covered, as there are alternative, dedicated lectures to each of these topics in our department: neural networks and deep learning; Bayesian and probabilistic machine learning; reinforcement learning.

Prerequisites

You need to know the basics of probability theory and linear algebra, as taught in the mathematics for computer science lectures in your bachelor degree, or even better as taught in the class Mathematics for Machine Learning. If you cannot remember them so well, I strongly urge you to recap the material (see below for links).

Assessment criteria and exams

There will be weekly assignments that you have to solve and hand in: theoretical assignments and programming assignments in python (an introduction to python will be given in one of the first tutorials; it will not be possible to use any other programming language). You need to achieve at least 50% of all assignment points to be admitted to the final exam.
Admissions of previous years: if you have attended the course Machine learning: algorithms and theory in 2019, and you got admitted to the exam, you do not need to get a new admission (we strongly encourage you to take part in the exercises, but you do not have to). Any admission before 2019 is invalid, you will need to the assignments again.
The final exams are written (Klausur), dates not confirmed yet.
You are not allowed to bring any material (books, slides, etc) except for what we call the controlled cheat sheet: one side (A4, one side only) of handwritten (!) notes, made by yourself. To get an idea how the exam will look like, here are some questions from an old exam.

Literature

There is no textbook that covers machine learning the way I want. We are going to use individual chapters of various books. Here is a list of some relevant books:

Shalev-Shwartz, Ben-David: Understanding Machine Learning. 2014.
Chris Bishop: Pattern recognition and Machine Learning. 2006.
Hastie, Tibshirani, Friedman: Elements of statistical learning. 2001. (the statistics point of view on machine learning, written by statisticians)
Kevin Murphy: Machine Learning, a probabilistic perspective, 2012 (for the probabilistic point of view)
Schölkopf, Smola: Learning with kernels. 2002. (for support vector machines and kernel algorithms)
Hastie, Tibshirani, Wainwright: Statistical learning with sparsity. 2015. (Lasso, low rank matrix methods)
Duda, Hart, Stork: Pattern classification. 2012. (a somewhat outdated classic, but some sections still good to read)
Mohri, Rostamizadeh, Talwalkar: Foundations of Machine Learning. 2012.(rather theoretical point of view)
Devroye, Györfi, Lugosi: A Probabilistic Theory of Pattern Recognition. 2013. (Covers statistical learning theory, we do not use much of it in this lecture.)