Theory of Machine Learning Group University of Tübingen

Machine Learning: Algorithms and Theory
(Summer term 2018)

Due to the many registrations for the course, we decided to
  1. move the Python Introduction to a larger room. Therefore we had to reschedule the Python introduction to Wednesday April 25 16:15-18:00 in room F119.
  2. have a fourth tutorial. So there will be 2 groups on Tuesday and 2 on Wednesday.
You can find the Allocation to the tutorials here.
Please note the following
  • Due to the free day on 1. of May next Tuesday, the tutorials on Tuesday (Group 1 and 2) will be postponed to Thursday 3. Mai 16-18 (Group 1 in room C215 and Group 2 in room C118).
  • If you hand in your assignment by E-Mail, please send it to your assigned tutor. If you and your partner are in different groups you can choose which tutor you send it to.
  • We will open a thread in the Ilias Forum, where you can search/ask for an exchange partner, in case you want to change your tutorial group.
The lecture is intended for Master students in computer science or mathematics, but might also be interesting for students from other disciplines like physics, linguistics, etc. If in doubt, please simply attend the first lecture and talk to me in the end.

Who, when, where

Lectures (by Prof. U. von Luxburg): Tuesday 12:15-14:00 and Thursday 10:15 - 12:00, Room F119 in Sand 6 (Computer Science Campus, Tuebingen). First lecture is on April 17.
Tutorials (by Diego Fioravanti, Tobias Frangen, Mortiz Haas, Siavash Haghiri): You can choose among Tue 16-18 (2 groups, A302 and C118) or Wed 16-18 (2 groups, A302 and C118). The regular tutorials start from the third week. In the first two weeks there will be a math recap on April 17 (in room F119) and a introduction to Python on Wednesday April 25 16:15-18:00 in room F119.
Please join the ILIAS room so that we can give relevant information to you.

Course material

You can find the most important information on our information sheet.
Ilias room Students who want to participate in this course please register in the Ilias room (link to come soon).
Recap material:
  • linear algebra
  • probability theory
Slides:
  • Version of March 2018
  • Latest version (updated 2018-04-24)
Weekly assignments:
  • Assignment 1 (due April 24), jupyter_test.ipynb
  • Assignment 2 (due May 3rd, exceptional on thursday due to public holiday on tuesday), assignment02.ipynb, train.csv and test.csv
  • Assignment 3 (due May 8), assignment03.ipynb

Literature

There is no textbook that covers machine learning the way I want. We are going to use individual chapters of various books. Here is a list of relevant books:
  • Shalev-Shwartz, Ben-David: Understanding Machine Learning. 2014.
  • Chris Bishop: Pattern recognition and Machine Learning. 2006.
  • Hastie, Tibshirani, Friedman: Elements of statistical learning. 2001. (the statistics point of view on machine learning, written by statisticians, not machine learners)
  • Mohri, Rostamizadeh, Talwalkar: Foundations of Machine Learning. 2012.
  • Schölkopf, Smola: Learning with kernels. 2002. (mainly support vector machines and kernel algorithms)
  • Hastie, Tibshirani, Wainwright: Statistical learning with sparsity. 2015. (Lasso, low rank matrix methods)
  • Duda, Hart, Stork: Pattern classification. 2012. (a classic, more than 20 years old, some sections still good to read)
  • Devroye, Györfi, Lugosi: A Probabilistic Theory of Pattern Recognition. 2013. (Covers statistical learning theory, we do not use much of it in this lecture.)
  • Dasgupta, Papadimitriou, Varzirani: Algorithms. 2006.

Contents of the lecture

The focus of the lecture is on both algorithmic and theoretic aspects of machine learning. We will cover many of the standard algorithms and learn about the general principles and theoretic results for building good machine learning algorithms. Topics range from well-established results to very recent results.
  • Bayesian decision theory, no free lunch theorem.
  • Supervised learning problems (regression, classification): Linear methods; regularization; non-linear kernel methods and the theory behind it
  • Unsupervised learning problems: Dimension reduction: (kernel) PCA, multi-dimensional scaling, manifold methods); spectral clustering and spectral graph theory
  • How to model machine learning problems
  • Learning Theory: ERM, consistency, generalization bounds
  • Low rank matrix completion, compressed sensing
  • Ranking
  • Online learning and its theory
The following topics are NOT going to be covered: decision trees, neural networks, deep networks, graphical models, Bayesian approaches to machine learning. You can learn about some of them in other courses in the department.

Prerequisites

You need to know the basics of probability theory and linear algebra, as taught in the mathematics for computer science lectures in your bachelor degree. If you cannot remember them so well, I strongly urge you to recap the material (see below for links).

Assessment criteria and exams

Weekly assignments (theoretical assignments and programming assignments in python --- an introduction to python will be given in one of the first tutorials ; it will not be possible to use any other programming language). You need to achieve at least 50% of all assignment points to be admitted to the final exam.
The final exams are written (Klausur), the first one at July 24, 12:00-14:00, the second one at October 10, 10 - 12:00 (roooms to be announced).