Machine learning algorithms can leak a significant amount of information about their training data. A legitimate user of a model can reconstruct sensitive information about the training data, by having access to its predictions or parameters. Given that all privacy policies and regulations require auditing data-driven algorithms to assess their privacy risks, we are interested in a generic yet rigorous approach to perform a quantitative reasoning about the privacy risks of various machine learning algorithms. We are also interested in explaining the information leakage through machine learning algorithms, and understanding the sources of privacy risks.

Membership inference is emerging as a foundational technique to empirically quantify a lower bound on the information leakage of machine learning algorithms about the individual data records in their training set. This notion of leakage is what differentially private machine learning aims to mitigate. This makes membership inference a perfect tool to audit all different types of machine learning algorithms, in a consistent manner. It can also help making privacy guarantees (e.g., epsilon in differential privacy) more interpretable.

In this tutorial, we present a unified view of recent works on computing more accurate estimates of privacy loss for iterative learning algorithms. We will cover the foundations of practical inference attacks and provide a rigorous quantitative understanding of differentially private machine learning. The objective is to link the underlying relation between privacy concepts, attacks, protection mechanisms, and tools. We will go beyond the mechanics of the techniques, and explain why machine learning algorithms leak information about their training data, and how different membership inference algorithms partially exploit the information leakage. We give examples on how to audit machine learning algorithms using ML Privacy Meter, a popular open source tool developed for this purpose.