Learning Privately in High Dimensions

Add to your list(s) Download to your calendar using vCal

• Prof. Marco Mondelli, Institute of Science and Technology Austria
• Wednesday 28 May 2025, 14:00-15:00
• MR5, CMS Pavilion A.

If you have a question about this talk, please contact Prof. Ramji Venkataramanan.

Deep learning models memorize training samples and, as such, they are vulnerable to various attacks directed to retrieve information about the training dataset. The goal of the talk is to quantify this phenomenon, as well as the corresponding defenses in terms of differentially private algorithms, through the lens of high-dimensional regression.

The first part of the talk considers empirical risk minimization, focusing on the memorization of spurious features that are uncorrelated with the learning task. We relate such memorization to two separate terms: (i) the stability of the model with respect to individual training samples, and (ii) the feature alignment between the spurious feature and the full sample. This shows that memorization weakens as the generalization capability increases and, through the precise analysis of the feature alignment, we describe the role of the model and of its activation function. We then discuss spurious correlations between non-predictive features and the associated labels in the training data. We provide a statistical characterization of how such correlations are learnt in high-dimensional regression, unveiling the role of the data covariance, the regularization strength and the over-parameterization.

The second part of the talk considers differentially private gradient descent, a popular algorithm with provable guarantees on the privacy of the training data. While understanding its performance cost with respect to standard gradient descent has received remarkable attention, existing bounds on the excess population risk degrade with over-parameterization. This leaves practitioners without clear guidance, leading some to reduce the effective number of trainable parameters to improve performance, while others use larger models to achieve better results through scale. We show that, for any sufficiently over-parameterized random features model, privacy can be obtained for free, i.e., the excess population risk is negligible not only when the privacy parameter \epsilon has constant order, but also in the strongly private setting \epsilon = o(1). This challenges the common wisdom that over-parameterization inherently hinders performance in private learning.

Bio: Marco Mondelli received the B.S. and M.S. degree in Telecommunications Engineering from the University of Pisa, Italy, in 2010 and 2012, respectively. In 2016, he obtained his Ph.D. degree in Computer and Communication Sciences at EPFL . In 2017-2019, he was a Postdoctoral Scholar in the Department of Electrical Engineering at Stanford University. In 2018, he was also a Research Fellow with the Simons Institute for the Theory of Computing, for the program on “Foundations of Data Science”. He has been a faculty member at the Institute of Science and Technology Austria (ISTA) since 2019, first as an Assistant Professor and, since 2025, as a Professor. His research interests include data science, machine learning, high-dimensional statistics, information theory, and coding theory. He is the recipient of a number of fellowships and awards, including the Jack K. Wolf ISIT Student Paper Award in 2015, the STOC Best Paper Award in 2016, the EPFL Doctorate Award in 2018, the Simons-Berkeley Research Fellowship in 2018, the Lopez-Loreta Prize in 2019, the Information Theory Society Best Paper Award in 2021 and the ERC Starting Grant in 2024.

This talk is part of the Information Theory Seminar series.

This talk is included in these lists:

• All CMS events
• All Talks (aka the CURE list)
• CMS Events
• DPMMS Lists
• DPMMS info aggregator
• DPMMS lists
• Hanchen DaDaDash
• Information Theory Seminar
• Interested Talks
• MR5, CMS Pavilion A
• School of Physical Sciences
• Statistical Laboratory info aggregator
• bld31

Note that ex-directory lists are not shown.